"` when the target is a LiteLLM endpoint that requires authentication. Adjust the headers, or payload to target other MCP methods. Matching failures between `curl` and LiteLLM confirm that the MCP server or network/OAuth layer is the culprit.
+
+## Review Logs
+
+Well-scoped logs make it clear whether LiteLLM reached the MCP server and what happened next.
+
+### Access Log Example (successful MCP call)
+```text
+INFO: 127.0.0.1:57230 - "POST /everything/mcp HTTP/1.1" 200 OK
+```
+
+### Error Log Example (failed MCP call)
+```text
+07:22:00 - LiteLLM:ERROR: client.py:224 - MCP client list_tools failed - Error Type: ExceptionGroup, Error: unhandled errors in a TaskGroup (1 sub-exception), Server: http://localhost:3001/mcp, Transport: MCPTransport.http
+ httpcore.ConnectError: All connection attempts failed
+ERROR:LiteLLM:MCP client list_tools failed - Error Type: ExceptionGroup, Error: unhandled errors in a TaskGroup (1 sub-exception)...
+ httpx.ConnectError: All connection attempts failed
+```
diff --git a/docs/my-website/docs/observability/logfire_integration.md b/docs/my-website/docs/observability/logfire_integration.md
index b75c5bfd496..a1bd43a4bc4 100644
--- a/docs/my-website/docs/observability/logfire_integration.md
+++ b/docs/my-website/docs/observability/logfire_integration.md
@@ -40,6 +40,10 @@ import os
# from https://logfire.pydantic.dev/
os.environ["LOGFIRE_TOKEN"] = ""
+# Optionally customize the base url
+# from https://logfire.pydantic.dev/
+os.environ["LOGFIRE_BASE_URL"] = ""
+
# LLM API Keys
os.environ['OPENAI_API_KEY']=""
diff --git a/docs/my-website/docs/providers/azure_ai/azure_model_router.md b/docs/my-website/docs/providers/azure_ai/azure_model_router.md
new file mode 100644
index 00000000000..5e14c7283f6
--- /dev/null
+++ b/docs/my-website/docs/providers/azure_ai/azure_model_router.md
@@ -0,0 +1,232 @@
+# Azure Model Router
+
+Azure Model Router is a feature in Azure AI Foundry that automatically routes your requests to the best available model based on your requirements. This allows you to use a single endpoint that intelligently selects the optimal model for each request.
+
+## Key Features
+
+- **Automatic Model Selection**: Azure Model Router dynamically selects the best model for your request
+- **Cost Tracking**: LiteLLM automatically tracks costs based on the actual model used (e.g., `gpt-4.1-nano`), not the router endpoint
+- **Streaming Support**: Full support for streaming responses with accurate cost calculation
+
+## LiteLLM Python SDK
+
+### Basic Usage
+
+```python
+import litellm
+import os
+
+response = litellm.completion(
+ model="azure_ai/azure-model-router",
+ messages=[{"role": "user", "content": "Hello!"}],
+ api_base="https://your-endpoint.cognitiveservices.azure.com/openai/v1/",
+ api_key=os.getenv("AZURE_MODEL_ROUTER_API_KEY"),
+)
+
+print(response)
+```
+
+### Streaming with Usage Tracking
+
+```python
+import litellm
+import os
+
+response = await litellm.acompletion(
+ model="azure_ai/azure-model-router",
+ messages=[{"role": "user", "content": "hi"}],
+ api_base="https://your-endpoint.cognitiveservices.azure.com/openai/v1/",
+ api_key=os.getenv("AZURE_MODEL_ROUTER_API_KEY"),
+ stream=True,
+ stream_options={"include_usage": True},
+)
+
+async for chunk in response:
+ print(chunk)
+```
+
+## LiteLLM Proxy (AI Gateway)
+
+### config.yaml
+
+```yaml
+model_list:
+ - model_name: azure-model-router
+ litellm_params:
+ model: azure_ai/azure-model-router
+ api_base: https://your-endpoint.cognitiveservices.azure.com/openai/v1/
+ api_key: os.environ/AZURE_MODEL_ROUTER_API_KEY
+```
+
+### Start Proxy
+
+```bash
+litellm --config config.yaml
+```
+
+### Test Request
+
+```bash
+curl -X POST http://localhost:4000/chat/completions \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer sk-1234" \
+ -d '{
+ "model": "azure-model-router",
+ "messages": [{"role": "user", "content": "Hello!"}]
+ }'
+```
+
+## Add Azure Model Router via LiteLLM UI
+
+This walkthrough shows how to add an Azure Model Router endpoint to LiteLLM using the Admin Dashboard.
+
+### Select Provider
+
+Navigate to the Models page and select "Azure AI Foundry (Studio)" as the provider.
+
+#### Navigate to Models Page
+
+
+
+#### Click Provider Dropdown
+
+
+
+#### Choose Azure AI Foundry
+
+
+
+### Configure Model Name
+
+Set up the model name by entering `azure_ai/` followed by your model router deployment name from Azure.
+
+#### Click Model Name Field
+
+
+
+#### Select Custom Model Name
+
+
+
+#### Enter LiteLLM Model Name
+
+
+
+#### Click Custom Model Name Field
+
+
+
+#### Type Model Prefix
+
+Type `azure_ai/` as the prefix.
+
+
+
+#### Copy Model Name from Azure Portal
+
+Switch to Azure AI Foundry and copy your model router deployment name.
+
+
+
+
+
+#### Paste Model Name
+
+Paste to get `azure_ai/azure-model-router`.
+
+
+
+### Configure API Base and Key
+
+Copy the endpoint URL and API key from Azure portal.
+
+#### Copy API Base URL from Azure
+
+
+
+#### Enter API Base in LiteLLM
+
+
+
+
+
+#### Copy API Key from Azure
+
+
+
+#### Enter API Key in LiteLLM
+
+
+
+### Test and Add Model
+
+Verify your configuration works and save the model.
+
+#### Test Connection
+
+
+
+#### Close Test Dialog
+
+
+
+#### Add Model
+
+
+
+### Verify in Playground
+
+Test your model and verify cost tracking is working.
+
+#### Open Playground
+
+
+
+#### Select Model
+
+
+
+#### Send Test Message
+
+
+
+#### View Logs
+
+
+
+#### Verify Cost Tracking
+
+Cost is tracked based on the actual model used (e.g., `gpt-4.1-nano`).
+
+
+
+## Cost Tracking
+
+LiteLLM automatically handles cost tracking for Azure Model Router by:
+
+1. **Detecting the actual model**: When Azure Model Router routes your request to a specific model (e.g., `gpt-4.1-nano-2025-04-14`), LiteLLM extracts this from the response
+2. **Calculating accurate costs**: Costs are calculated based on the actual model used, not the router endpoint name
+3. **Streaming support**: Cost tracking works correctly for both streaming and non-streaming requests
+
+### Example Response with Cost
+
+```python
+import litellm
+
+response = litellm.completion(
+ model="azure_ai/azure-model-router",
+ messages=[{"role": "user", "content": "Hello!"}],
+ api_base="https://your-endpoint.cognitiveservices.azure.com/openai/v1/",
+ api_key="your-api-key",
+)
+
+# The response will show the actual model used
+print(f"Model used: {response.model}") # e.g., "gpt-4.1-nano-2025-04-14"
+
+# Get cost
+from litellm import completion_cost
+cost = completion_cost(completion_response=response)
+print(f"Cost: ${cost}")
+```
+
+
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_01.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_01.jpeg
new file mode 100644
index 00000000000..42654600f74
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_01.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_02.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_02.jpeg
new file mode 100644
index 00000000000..b9feab050ec
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_02.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_03.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_03.jpeg
new file mode 100644
index 00000000000..3f55ebf0121
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_03.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_04.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_04.jpeg
new file mode 100644
index 00000000000..1626c78bd1b
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_04.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_05.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_05.jpeg
new file mode 100644
index 00000000000..bef736e361d
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_05.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_06.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_06.jpeg
new file mode 100644
index 00000000000..bfeb767eea7
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_06.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_07.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_07.jpeg
new file mode 100644
index 00000000000..eed742a8c68
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_07.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_08.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_08.jpeg
new file mode 100644
index 00000000000..e72a6e92e77
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_08.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_09.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_09.jpeg
new file mode 100644
index 00000000000..5fe1421c2a4
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_09.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_10.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_10.jpeg
new file mode 100644
index 00000000000..60aa80063fc
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_10.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_11.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_11.jpeg
new file mode 100644
index 00000000000..98694fbb9be
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_11.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_12.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_12.jpeg
new file mode 100644
index 00000000000..77922ccea01
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_12.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_13.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_13.jpeg
new file mode 100644
index 00000000000..2cb80d0826a
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_13.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_14.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_14.jpeg
new file mode 100644
index 00000000000..8225023658c
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_14.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_15.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_15.jpeg
new file mode 100644
index 00000000000..7bd72852881
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_15.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_16.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_16.jpeg
new file mode 100644
index 00000000000..e3dbd75acae
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_16.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_17.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_17.jpeg
new file mode 100644
index 00000000000..ba5fd539138
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_17.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_18.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_18.jpeg
new file mode 100644
index 00000000000..1ead4bee962
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_18.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_19.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_19.jpeg
new file mode 100644
index 00000000000..ec7fa9c3bcb
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_19.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_20.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_20.jpeg
new file mode 100644
index 00000000000..2999fcd678e
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_20.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_21.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_21.jpeg
new file mode 100644
index 00000000000..1226e29d648
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_21.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_22.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_22.jpeg
new file mode 100644
index 00000000000..4455b552b81
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_22.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_23.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_23.jpeg
new file mode 100644
index 00000000000..4fa88bdb965
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_23.jpeg differ
diff --git a/docs/my-website/docs/providers/azure_ai/img/azure_model_router_24.jpeg b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_24.jpeg
new file mode 100644
index 00000000000..7fb61d1cce1
Binary files /dev/null and b/docs/my-website/docs/providers/azure_ai/img/azure_model_router_24.jpeg differ
diff --git a/docs/my-website/docs/providers/openrouter.md b/docs/my-website/docs/providers/openrouter.md
index a1ed6c4466e..38eb998c98b 100644
--- a/docs/my-website/docs/providers/openrouter.md
+++ b/docs/my-website/docs/providers/openrouter.md
@@ -93,3 +93,120 @@ response = embedding(
)
print(response)
```
+
+## Image Generation
+
+OpenRouter supports image generation through select models like Google Gemini image generation models. LiteLLM transforms standard image generation requests to OpenRouter's chat completion format.
+
+### Supported Parameters
+
+- `size`: Maps to OpenRouter's `aspect_ratio` format
+ - `1024x1024` → `1:1` (square)
+ - `1536x1024` → `3:2` (landscape)
+ - `1024x1536` → `2:3` (portrait)
+ - `1792x1024` → `16:9` (wide landscape)
+ - `1024x1792` → `9:16` (tall portrait)
+
+- `quality`: Maps to OpenRouter's `image_size` format (Gemini models)
+ - `low` or `standard` → `1K`
+ - `medium` → `2K`
+ - `high` or `hd` → `4K`
+
+- `n`: Number of images to generate
+
+### Usage
+
+```python
+from litellm import image_generation
+import os
+
+os.environ["OPENROUTER_API_KEY"] = "your-api-key"
+
+# Basic image generation
+response = image_generation(
+ model="openrouter/google/gemini-2.5-flash-image",
+ prompt="A beautiful sunset over a calm ocean",
+)
+print(response)
+```
+
+### Advanced Usage with Parameters
+
+```python
+from litellm import image_generation
+import os
+
+os.environ["OPENROUTER_API_KEY"] = "your-api-key"
+
+# Generate high-quality landscape image
+response = image_generation(
+ model="openrouter/google/gemini-2.5-flash-image",
+ prompt="A serene mountain landscape with a lake",
+ size="1536x1024", # Landscape format
+ quality="high", # High quality (4K)
+)
+
+# Access the generated image
+image_data = response.data[0]
+if image_data.b64_json:
+ # Base64 encoded image
+ print(f"Generated base64 image: {image_data.b64_json[:50]}...")
+elif image_data.url:
+ # Image URL
+ print(f"Generated image URL: {image_data.url}")
+```
+
+### Using OpenRouter-Specific Parameters
+
+You can also pass OpenRouter-specific parameters directly using `image_config`:
+
+```python
+from litellm import image_generation
+import os
+
+os.environ["OPENROUTER_API_KEY"] = "your-api-key"
+
+response = image_generation(
+ model="openrouter/google/gemini-2.5-flash-image",
+ prompt="A futuristic cityscape at night",
+ image_config={
+ "aspect_ratio": "16:9", # OpenRouter native format
+ "image_size": "4K" # OpenRouter native format
+ }
+)
+print(response)
+```
+
+### Response Format
+
+The response follows the standard LiteLLM ImageResponse format:
+
+```python
+{
+ "created": 1703658209,
+ "data": [{
+ "b64_json": "iVBORw0KGgoAAAANSUhEUgAA...", # Base64 encoded image
+ "url": None,
+ "revised_prompt": None
+ }],
+ "usage": {
+ "input_tokens": 10,
+ "output_tokens": 1290,
+ "total_tokens": 1300
+ }
+}
+```
+
+### Cost Tracking
+
+OpenRouter provides cost information in the response, which LiteLLM automatically tracks:
+
+```python
+response = image_generation(
+ model="openrouter/google/gemini-2.5-flash-image",
+ prompt="A cute baby sea otter",
+)
+
+# Cost is available in the response metadata
+print(f"Request cost: ${response._hidden_params['additional_headers']['llm_provider-x-litellm-response-cost']}")
+```
diff --git a/docs/my-website/docs/providers/sap.md b/docs/my-website/docs/providers/sap.md
index 4bc72c27045..16f30a2e99c 100644
--- a/docs/my-website/docs/providers/sap.md
+++ b/docs/my-website/docs/providers/sap.md
@@ -12,100 +12,340 @@ LiteLLM supports SAP Generative AI Hub's Orchestration Service.
| Supported Endpoints | `/chat/completions`, `/embeddings` |
| API Reference | [SAP AI Core Documentation](https://help.sap.com/docs/sap-ai-core) |
+## Prerequisites
+
+Before you begin, ensure you have:
+
+1. **SAP BTP Account** with access to SAP AI Core
+2. **AI Core Service Instance** provisioned in your subaccount
+3. **Service Key** created for your AI Core instance (this contains your credentials)
+4. **Resource Group** with deployed AI models (check with your SAP administrator)
+
+:::tip Where to Find Your Credentials
+Your credentials come from the **Service Key** you create in SAP BTP Cockpit:
+
+1. Navigate to your **Subaccount** → **Instances and Subscriptions**
+2. Find your **AI Core** instance and click on it
+3. Go to **Service Keys** and create one (or use existing)
+4. The JSON contains all values needed below
+
+The service key JSON looks like this:
+
+```json
+{
+ "clientid": "sb-abc123...",
+ "clientsecret": "xyz789...",
+ "url": "https://myinstance.authentication.eu10.hana.ondemand.com",
+ "serviceurls": {
+ "AI_API_URL": "https://api.ai.prod.eu-central-1.aws.ml.hana.ondemand.com"
+ }
+}
+```
+
+:::info Resource Group
+The resource group is typically configured separately in your AI Core deployment, not in the service key itself. You can set it via the `AICORE_RESOURCE_GROUP` environment variable (defaults to "default").
+:::
+
+## Quick Start
+
+### Step 1: Install LiteLLM
+
+```bash
+pip install litellm
+```
+
+### Step 2: Set Your Credentials
+
+Choose **one** of these authentication methods:
+
+
+
+
+The simplest approach - paste your entire service key as a single environment variable. The service key must be wrapped in a `credentials` object:
+
+```bash
+export AICORE_SERVICE_KEY='{
+ "credentials": {
+ "clientid": "your-client-id",
+ "clientsecret": "your-client-secret",
+ "url": "https://.authentication.sap.hana.ondemand.com",
+ "serviceurls": {
+ "AI_API_URL": "https://api.ai..aws.ml.hana.ondemand.com"
+ }
+ }
+}'
+export AICORE_RESOURCE_GROUP="default"
+```
+
+
+
+
+Alternatively, instead of using the service key above, you could set each credential separately:
+
+```bash
+export AICORE_AUTH_URL="https://.authentication.sap.hana.ondemand.com/oauth/token"
+export AICORE_CLIENT_ID="your-client-id"
+export AICORE_CLIENT_SECRET="your-client-secret"
+export AICORE_RESOURCE_GROUP="default"
+export AICORE_BASE_URL="https://api.ai..aws.ml.hana.ondemand.com/v2"
+```
+
+
+
+
+### Step 3: Make Your First Request
+
+```python title="test_sap.py"
+from litellm import completion
+
+response = completion(
+ model="sap/gpt-4o",
+ messages=[{"role": "user", "content": "Hello from LiteLLM!"}]
+)
+print(response.choices[0].message.content)
+```
+
+Run it:
+
+```bash
+python test_sap.py
+```
+
+**Expected output:**
+
+```text
+Hello! How can I assist you today?
+```
+
+### Step 4: Verify Your Setup (Optional)
+
+Test that everything is working with this diagnostic script:
+
+```python title="verify_sap_setup.py"
+import os
+import litellm
+
+# Enable debug logging to see what's happening
+import os
+os.environ["LITELLM_LOG"] = "DEBUG"
+
+# Either use AICORE_SERVICE_KEY (contains all credentials including resourcegroup)
+# OR use individual variables (all required together)
+individual_vars = ["AICORE_AUTH_URL", "AICORE_CLIENT_ID", "AICORE_CLIENT_SECRET", "AICORE_BASE_URL", "AICORE_RESOURCE_GROUP"]
+
+print("=== SAP Gen AI Hub Setup Verification ===\n")
+
+# Check for service key method
+if os.environ.get("AICORE_SERVICE_KEY"):
+ print("✓ Using AICORE_SERVICE_KEY authentication (includes resource group)")
+else:
+ # Check individual variables
+ missing = [v for v in individual_vars if not os.environ.get(v)]
+ if missing:
+ print(f"✗ Missing environment variables: {missing}")
+ else:
+ print("✓ Using individual variable authentication")
+ print(f"✓ Resource group: {os.environ.get('AICORE_RESOURCE_GROUP')}")
+
+# Test API connection
+print("\n=== Testing API Connection ===\n")
+try:
+ response = litellm.completion(
+ model="sap/gpt-4o",
+ messages=[{"role": "user", "content": "Say 'Connection successful!' and nothing else."}],
+ max_tokens=20
+ )
+ print(f"✓ API Response: {response.choices[0].message.content}")
+ print("\n🎉 Setup complete! You're ready to use SAP Gen AI Hub with LiteLLM.")
+except Exception as e:
+ print(f"✗ API Error: {e}")
+ print("\nTroubleshooting tips:")
+ print(" 1. Verify your service key credentials are correct")
+ print(" 2. Check that 'gpt-4o' is deployed in your resource group")
+ print(" 3. Ensure your SAP AI Core instance is running")
+```
+
+Run the verification:
+
+```bash
+python verify_sap_setup.py
+```
+
+**Expected output on success:**
+
+```text
+=== SAP Gen AI Hub Setup Verification ===
+
+✓ Using AICORE_SERVICE_KEY authentication
+✓ Resource group: default
+
+=== Testing API Connection ===
+
+✓ API Response: Connection successful!
+
+🎉 Setup complete! You're ready to use SAP Gen AI Hub with LiteLLM.
+```
+
## Authentication
-SAP Generative AI Hub uses service key authentication. You can provide credentials via:
+SAP Generative AI Hub uses OAuth2 service keys for authentication. See [Quick Start](#quick-start) for setup instructions.
-1. **Environment variable** - Set `AICORE_SERVICE_KEY` with your service key JSON
-2. **Direct parameter** - Pass `api_key` with the service key JSON string
+### Environment Variables Reference
-```python showLineNumbers title="Environment Variable"
-import os
-os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `AICORE_SERVICE_KEY` | Yes* | Complete service key JSON (recommended method) |
+| `AICORE_RESOURCE_GROUP` | Yes | Your AI Core resource group name |
+| `AICORE_AUTH_URL` | Yes* | OAuth token URL (alternative to service key) |
+| `AICORE_CLIENT_ID` | Yes* | OAuth client ID (alternative to service key) |
+| `AICORE_CLIENT_SECRET` | Yes* | OAuth client secret (alternative to service key) |
+| `AICORE_BASE_URL` | Yes* | AI Core API base URL (alternative to service key) |
+
+*Choose either `AICORE_SERVICE_KEY` OR the individual variables (`AICORE_AUTH_URL`, `AICORE_CLIENT_ID`, `AICORE_CLIENT_SECRET`, `AICORE_BASE_URL`).
+
+## Model Naming Conventions
+
+Understanding model naming is crucial for using SAP Gen AI Hub correctly. The naming pattern differs depending on whether you're using the SDK directly or through the proxy.
+
+### Direct SDK Usage
+
+When calling LiteLLM's SDK directly, you **must** include the `sap/` prefix in the model name:
+
+```python
+# Correct - includes sap/ prefix
+model="sap/gpt-4o"
+model="sap/anthropic--claude-4.5-sonnet"
+model="sap/gemini-2.5-pro"
+
+# Incorrect - missing prefix
+model="gpt-4o" # ❌ Won't work
```
-3. **Environment variables** - Set the following list of credentials in .env file
-
-AICORE_AUTH_URL = "https://* * * .authentication.sap.hana.ondemand.com/oauth/token",
-AICORE_CLIENT_ID = " *** ",
-AICORE_CLIENT_SECRET = " *** ",
-AICORE_RESOURCE_GROUP = " *** ",
-AICORE_BASE_URL = "https://api.ai.***.cfapps.sap.hana.ondemand.com/v2"
-
-## Usage - LiteLLM Python SDK
-```python showLineNumbers title="SAP Chat Completion"
-from litellm import completion
-import os
+### Proxy Usage
-os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
+When using the LiteLLM Proxy, you use the **friendly `model_name`** defined in your configuration. The proxy automatically handles the `sap/` prefix routing.
-response = completion(
- model="sap/gpt-4",
- messages=[{"role": "user", "content": "Hello from LiteLLM"}]
+```yaml
+# In config.yaml, define the mapping
+model_list:
+ - model_name: gpt-4o # ← Use this name in client requests
+ litellm_params:
+ model: sap/gpt-4o # ← Proxy handles the sap/ prefix
+```
+
+```python
+# Client request - no sap/ prefix needed
+client.chat.completions.create(
+ model="gpt-4o", # ✓ Correct for proxy usage
+ messages=[...]
)
-print(response)
```
-```python showLineNumbers title="SAP Chat Completion - Streaming"
+### Anthropic Models Special Syntax
+
+Anthropic models use a double-dash (`--`) prefix convention:
+
+| Provider | Model Example | LiteLLM Format |
+|----------|---------------|----------------|
+| OpenAI | GPT-4o | `sap/gpt-4o` |
+| Anthropic | Claude 4.5 Sonnet | `sap/anthropic--claude-4.5-sonnet` |
+| Google | Gemini 2.5 Pro | `sap/gemini-2.5-pro` |
+| Mistral | Mistral Large | `sap/mistral-large` |
+
+### Quick Reference Table
+
+| Usage Type | Model Format | Example |
+|------------|--------------|---------|
+| Direct SDK | `sap/` | `sap/gpt-4o` |
+| Direct SDK (Anthropic) | `sap/anthropic--` | `sap/anthropic--claude-4.5-sonnet` |
+| Proxy Client | `` | `gpt-4o` or `claude-sonnet` |
+
+## Using the Python SDK
+
+The LiteLLM Python SDK automatically detects your authentication method. Simply set your environment variables and make requests.
+
+```python showLineNumbers title="Basic Completion"
from litellm import completion
-import os
-
-os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
+# Assumes AICORE_AUTH_URL, AICORE_CLIENT_ID, etc. are set
response = completion(
- model="sap/gpt-4",
- messages=[{"role": "user", "content": "Hello from LiteLLM"}],
- stream=True
+ model="sap/anthropic--claude-4.5-sonnet",
+ messages=[{"role": "user", "content": "Explain quantum computing"}]
)
-
-for chunk in response:
- print(chunk.choices[0].delta.content or "", end="")
+print(response.choices[0].message.content)
```
-```python showLineNumbers title="SAP Embedding"
-from litellm import embedding
-import os
+Both authentication methods (individual variables or service key JSON) work automatically - no code changes required.
-os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
+## Using the Proxy Server
-result = embedding(
- model="sap/text-embedding-3-small",
- input="Answer to the ultimate question of life, the universe, and everything is 42")
-print(result.data[0])
-```
+The LiteLLM Proxy provides a unified OpenAI-compatible API for your SAP models.
-## Usage - LiteLLM Proxy
+### Configuration
-Add to your LiteLLM Proxy config:
+Create a `config.yaml` file in your project directory with your model mappings and credentials:
```yaml showLineNumbers title="config.yaml"
model_list:
- - model_name: "sap/*"
+ # OpenAI models
+ - model_name: gpt-5
litellm_params:
- model: "sap/*"
+ model: sap/gpt-5
-general_settings:
- master_key: your-proxy-api-key
+ # Anthropic models (note the double-dash)
+ - model_name: claude-sonnet
+ litellm_params:
+ model: sap/anthropic--claude-4.5-sonnet
+ - model_name: claude-opus
+ litellm_params:
+ model: sap/anthropic--claude-4.5-opus
+
+ # Embeddings
+ - model_name: text-embedding-3-small
+ litellm_params:
+ model: sap/text-embedding-3-small
+
+litellm_settings:
+ drop_params: true
+ set_verbose: false
+ request_timeout: 600
+ num_retries: 2
+ forward_client_headers_to_llm_api: ["anthropic-version"]
+
+general_settings:
+ master_key: "sk-1234" # Enter here your desired master key starting with 'sk-'.
+
+ # UI Admin is not required but helpful including the management of keys for your team(s). If you are using a database, these parameters are required:
+ database_url: "Enter you database URL."
+ UI_USERNAME: "Your desired UI admin account name"
+ UI_PASSWORD: "Your desired and strong pwd"
+
+# Authentication
environment_variables:
- AICORE_SERVICE_KEY: '{"clientid": "...", "clientsecret": "...", ...}'
+ AICORE_SERVICE_KEY: '{"credentials": {"clientid": "...", "clientsecret": "...", "url": "...", "serviceurls": {"AI_API_URL": "..."}}}'
+ AICORE_RESOURCE_GROUP: "default"
```
-Start the proxy:
+### Starting the Proxy
```bash showLineNumbers title="Start Proxy"
litellm --config config.yaml
```
+The proxy will start on `http://localhost:4000` by default.
+
+### Making Requests
+
```bash showLineNumbers title="Test Request"
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
- -H "Authorization: Bearer your-proxy-api-key" \
+ -H "Authorization: Bearer sk-1234" \
-d '{
- "model": "sap/gpt-4",
+ "model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
@@ -118,11 +358,11 @@ from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000",
- api_key="your-proxy-api-key"
+ api_key="sk-1234"
)
response = client.chat.completions.create(
- model="sap/gpt-4",
+ model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)
@@ -134,12 +374,14 @@ print(response.choices[0].message.content)
```python showLineNumbers title="LiteLLM SDK"
import os
import litellm
-os.environ["LITELLM_PROXY_API_KEY"] = "your-proxy-api-key"
-litellm.use_litellm_proxy = True # it is important to set this parameter
+
+os.environ["LITELLM_PROXY_API_KEY"] = "sk-1234"
+litellm.use_litellm_proxy = True
+
response = litellm.completion(
- model="sap/gpt-4o",
- messages=[{ "content": "Hello, how are you?","role": "user"}],
- api_base="http://your-proxy-api-base"
+ model="claude-sonnet",
+ messages=[{"content": "Hello, how are you?", "role": "user"}],
+ api_base="http://localhost:4000"
)
print(response)
@@ -148,15 +390,170 @@ print(response)
-## Supported Parameters
+## Features
-| Parameter | Description |
-|-----------|-------------|
-| `temperature` | Controls randomness |
-| `max_tokens` | Maximum tokens in response |
-| `top_p` | Nucleus sampling |
-| `tools` | Function calling tools |
-| `tool_choice` | Tool selection behavior |
-| `response_format` | Output format (json_object, json_schema) |
-| `stream` | Enable streaming |
+### Streaming Responses
+Stream responses in real-time for better user experience:
+
+```python showLineNumbers title="Streaming Chat Completion"
+from litellm import completion
+
+response = completion(
+ model="sap/gpt-4o",
+ messages=[{"role": "user", "content": "Count from 1 to 10"}],
+ stream=True
+)
+
+for chunk in response:
+ if chunk.choices[0].delta.content:
+ print(chunk.choices[0].delta.content, end="", flush=True)
+```
+
+### Structured Output
+
+#### JSON Schema (Recommended)
+
+Use JSON Schema for structured output with strict validation:
+
+```python showLineNumbers title="JSON Schema Response"
+from litellm import completion
+
+response = completion(
+ model="sap/gpt-4o",
+ messages=[{
+ "role": "user",
+ "content": "Generate info about Tokyo"
+ }],
+ response_format={
+ "type": "json_schema",
+ "json_schema": {
+ "name": "city_info",
+ "schema": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string"},
+ "population": {"type": "number"},
+ "country": {"type": "string"}
+ },
+ "required": ["name", "population", "country"],
+ "additionalProperties": False
+ },
+ "strict": True
+ }
+ }
+)
+
+print(response.choices[0].message.content)
+# Output: {"name":"Tokyo","population":37000000,"country":"Japan"}
+```
+
+#### JSON Object Format
+
+For flexible JSON output without schema validation:
+
+```python showLineNumbers title="JSON Object Response"
+from litellm import completion
+
+response = completion(
+ model="sap/gpt-4o",
+ messages=[{
+ "role": "user",
+ "content": "Generate a person object in JSON format with name and age"
+ }],
+ response_format={"type": "json_object"}
+)
+
+print(response.choices[0].message.content)
+```
+
+:::note SAP Platform Requirement
+When using `json_object` type, SAP's orchestration service requires the word "json" to appear in your prompt. This ensures explicit intent for JSON formatting. For schema-validated output without this requirement, use `json_schema` instead (recommended).
+:::
+
+### Multi-turn Conversations
+
+Maintain conversation context across multiple turns:
+
+```python showLineNumbers title="Multi-turn Conversation"
+from litellm import completion
+
+response = completion(
+ model="sap/gpt-4o",
+ messages=[
+ {"role": "user", "content": "My name is Alice"},
+ {"role": "assistant", "content": "Hello Alice! Nice to meet you."},
+ {"role": "user", "content": "What is my name?"}
+ ]
+)
+
+print(response.choices[0].message.content)
+# Output: Your name is Alice.
+```
+
+### Embeddings
+
+Generate vector embeddings for semantic search and retrieval:
+
+```python showLineNumbers title="Create Embeddings"
+from litellm import embedding
+
+response = embedding(
+ model="sap/text-embedding-3-small",
+ input=["Hello world", "Machine learning is fascinating"]
+)
+
+print(response.data[0]["embedding"]) # Vector representation
+```
+
+## Reference
+
+### Supported Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `model` | string | Model identifier (with `sap/` prefix for SDK) |
+| `messages` | array | Conversation messages |
+| `temperature` | float | Controls randomness (0-2) |
+| `max_tokens` | integer | Maximum tokens in response |
+| `top_p` | float | Nucleus sampling threshold |
+| `stream` | boolean | Enable streaming responses |
+| `response_format` | object | Output format (`json_object`, `json_schema`) |
+| `tools` | array | Function calling tool definitions |
+| `tool_choice` | string/object | Tool selection behavior |
+
+### Supported Models
+
+For the complete and up-to-date list of available models provided by SAP Gen AI Hub, please refer to the [SAP AI Core Generative AI Hub documentation](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/models-and-scenarios-in-generative-ai-hub).
+
+:::info Model Availability
+Model availability varies by SAP deployment region and your subscription. Contact your SAP administrator to confirm which models are available in your environment.
+:::
+
+### Troubleshooting
+
+**Authentication Errors**
+
+If you receive authentication errors:
+
+1. Verify all required environment variables are set correctly
+2. Check that your service key hasn't expired
+3. Confirm your resource group has access to the desired models
+4. Ensure the `AICORE_AUTH_URL` and `AICORE_BASE_URL` match your SAP region
+
+**Model Not Found**
+
+If a model returns "not found":
+
+1. Verify the model is available in your SAP deployment
+2. Check you're using the correct model name format (`sap/` prefix for SDK)
+3. Confirm your resource group has access to that specific model
+4. For Anthropic models, ensure you're using the `anthropic--` double-dash prefix
+
+**Rate Limiting**
+
+SAP Gen AI Hub enforces rate limits based on your subscription. If you hit limits:
+
+1. Implement exponential backoff retry logic
+2. Consider using the proxy's built-in rate limiting features
+3. Contact your SAP administrator to review quota allocations
diff --git a/docs/my-website/docs/providers/stability.md b/docs/my-website/docs/providers/stability.md
index 6b340267e69..62a8ab43cd8 100644
--- a/docs/my-website/docs/providers/stability.md
+++ b/docs/my-website/docs/providers/stability.md
@@ -416,7 +416,6 @@ response = image_edit(
image=open("original_image.png", "rb"),
mask=open("mask_image.png", "rb"),
prompt="Add flowers in the masked area",
- size="1024x1024",
)
print(response)
```
diff --git a/docs/my-website/docs/proxy/admin_ui_sso.md b/docs/my-website/docs/proxy/admin_ui_sso.md
index dba563a327b..7b299429db7 100644
--- a/docs/my-website/docs/proxy/admin_ui_sso.md
+++ b/docs/my-website/docs/proxy/admin_ui_sso.md
@@ -73,8 +73,21 @@ GOOGLE_CLIENT_SECRET=
```shell
MICROSOFT_CLIENT_ID="84583a4d-"
MICROSOFT_CLIENT_SECRET="nbk8Q~"
-MICROSOFT_TENANT="5a39737
+MICROSOFT_TENANT="5a39737"
```
+
+**Optional: Custom Microsoft SSO Endpoints**
+
+If you need to use custom Microsoft SSO endpoints (e.g., for a custom identity provider, sovereign cloud, or proxy), you can override the default endpoints:
+
+```shell
+MICROSOFT_AUTHORIZATION_ENDPOINT="https://your-custom-url.com/oauth2/v2.0/authorize"
+MICROSOFT_TOKEN_ENDPOINT="https://your-custom-url.com/oauth2/v2.0/token"
+MICROSOFT_USERINFO_ENDPOINT="https://your-custom-graph-api.com/v1.0/me"
+```
+
+If these are not set, the default Microsoft endpoints are used based on your tenant.
+
- Set Redirect URI on your App Registration on https://portal.azure.com/
- Set a redirect url = `/sso/callback`
```shell
@@ -98,6 +111,42 @@ To set up app roles:
4. Assign users to these roles in your Enterprise Application
5. When users sign in via SSO, LiteLLM will automatically assign them the corresponding role
+**Advanced: Custom User Attribute Mapping**
+
+For certain Microsoft Entra ID configurations, you may need to override the default user attribute field names. This is useful when your organization uses custom claims or non-standard attribute names in the SSO response.
+
+**Step 1: Debug SSO Response**
+
+First, inspect the JWT fields returned by your Microsoft SSO provider using the [SSO Debug Route](#debugging-sso-jwt-fields).
+
+1. Add `/sso/debug/callback` as a redirect URL in your Azure App Registration
+2. Navigate to `https:///sso/debug/login`
+3. Complete the SSO flow to see the returned user attributes
+
+**Step 2: Identify Field Attribute Names**
+
+From the debug response, identify the field names used for email, display name, user ID, first name, and last name.
+
+**Step 3: Set Environment Variables**
+
+Override the default attribute names by setting these environment variables:
+
+| Environment Variable | Description | Default Value |
+|---------------------|-------------|---------------|
+| `MICROSOFT_USER_EMAIL_ATTRIBUTE` | Field name for user email | `userPrincipalName` |
+| `MICROSOFT_USER_DISPLAY_NAME_ATTRIBUTE` | Field name for display name | `displayName` |
+| `MICROSOFT_USER_ID_ATTRIBUTE` | Field name for user ID | `id` |
+| `MICROSOFT_USER_FIRST_NAME_ATTRIBUTE` | Field name for first name | `givenName` |
+| `MICROSOFT_USER_LAST_NAME_ATTRIBUTE` | Field name for last name | `surname` |
+
+**Step 4: Restart the Proxy**
+
+After setting the environment variables, restart the proxy:
+
+```bash
+litellm --config /path/to/config.yaml
+```
+
diff --git a/docs/my-website/docs/proxy/caching.md b/docs/my-website/docs/proxy/caching.md
index 87e6a6fdb6e..3cb9e9f3fe4 100644
--- a/docs/my-website/docs/proxy/caching.md
+++ b/docs/my-website/docs/proxy/caching.md
@@ -282,6 +282,10 @@ Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable cac
```
**Additional kwargs**
+:::info
+Use `REDIS_*` environment variables to configure all Redis client library parameters. This is the suggested mechanism for toggling Redis settings as it automatically maps environment variables to Redis client kwargs.
+:::
+
You can pass in any additional redis.Redis arg, by storing the variable + value in your os
environment, like this:
@@ -289,6 +293,17 @@ environment, like this:
REDIS_ = ""
```
+For example:
+```shell
+REDIS_SSL = "True"
+REDIS_SSL_CERT_REQS = "None"
+REDIS_CONNECTION_POOL_KWARGS = '{"max_connections": 20}'
+```
+
+:::warning
+**Note**: For non-string Redis parameters (like integers, booleans, or complex objects), avoid using `REDIS_*` environment variables as they may fail during Redis client initialization. Instead, use `cache_kwargs` in your router configuration for such parameters.
+:::
+
[**See how it's read from the environment**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/_redis.py#L40)
#### Step 3: Run proxy with config
diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md
index 42494c1e5f5..b941f21b33e 100644
--- a/docs/my-website/docs/proxy/config_settings.md
+++ b/docs/my-website/docs/proxy/config_settings.md
@@ -339,7 +339,7 @@ router_settings:
| stream_timeout | Optional[float] | The default timeout for a streaming request. If not set, the 'timeout' value is used. |
| debug_level | Literal["DEBUG", "INFO"] | The debug level for the logging library in the router. Defaults to "INFO". |
| client_ttl | int | Time-to-live for cached clients in seconds. Defaults to 3600. |
-| cache_kwargs | dict | Additional keyword arguments for the cache initialization. |
+| cache_kwargs | dict | Additional keyword arguments for the cache initialization. Use this for non-string Redis parameters that may fail when set via `REDIS_*` environment variables. |
| routing_strategy_args | dict | Additional keyword arguments for the routing strategy - e.g. lowest latency routing default ttl |
| model_group_alias | dict | Model group alias mapping. E.g. `{"claude-3-haiku": "claude-3-haiku-20240229"}` |
| num_retries | int | Number of retries for a request. Defaults to 3. |
@@ -744,6 +744,7 @@ router_settings:
| LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD | If true, prints the standard logging payload to the console - useful for debugging
| LITELM_ENVIRONMENT | Environment for LiteLLM Instance. This is currently only logged to DeepEval to determine the environment for DeepEval integration.
| LOGFIRE_TOKEN | Token for Logfire logging service
+| LOGFIRE_BASE_URL | Base URL for Logfire logging service (useful for self hosted deployments)
| LOGGING_WORKER_CONCURRENCY | Maximum number of concurrent coroutine slots for the logging worker on the asyncio event loop. Default is 100. Setting too high will flood the event loop with logging tasks which will lower the overall latency of the requests.
| LOGGING_WORKER_MAX_QUEUE_SIZE | Maximum size of the logging worker queue. When the queue is full, the worker aggressively clears tasks to make room instead of dropping logs. Default is 50,000
| LOGGING_WORKER_MAX_TIME_PER_COROUTINE | Maximum time in seconds allowed for each coroutine in the logging worker before timing out. Default is 20.0
@@ -754,6 +755,7 @@ router_settings:
| LOGGING_WORKER_AGGRESSIVE_CLEAR_COOLDOWN_SECONDS | Cooldown time in seconds before allowing another aggressive clear operation when the queue is full. Default is 0.5
| MAX_STRING_LENGTH_PROMPT_IN_DB | Maximum length for strings in spend logs when sanitizing request bodies. Strings longer than this will be truncated. Default is 1000
| MAX_IN_MEMORY_QUEUE_FLUSH_COUNT | Maximum count for in-memory queue flush operations. Default is 1000
+| MAX_IMAGE_URL_DOWNLOAD_SIZE_MB | Maximum size in MB for downloading images from URLs. Prevents memory issues from downloading very large images. Images exceeding this limit will be rejected before download. Set to 0 to completely disable image URL handling (all image_url requests will be blocked). Default is 50MB (matching [OpenAI's limit](https://platform.openai.com/docs/guides/images-vision?api-mode=chat#image-input-requirements))
| MAX_LONG_SIDE_FOR_IMAGE_HIGH_RES | Maximum length for the long side of high-resolution images. Default is 2000
| MAX_REDIS_BUFFER_DEQUEUE_COUNT | Maximum count for Redis buffer dequeue operations. Default is 100
| MAX_SHORT_SIDE_FOR_IMAGE_HIGH_RES | Maximum length for the short side of high-resolution images. Default is 768
@@ -771,10 +773,18 @@ router_settings:
| MINIMUM_PROMPT_CACHE_TOKEN_COUNT | Minimum token count for caching a prompt. Default is 1024
| MISTRAL_API_BASE | Base URL for Mistral API. Default is https://api.mistral.ai
| MISTRAL_API_KEY | API key for Mistral API
+| MICROSOFT_AUTHORIZATION_ENDPOINT | Custom authorization endpoint URL for Microsoft SSO (overrides default Microsoft OAuth authorization endpoint)
| MICROSOFT_CLIENT_ID | Client ID for Microsoft services
| MICROSOFT_CLIENT_SECRET | Client secret for Microsoft services
-| MICROSOFT_TENANT | Tenant ID for Microsoft Azure
| MICROSOFT_SERVICE_PRINCIPAL_ID | Service Principal ID for Microsoft Enterprise Application. (This is an advanced feature if you want litellm to auto-assign members to Litellm Teams based on their Microsoft Entra ID Groups)
+| MICROSOFT_TENANT | Tenant ID for Microsoft Azure
+| MICROSOFT_TOKEN_ENDPOINT | Custom token endpoint URL for Microsoft SSO (overrides default Microsoft OAuth token endpoint)
+| MICROSOFT_USER_DISPLAY_NAME_ATTRIBUTE | Field name for user display name in Microsoft SSO response. Default is `displayName`
+| MICROSOFT_USER_EMAIL_ATTRIBUTE | Field name for user email in Microsoft SSO response. Default is `userPrincipalName`
+| MICROSOFT_USER_FIRST_NAME_ATTRIBUTE | Field name for user first name in Microsoft SSO response. Default is `givenName`
+| MICROSOFT_USER_ID_ATTRIBUTE | Field name for user ID in Microsoft SSO response. Default is `id`
+| MICROSOFT_USER_LAST_NAME_ATTRIBUTE | Field name for user last name in Microsoft SSO response. Default is `surname`
+| MICROSOFT_USERINFO_ENDPOINT | Custom userinfo endpoint URL for Microsoft SSO (overrides default Microsoft Graph userinfo endpoint)
| NO_DOCS | Flag to disable Swagger UI documentation
| NO_REDOC | Flag to disable Redoc documentation
| NO_PROXY | List of addresses to bypass proxy
diff --git a/docs/my-website/docs/proxy/customer_usage.md b/docs/my-website/docs/proxy/customer_usage.md
index 8e366586b15..5a6c06fdc81 100644
--- a/docs/my-website/docs/proxy/customer_usage.md
+++ b/docs/my-website/docs/proxy/customer_usage.md
@@ -22,19 +22,22 @@ Customer Usage enables you to track spend and usage for individual customers (en
## How to Track Spend
-Track customer spend by including a `user` field in your API requests. The customer ID will be automatically tracked and associated with all spend from that request.
+Track customer spend by including a `user` field in your API requests or by passing a customer ID header. The customer ID will be automatically tracked and associated with all spend from that request.
-### Example using cURL
+
+
+
+### Using Request Body
Make a `/chat/completions` call with the `user` field containing your customer ID:
-```bash showLineNumbers title="Track spend with customer ID"
+```bash showLineNumbers title="Track spend with customer ID in body"
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
- --header 'Authorization: Bearer sk-1234' \ # 👈 YOUR PROXY KEY
+ --header 'Authorization: Bearer sk-1234' \
--data '{
"model": "gpt-3.5-turbo",
- "user": "customer-123", # 👈 CUSTOMER ID
+ "user": "customer-123",
"messages": [
{
"role": "user",
@@ -44,7 +47,49 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
}'
```
-The customer ID (`customer-123`) will be automatically upserted into the database with the new spend. If the customer ID already exists, spend will be incremented.
+
+
+
+### Using Request Headers
+
+You can also pass the customer ID via HTTP headers. This is useful for tools that support custom headers but don't allow modifying the request body (like Claude Code with `ANTHROPIC_CUSTOM_HEADERS`).
+
+LiteLLM automatically recognizes these standard headers (no configuration required):
+- `x-litellm-customer-id`
+- `x-litellm-end-user-id`
+
+```bash showLineNumbers title="Track spend with customer ID in header"
+curl -X POST 'http://0.0.0.0:4000/chat/completions' \
+ --header 'Content-Type: application/json' \
+ --header 'Authorization: Bearer sk-1234' \
+ --header 'x-litellm-customer-id: customer-123' \
+ --data '{
+ "model": "gpt-3.5-turbo",
+ "messages": [
+ {
+ "role": "user",
+ "content": "What is the capital of France?"
+ }
+ ]
+ }'
+```
+
+#### Using with Claude Code
+
+Claude Code supports custom headers via the `ANTHROPIC_CUSTOM_HEADERS` environment variable. Set it to pass your customer ID:
+
+```bash title="Configure Claude Code with customer tracking"
+export ANTHROPIC_BASE_URL="http://0.0.0.0:4000/v1/messages"
+export ANTHROPIC_API_KEY="sk-1234"
+export ANTHROPIC_CUSTOM_HEADERS="x-litellm-customer-id: my-customer-id"
+```
+
+Now all requests from Claude Code will automatically track spend under `my-customer-id`.
+
+
+
+
+The customer ID will be automatically upserted into the database with the new spend. If the customer ID already exists, spend will be incremented.
### Example using OpenWebUI
diff --git a/docs/my-website/docs/proxy/deleted_keys_teams.md b/docs/my-website/docs/proxy/deleted_keys_teams.md
new file mode 100644
index 00000000000..a4736ed5ed2
--- /dev/null
+++ b/docs/my-website/docs/proxy/deleted_keys_teams.md
@@ -0,0 +1,106 @@
+import Image from '@theme/IdealImage';
+
+# Deleted Keys & Teams Audit Logs
+
+
+
+View deleted API keys and teams along with their spend and budget information at the time of deletion for auditing and compliance purposes.
+
+## Overview
+
+The Deleted Keys & Teams feature provides a comprehensive audit trail for deleted entities in your LiteLLM proxy. This feature was implemented to easily allow audits of which key or team was deleted along with the spend/budget at the time of deletion.
+
+When a key or team is deleted, LiteLLM automatically captures:
+
+- **Deletion timestamp** - When the entity was deleted
+- **Deleted by** - Who performed the deletion action
+- **Spend at deletion** - The total spend accumulated at the time of deletion
+- **Original budget** - The budget that was set for the entity before deletion
+- **Entity details** - Key or team identification information
+
+This information is preserved even after deletion, allowing you to maintain accurate financial records and audit trails for compliance purposes.
+
+## Viewing Deleted Keys
+
+### Step 1: Navigate to API Keys Page
+
+Navigate to the API Keys page in the LiteLLM UI:
+
+```
+http://localhost:4000/ui/?login=success&page=api-keys
+```
+
+
+
+### Step 2: Access Logs Section
+
+Click on the "Logs" menu item in the navigation.
+
+
+
+### Step 3: View Deleted Keys
+
+Click on "Deleted Keys" to view the table of all deleted API keys.
+
+
+
+### Step 4: Review Deletion Information
+
+The Deleted Keys table includes comprehensive information about each deleted key:
+
+- **When** the key was deleted (timestamp)
+- **Who** deleted the key (user/admin information)
+- **Key identification** details
+
+
+
+### Step 5: View Financial Information
+
+The table also displays financial information captured at the time of deletion:
+
+- **Spend at deletion** - Total spend accumulated when the key was deleted
+- **Original budget** - The budget limit that was set for the key
+
+
+
+## Viewing Deleted Teams
+
+### Step 1: Access Deleted Teams
+
+From the Logs section, click on "Deleted Teams" to view all deleted teams.
+
+
+
+### Step 2: Review Team Deletion Information
+
+The Deleted Teams table provides detailed information about each deleted team:
+
+- **When** the team was deleted (timestamp)
+- **Who** deleted the team (user/admin information)
+- **Team identification** details
+
+
+
+### Step 3: View Team Financial Information
+
+Similar to deleted keys, the Deleted Teams table shows financial information:
+
+- **Spend at deletion** - Total spend accumulated when the team was deleted
+- **Original budget** - The budget limit that was set for the team
+
+
+
+## Use Cases
+
+This feature is particularly useful for:
+
+- **Financial Auditing** - Track spend and budgets for deleted entities
+- **Compliance** - Maintain records of who deleted what and when
+- **Cost Analysis** - Understand spending patterns before deletion
+- **Accountability** - Identify which admin or user performed deletions
+- **Historical Records** - Preserve financial data even after entity deletion
+
+## Related Features
+
+- [Audit Logs](./multiple_admins.md) - View comprehensive audit logs for all entity changes
+- [UI Logs](./ui_logs.md) - View request logs and spend tracking
diff --git a/docs/my-website/docs/proxy/fallback_management.md b/docs/my-website/docs/proxy/fallback_management.md
new file mode 100644
index 00000000000..9e565fee133
--- /dev/null
+++ b/docs/my-website/docs/proxy/fallback_management.md
@@ -0,0 +1,267 @@
+# [New] Fallback Management Endpoints
+
+Dedicated endpoints for managing model fallbacks separately from the general configuration.
+
+## Overview
+
+These endpoints allow you to configure, retrieve, and delete fallback models without modifying the entire proxy configuration. This provides a cleaner and safer way to manage fallbacks compared to using the `/config/update` endpoint.
+
+## Prerequisites
+
+- Database storage must be enabled: Set `STORE_MODEL_IN_DB=True` in your environment
+- Models must exist in the router before configuring fallbacks
+
+## Endpoints
+
+### POST /fallback
+
+Create or update fallbacks for a specific model.
+
+**Request Body:**
+```json
+{
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4", "claude-3-haiku"],
+ "fallback_type": "general"
+}
+```
+
+**Parameters:**
+- `model` (string, required): The primary model name to configure fallbacks for
+- `fallback_models` (array of strings, required): List of fallback model names in priority order
+- `fallback_type` (string, optional): Type of fallback. Options:
+ - `"general"` (default): Standard fallbacks for any error
+ - `"context_window"`: Fallbacks for context window exceeded errors
+ - `"content_policy"`: Fallbacks for content policy violations
+
+**Response:**
+```json
+{
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4", "claude-3-haiku"],
+ "fallback_type": "general",
+ "message": "Fallback configuration created successfully"
+}
+```
+
+**Example using cURL:**
+```bash
+curl -X POST "http://localhost:4000/fallback" \
+ -H "Authorization: Bearer sk-1234" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4", "claude-3-haiku"],
+ "fallback_type": "general"
+ }'
+```
+
+**Example using Python:**
+```python
+import requests
+
+response = requests.post(
+ "http://localhost:4000/fallback",
+ headers={
+ "Authorization": "Bearer sk-1234",
+ "Content-Type": "application/json"
+ },
+ json={
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4", "claude-3-haiku"],
+ "fallback_type": "general"
+ }
+)
+
+print(response.json())
+```
+
+### GET /fallback/\{model\}
+
+Get fallback configuration for a specific model.
+
+**Parameters:**
+- `model` (path parameter, required): The model name to get fallbacks for
+- `fallback_type` (query parameter, optional): Type of fallback to retrieve (default: "general")
+
+**Response:**
+```json
+{
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4", "claude-3-haiku"],
+ "fallback_type": "general"
+}
+```
+
+**Example using cURL:**
+```bash
+curl -X GET "http://localhost:4000/fallback/gpt-3.5-turbo?fallback_type=general" \
+ -H "Authorization: Bearer sk-1234"
+```
+
+**Example using Python:**
+```python
+import requests
+
+response = requests.get(
+ "http://localhost:4000/fallback/gpt-3.5-turbo",
+ headers={"Authorization": "Bearer sk-1234"},
+ params={"fallback_type": "general"}
+)
+
+print(response.json())
+```
+
+### DELETE /fallback/\{model\}
+
+Delete fallback configuration for a specific model.
+
+**Parameters:**
+- `model` (path parameter, required): The model name to delete fallbacks for
+- `fallback_type` (query parameter, optional): Type of fallback to delete (default: "general")
+
+**Response:**
+```json
+{
+ "model": "gpt-3.5-turbo",
+ "fallback_type": "general",
+ "message": "Fallback configuration deleted successfully"
+}
+```
+
+**Example using cURL:**
+```bash
+curl -X DELETE "http://localhost:4000/fallback/gpt-3.5-turbo?fallback_type=general" \
+ -H "Authorization: Bearer sk-1234"
+```
+
+**Example using Python:**
+```python
+import requests
+
+response = requests.delete(
+ "http://localhost:4000/fallback/gpt-3.5-turbo",
+ headers={"Authorization": "Bearer sk-1234"},
+ params={"fallback_type": "general"}
+)
+
+print(response.json())
+```
+
+### Test fallback
+
+```bash
+curl -X POST 'http://0.0.0.0:4000/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+ "model": "gpt-3.5-turbo",
+ "messages": [
+ {
+ "role": "user",
+ "content": "ping"
+ }
+ ],
+ "mock_testing_fallbacks": true
+}
+'
+```
+
+
+
+## Validation
+
+The endpoints perform the following validations:
+
+1. **Model Existence**: Verifies that the primary model exists in the router
+2. **Fallback Model Existence**: Ensures all fallback models exist in the router
+3. **No Self-Fallback**: Prevents a model from being its own fallback
+4. **No Duplicates**: Ensures no duplicate models in the fallback list
+5. **Database Enabled**: Requires `STORE_MODEL_IN_DB=True` to be set
+
+## Error Responses
+
+### 400 Bad Request
+```json
+{
+ "detail": {
+ "error": "Invalid fallback models: ['non-existent-model']",
+ "available_models": ["gpt-3.5-turbo", "gpt-4", "claude-3-haiku"]
+ }
+}
+```
+
+### 404 Not Found
+```json
+{
+ "detail": {
+ "error": "Model 'gpt-3.5-turbo' not found in router",
+ "available_models": ["gpt-4", "claude-3-haiku"]
+ }
+}
+```
+
+### 500 Internal Server Error
+```json
+{
+ "detail": {
+ "error": "Router not initialized"
+ }
+}
+```
+
+## Fallback Types Explained
+
+### General Fallbacks
+Used for any type of error that occurs during model invocation. This is the most common type of fallback.
+
+**Use Case:** When a model is unavailable, rate-limited, or returns an error.
+
+```json
+{
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4", "claude-3-haiku"],
+ "fallback_type": "general"
+}
+```
+
+### Context Window Fallbacks
+Specifically triggered when a context window exceeded error occurs.
+
+**Use Case:** When the input is too long for the primary model, fallback to a model with a larger context window.
+
+```json
+{
+ "model": "gpt-3.5-turbo",
+ "fallback_models": ["gpt-4-32k", "claude-3-opus"],
+ "fallback_type": "context_window"
+}
+```
+
+### Content Policy Fallbacks
+Specifically triggered when content policy violations occur.
+
+**Use Case:** When the primary model rejects content due to safety filters, fallback to a model with different content policies.
+
+```json
+{
+ "model": "gpt-4",
+ "fallback_models": ["claude-3-haiku"],
+ "fallback_type": "content_policy"
+}
+```
+
+## Benefits Over /config/update
+
+1. **Safety**: Only modifies fallback configuration, won't accidentally change other settings
+2. **Simplicity**: Focused API with clear validation messages
+3. **Granularity**: Manage fallbacks per model and per type
+4. **Validation**: Comprehensive checks ensure configuration is valid before applying
+5. **Clarity**: Clear error messages with available models listed
+
+## Notes
+
+- Fallbacks are triggered after the configured number of retries fails
+- Fallbacks are attempted in the order specified in `fallback_models`
+- The maximum number of fallbacks attempted is controlled by the router's `max_fallbacks` setting
+- Changes take effect immediately and are persisted to the database
diff --git a/docs/my-website/docs/proxy/guardrails/panw_prisma_airs.md b/docs/my-website/docs/proxy/guardrails/panw_prisma_airs.md
index 53f8a03f5bb..e3273a01c17 100644
--- a/docs/my-website/docs/proxy/guardrails/panw_prisma_airs.md
+++ b/docs/my-website/docs/proxy/guardrails/panw_prisma_airs.md
@@ -206,6 +206,7 @@ Expected successful response:
| `mode` | No | When to run the guardrail | `pre_call` |
| `fallback_on_error` | No | Action when PANW API is unavailable: `"block"` (fail-closed, default) or `"allow"` (fail-open). Config errors always block. | `block` |
| `timeout` | No | PANW API call timeout in seconds (1-60) | `10.0` |
+| `violation_message_template` | No | Custom template for error message when request is blocked. Supports `{guardrail_name}`, `{category}`, `{action_type}`, `{default_message}` placeholders. | - |
### Regional Endpoints
@@ -449,6 +450,33 @@ LiteLLM does not alter or configure your PANW security profile. To change what c
The guardrail is **fail-closed** by default - if the PANW API is unavailable, requests are blocked to ensure no unscanned content reaches your LLM. This provides maximum security.
:::
+### Custom Violation Messages
+
+You can customize the error message returned to the user when a request is blocked by configuring the `violation_message_template` parameter. This is useful for providing user-friendly feedback instead of technical details.
+
+```yaml
+guardrails:
+ - guardrail_name: "panw-custom-message"
+ litellm_params:
+ guardrail: panw_prisma_airs
+ api_key: os.environ/PANW_PRISMA_AIRS_API_KEY
+ # Simple message
+ violation_message_template: "Your request was blocked by our AI Security Policy."
+
+ - guardrail_name: "panw-detailed-message"
+ litellm_params:
+ guardrail: panw_prisma_airs
+ api_key: os.environ/PANW_PRISMA_AIRS_API_KEY
+ # Message with placeholders
+ violation_message_template: "{action_type} blocked due to {category} violation. Please contact support."
+```
+
+**Supported Placeholders:**
+- `{guardrail_name}`: Name of the guardrail (e.g. "panw-custom-message")
+- `{category}`: Violation category (e.g. "malicious", "injection", "dlp")
+- `{action_type}`: "Prompt" or "Response"
+- `{default_message}`: The original technical error message
+
### Fail-Open Configuration
By default, the PANW guardrail operates in **fail-closed** mode for maximum security. If the PANW API is unavailable (timeout, rate limit, network error), requests are blocked. You can configure **fail-open** mode for high-availability scenarios where service continuity is critical.
diff --git a/docs/my-website/docs/proxy/logging.md b/docs/my-website/docs/proxy/logging.md
index a27b6dcf083..80474a55afe 100644
--- a/docs/my-website/docs/proxy/logging.md
+++ b/docs/my-website/docs/proxy/logging.md
@@ -1827,6 +1827,64 @@ This approach allows you to:
- Share callbacks across different environments
- Version control callback files in cloud storage
+#### Step 2c - Mounting Custom Callbacks in Helm/Kubernetes (Alternative)
+
+When deploying with Helm or Kubernetes, you can mount custom callback Python files alongside your `config.yaml` using `subPath` to avoid overwriting the config directory.
+
+**The Problem:**
+Mounting a volume to a directory (e.g., `/app/`) would normally hide all existing files in that directory, including your `config.yaml`.
+
+**The Solution:**
+Use `subPath` in your `volumeMounts` to mount individual files without overwriting the entire directory.
+
+**Example - Helm values.yaml:**
+
+```yaml
+# values.yaml
+volumes:
+ - name: callback-files
+ configMap:
+ name: litellm-callback-files
+
+volumeMounts:
+ - name: callback-files
+ mountPath: /app/custom_callbacks.py # Mount to specific FILE path
+ subPath: custom_callbacks.py # Required to avoid overwriting directory
+```
+
+**Create the ConfigMap with your callback file:**
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: litellm-callback-files
+data:
+ custom_callbacks.py: |
+ from litellm.integrations.custom_logger import CustomLogger
+
+ class MyCustomHandler(CustomLogger):
+ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
+ print(f"Success! Model: {kwargs.get('model')}")
+
+ proxy_handler_instance = MyCustomHandler()
+```
+
+**Reference in your config.yaml:**
+
+```yaml
+litellm_settings:
+ callbacks: custom_callbacks.proxy_handler_instance
+```
+
+**How it works:**
+1. The `subPath` parameter tells Kubernetes to mount only the specific file
+2. This places `custom_callbacks.py` in `/app/` alongside your existing `config.yaml`
+3. LiteLLM automatically finds the callback file in the same directory as the config
+4. No files are overwritten or hidden
+
+**Note:** You can mount multiple callback files by adding more `volumeMounts` entries, each with its own `subPath`.
+
#### Step 3 - Start proxy + test request
```shell
diff --git a/docs/my-website/docs/proxy/spend_logs_deletion.md b/docs/my-website/docs/proxy/spend_logs_deletion.md
index 05627c07741..b021457173f 100644
--- a/docs/my-website/docs/proxy/spend_logs_deletion.md
+++ b/docs/my-website/docs/proxy/spend_logs_deletion.md
@@ -30,6 +30,9 @@ general_settings:
# Optional: set how frequently cleanup should run - default is daily
maximum_spend_logs_retention_interval: "1d" # Run cleanup daily
+ # Optional: set exact time for cleanup (Cron syntax)
+ maximum_spend_logs_cleanup_cron: "0 4 * * *" # Run at 04:00 AM daily
+
litellm_settings:
cache: true
cache_params:
@@ -51,6 +54,15 @@ How long logs should be kept before deletion. Supported formats:
How often the cleanup job should run. Uses the same format as above. If not set, cleanup will run every 24 hours if and only if `maximum_spend_logs_retention_period` is set.
+#### `maximum_spend_logs_cleanup_cron` (optional)
+
+Schedule the cleanup using standard cron syntax. This takes precedence over `maximum_spend_logs_retention_interval`.
+
+Examples:
+- `"0 4 * * *"` – Run at 04:00 AM daily
+- `"0 0 * * 0"` – Run at midnight every Sunday
+- `"*/30 * * * *"` – Run every 30 minutes
+
## How it works
### Step 1. Lock Acquisition (Optional with Redis)
diff --git a/docs/my-website/docs/routing.md b/docs/my-website/docs/routing.md
index 8ac56463d5e..47967775e1e 100644
--- a/docs/my-website/docs/routing.md
+++ b/docs/my-website/docs/routing.md
@@ -1333,6 +1333,10 @@ router = Router(model_list: Optional[list] = None,
cache_responses=True)
```
+:::info
+When configuring Redis caching in router settings, use `cache_kwargs` to pass additional Redis parameters, especially for non-string values that may fail when set via `REDIS_*` environment variables.
+:::
+
## Pre-Call Checks (Context Window, EU-Regions)
Enable pre-call checks to filter out:
diff --git a/docs/my-website/docs/troubleshoot.md b/docs/my-website/docs/troubleshoot.md
index 9aa9985e07b..f9ed47972eb 100644
--- a/docs/my-website/docs/troubleshoot.md
+++ b/docs/my-website/docs/troubleshoot.md
@@ -1,12 +1,60 @@
-# Support & Talk with founders
+# Troubleshooting & Support
+
+## Information to Provide When Seeking Help
+
+When reporting issues, please include as much of the following as possible. It's okay if you can't provide everything—especially in production scenarios where the trigger might be unknown. Sharing most of this information will help us assist you more effectively.
+
+### 1. LiteLLM Configuration File
+
+Your `config.yaml` file (redact sensitive info like API keys). Include number of workers if not in config.
+
+### 2. Initialization Command
+
+The command used to start LiteLLM (e.g., `litellm --config config.yaml --num_workers 8 --detailed_debug`).
+
+### 3. LiteLLM Version
+
+- Current version
+- Version when the issue first appeared (if different)
+- If upgraded, the version changed from → to
+
+### 4. Environment Variables
+
+Non-sensitive environment variables not in your config (e.g., `NUM_WORKERS`, `LITELLM_LOG`, `LITELLM_MODE`). Do not include passwords or API keys.
+
+### 5. Server Specifications
+
+CPU cores, RAM, OS, number of instances/replicas, etc.
+
+### 6. Database and Redis Usage
+
+- **Database:** Using database? (`DATABASE_URL` set), database type and version
+- **Redis:** Using Redis? Redis version, configuration type (Standalone/Cluster/Sentinel).
+
+### 7. Endpoints
+
+The endpoint(s) you're using that are experiencing issues (e.g., `/chat/completions`, `/embeddings`).
+
+### 8. Request Example
+
+A realistic example of the request causing issues, including expected vs. actual response and any error messages.
+
+### 9. Error Logs, Stack Traces, and Metrics
+
+Full error logs, stack traces, and any images from service metrics (CPU, memory, request rates, etc.) that might help diagnose the issue.
+
+---
+
+## Support Channels
+
[Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
[Community Discord 💭](https://discord.gg/wuPM9dRgDw)
[Community Slack 💭](https://www.litellm.ai/support)
-Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
+Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
-[](https://wa.link/huol9n) [](https://discord.gg/wuPM9dRgDw)
+[](https://wa.link/huol9n) [](https://discord.gg/wuPM9dRgDw)
diff --git a/docs/my-website/docs/troubleshoot/cpu_issues.md b/docs/my-website/docs/troubleshoot/cpu_issues.md
new file mode 100644
index 00000000000..8a9a8abe929
--- /dev/null
+++ b/docs/my-website/docs/troubleshoot/cpu_issues.md
@@ -0,0 +1,31 @@
+# CPU Issue Classification & Reproduction
+
+## 1. Classify the CPU Issue
+
+Select the options that best describes the CPU behavior observed.
+
+- [ ] CPU scales with traffic (RPS-driven)
+- [ ] CPU increases without a traffic increase
+- [ ] CPU increases after a LiteLLM upgrade
+
+## 2. Can you reproduce the issue?
+
+Before escalating, verify whether the CPU issue can be reproduced in a test environment that mirrors your production setup.
+
+If reproducible, provide **detailed reproduction steps** along with any relevant requests or configuration used.
+For guidance on the type of information we're looking for, see the [LiteLLM Troubleshooting Guide](../troubleshoot).
+
+## 3. Issue Cannot Be Reproduced
+
+If the CPU issue cannot be reproduced in a test environment that mirrors your production setup, please provide:
+
+1. **Information from Section 1 and 2**
+ - CPU classification (Section 1)
+ - Reproduction attempts and environment details (Section 2)
+
+2. **Additional context** to help investigate:
+ - **Workload:** A realistic sample of requests processed before and during the spike, including any recent configuration changes.
+ - **Metrics:** CPU usage, P50/P99 latency, memory usage. Please include **screenshots** of the metrics whenever possible.
+ - **Logs / Alerts:** Any relevant logs or alerts captured **before and during the spike**.
+
+> Providing this information allows the team to analyze patterns, correlate spikes with traffic or configuration, and attempt to reproduce the issue internally. Without it, our engineers won't have enough information to look into the problem.
diff --git a/docs/my-website/docs/troubleshoot/memory_issues.md b/docs/my-website/docs/troubleshoot/memory_issues.md
new file mode 100644
index 00000000000..1a3eb53f1c8
--- /dev/null
+++ b/docs/my-website/docs/troubleshoot/memory_issues.md
@@ -0,0 +1,37 @@
+# Memory Issue Classification & Reproduction
+
+## 1. Classify the Memory Issue
+
+Select the option(s) that best describe the memory behavior observed:
+
+- [ ] Memory scales with traffic (RPS-driven)
+- [ ] Memory increases without a traffic increase
+- [ ] Memory increases after a LiteLLM upgrade
+- [ ] Memory leak (memory continuously grows over time)
+- [ ] Out of Memory (OOM) events or pod restarts
+
+---
+
+## 2. Can you reproduce the issue?
+
+Before escalating, verify whether the memory or OOM issue can be reproduced in a test environment that mirrors your production deployment.
+
+If reproducible, provide **detailed reproduction steps** along with any relevant requests, workloads, or configuration used.
+For guidance on the type of information we’re looking for, see the [LiteLLM Troubleshooting Guide](../troubleshoot).
+
+---
+
+## 3. Issue Cannot Be Reproduced
+
+If the memory or OOM issue cannot be reproduced in a test environment that mirrors production, please provide:
+
+1. **Information from Sections 1 and 2**
+ - Memory/issue classification (Section 1)
+ - Reproduction attempts and environment details (Section 2)
+
+2. **Additional context** to help investigate:
+ - **Workload:** A realistic sample of requests processed before and during the spike, including any recent configuration changes.
+ - **Metrics:** Memory usage, CPU usage, P50/P99 latency, and any pod restarts or OOM events. Please include **screenshots** of the metrics whenever possible.
+ - **Logs / Alerts:** Any relevant logs or alerts captured **before and during the spike**, including OOM errors or stack traces if available.
+
+> Providing this information allows the team to analyze patterns, correlate memory spikes or OOMs with traffic or configuration, and attempt to reproduce the issue internally. Without it, our engineers will not have enough information to investigate the problem.
diff --git a/docs/my-website/docs/tutorials/claude_code_customer_tracking.md b/docs/my-website/docs/tutorials/claude_code_customer_tracking.md
new file mode 100644
index 00000000000..fc6a3ccc9bb
--- /dev/null
+++ b/docs/my-website/docs/tutorials/claude_code_customer_tracking.md
@@ -0,0 +1,99 @@
+# Claude Code - Granular Cost Tracking
+
+Track Claude Code usage by customer or tags using LiteLLM proxy. This enables granular cost attribution for billing, budgeting, and analytics.
+
+## How It Works
+
+Claude Code supports custom headers via `ANTHROPIC_CUSTOM_HEADERS`. LiteLLM automatically tracks requests with specific headers for cost attribution.
+
+## Tracking Options
+
+Choose how you want to attribute costs:
+
+| Track By | Header | Use Case |
+|----------|--------|----------|
+| Customer | `x-litellm-customer-id` | Bill customers, per-user budgets |
+| Tags | `x-litellm-tags` | Project tracking, cost centers, environments |
+
+## Environment Variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `ANTHROPIC_BASE_URL` | LiteLLM proxy URL | `http://localhost:4000` |
+| `ANTHROPIC_API_KEY` | LiteLLM API key | `sk-1234` |
+| `ANTHROPIC_CUSTOM_HEADERS` | Custom headers (`header-name: value` format) | See examples below |
+
+## Option 1: Track by Customer
+
+Use this to attribute costs to specific customers or end-users.
+
+```bash
+export ANTHROPIC_BASE_URL=http://localhost:4000
+export ANTHROPIC_API_KEY=sk-1234
+export ANTHROPIC_CUSTOM_HEADERS="x-litellm-customer-id: claude-ishaan-local"
+```
+
+## Option 2: Track by Tags
+
+Use this to attribute costs to projects, cost centers, or environments. Pass comma-separated tags.
+
+```bash
+export ANTHROPIC_BASE_URL=http://localhost:4000
+export ANTHROPIC_API_KEY=sk-1234
+export ANTHROPIC_CUSTOM_HEADERS="x-litellm-tags: project:acme,env:prod,team:backend"
+```
+
+
+## Quick Start
+
+### 1. Set Environment Variables
+
+```bash
+export ANTHROPIC_BASE_URL=http://localhost:4000
+export ANTHROPIC_API_KEY=sk-1234
+export ANTHROPIC_CUSTOM_HEADERS="x-litellm-customer-id: claude-ishaan-local"
+```
+
+### 2. Use Claude Code
+
+```bash
+claude
+```
+
+All requests will now be tracked under the customer ID `claude-ishaan-local`.
+
+
+
+
+
+
+
+### 3. View Usage in LiteLLM UI
+
+Navigate to the **Logs** tab in the LiteLLM UI.
+
+
+
+Click on a request to see details.
+
+
+
+Filter by customer ID to see all requests for that customer.
+
+
+
+## Supported Headers
+
+| Header | Description |
+|--------|-------------|
+| `x-litellm-customer-id` | Track by customer/end-user ID |
+| `x-litellm-end-user-id` | Alternative customer ID header |
+| `x-litellm-tags` | Comma-separated tags for cost attribution |
+
+## Related
+
+- [Claude Code Quickstart](./claude_responses_api.md)
+- [Customer Budgets](../proxy/customers.md)
+- [Tag Budgets](../proxy/tag_budgets.md)
+- [Track Usage for Coding Tools](./cost_tracking_coding.md)
+
diff --git a/docs/my-website/docs/tutorials/claude_code_websearch.md b/docs/my-website/docs/tutorials/claude_code_websearch.md
new file mode 100644
index 00000000000..cc2f79666da
--- /dev/null
+++ b/docs/my-website/docs/tutorials/claude_code_websearch.md
@@ -0,0 +1,192 @@
+# Claude Code - WebSearch Across All Providers
+
+Enable Claude Code's web search tool to work with any provider (Bedrock, Azure, Vertex, etc.). LiteLLM automatically intercepts web search requests and executes them server-side.
+
+## Proxy Configuration
+
+Add WebSearch interception to your `litellm_config.yaml`:
+
+```yaml
+model_list:
+ - model_name: bedrock-sonnet
+ litellm_params:
+ model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
+ aws_region_name: us-east-1
+
+# Enable WebSearch interception for providers
+litellm_settings:
+ callbacks:
+ - websearch_interception:
+ enabled_providers:
+ - bedrock
+ - azure
+ - vertex_ai
+ search_tool_name: perplexity-search # Optional: specific search tool
+
+# Configure search provider
+search_tools:
+ - search_tool_name: perplexity-search
+ litellm_params:
+ search_provider: perplexity
+ api_key: os.environ/PERPLEXITY_API_KEY
+```
+
+## Quick Start
+
+### 1. Configure LiteLLM Proxy
+
+Create `config.yaml`:
+
+```yaml
+model_list:
+ - model_name: bedrock-sonnet
+ litellm_params:
+ model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
+ aws_region_name: us-east-1
+
+litellm_settings:
+ callbacks:
+ - websearch_interception:
+ enabled_providers: [bedrock]
+
+search_tools:
+ - search_tool_name: perplexity-search
+ litellm_params:
+ search_provider: perplexity
+ api_key: os.environ/PERPLEXITY_API_KEY
+```
+
+### 2. Start Proxy
+
+```bash
+export PERPLEXITY_API_KEY=your-key
+litellm --config config.yaml
+```
+
+### 3. Use with Claude Code
+
+```bash
+export ANTHROPIC_BASE_URL=http://localhost:4000
+export ANTHROPIC_API_KEY=sk-1234
+claude
+```
+
+Now use web search in Claude Code - it works with any provider!
+
+## How It Works
+
+When Claude Code sends a web search request, LiteLLM:
+1. Intercepts the native `web_search` tool
+2. Converts it to LiteLLM's standard format
+3. Executes the search via Perplexity/Tavily
+4. Returns the final answer to Claude Code
+
+```mermaid
+sequenceDiagram
+ participant CC as Claude Code
+ participant LP as LiteLLM Proxy
+ participant B as Bedrock/Azure/etc
+ participant P as Perplexity/Tavily
+
+ CC->>LP: Request with web_search tool
+ Note over LP: Convert native tool
to LiteLLM format
+ LP->>B: Request with converted tool
+ B-->>LP: Response: tool_use
+ Note over LP: Detect web search
tool_use
+ LP->>P: Execute search
+ P-->>LP: Search results
+ LP->>B: Follow-up with results
+ B-->>LP: Final answer
+ LP-->>CC: Final answer with search results
+```
+
+**Result**: One API call from Claude Code → Complete answer with search results
+
+## Supported Providers
+
+| Provider | Native Web Search | With LiteLLM |
+|----------|-------------------|--------------|
+| **Anthropic** | ✅ Yes | ✅ Yes |
+| **Bedrock** | ❌ No | ✅ Yes |
+| **Azure** | ❌ No | ✅ Yes |
+| **Vertex AI** | ❌ No | ✅ Yes |
+| **Other Providers** | ❌ No | ✅ Yes |
+
+## Search Providers
+
+Configure which search provider to use. LiteLLM supports multiple search providers:
+
+| Provider | Configuration |
+|----------|---------------|
+| **Perplexity** | `search_provider: perplexity` |
+| **Tavily** | `search_provider: tavily` |
+
+See [all supported search providers](../search/index.md) for the complete list.
+
+## Configuration Options
+
+### WebSearch Interception Parameters
+
+| Parameter | Type | Required | Description | Example |
+|-----------|------|----------|-------------|---------|
+| `enabled_providers` | List[String] | Yes | List of providers to enable web search interception for | `[bedrock, azure, vertex_ai]` |
+| `search_tool_name` | String | No | Specific search tool from `search_tools` config. If not set, uses first available search tool. | `perplexity-search` |
+
+### Supported Provider Values
+
+Use these values in `enabled_providers`:
+
+| Provider | Value | Description |
+|----------|-------|-------------|
+| AWS Bedrock | `bedrock` | Amazon Bedrock Claude models |
+| Azure OpenAI | `azure` | Azure-hosted models |
+| Google Vertex AI | `vertex_ai` | Google Cloud Vertex AI |
+| Any Other | Provider name | Any LiteLLM-supported provider |
+
+### Complete Configuration Example
+
+```yaml
+model_list:
+ - model_name: bedrock-sonnet
+ litellm_params:
+ model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
+ aws_region_name: us-east-1
+
+ - model_name: azure-gpt4
+ litellm_params:
+ model: azure/gpt-4
+ api_base: https://my-azure.openai.azure.com
+ api_key: os.environ/AZURE_API_KEY
+
+litellm_settings:
+ callbacks:
+ - websearch_interception:
+ enabled_providers:
+ - bedrock # Enable for AWS Bedrock
+ - azure # Enable for Azure OpenAI
+ - vertex_ai # Enable for Google Vertex
+ search_tool_name: perplexity-search # Optional: use specific search tool
+
+# Configure search tools
+search_tools:
+ - search_tool_name: perplexity-search
+ litellm_params:
+ search_provider: perplexity
+ api_key: os.environ/PERPLEXITY_API_KEY
+
+ - search_tool_name: tavily-search
+ litellm_params:
+ search_provider: tavily
+ api_key: os.environ/TAVILY_API_KEY
+```
+
+**How search tool selection works:**
+- If `search_tool_name` is specified → Uses that specific search tool
+- If `search_tool_name` is not specified → Uses first search tool in `search_tools` list
+- In example above: Without `search_tool_name`, would use `perplexity-search` (first in list)
+
+## Related
+
+- [Claude Code Quickstart](./claude_responses_api.md)
+- [Claude Code Cost Tracking](./claude_code_customer_tracking.md)
+- [Using Non-Anthropic Models](./claude_non_anthropic_models.md)
diff --git a/docs/my-website/docs/tutorials/claude_mcp.md b/docs/my-website/docs/tutorials/claude_mcp.md
new file mode 100644
index 00000000000..07c3cead0be
--- /dev/null
+++ b/docs/my-website/docs/tutorials/claude_mcp.md
@@ -0,0 +1,93 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Use Claude Code with MCPs
+
+This tutorial shows how to connect MCP servers to Claude Code via LiteLLM Proxy.
+
+Note: LiteLLM supports OAuth for MCP servers as well. [Learn more](https://docs.litellm.ai/docs/mcp#mcp-oauth)
+
+## Connecting MCP Servers
+
+You can also connect MCP servers to Claude Code via LiteLLM Proxy.
+
+
+1. Add the MCP server to your `config.yaml`
+
+
+
+
+In this example, we'll add the Github MCP server to our `config.yaml`
+
+```yaml title="config.yaml" showLineNumbers
+mcp_servers:
+ github_mcp:
+ url: "https://api.githubcopilot.com/mcp"
+ auth_type: oauth2
+ client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
+ client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
+```
+
+
+
+
+In this example, we'll add the Atlassian MCP server to our `config.yaml`
+
+```yaml title="config.yaml" showLineNumbers
+atlassian_mcp:
+ server_id: atlassian_mcp_id
+ url: "https://mcp.atlassian.com/v1/sse"
+ transport: "sse"
+ auth_type: oauth2
+```
+
+
+
+
+2. Start LiteLLM Proxy
+
+```bash
+litellm --config /path/to/config.yaml
+
+# RUNNING on http://0.0.0.0:4000
+```
+
+3. Use the MCP server in Claude Code
+
+```bash
+claude mcp add --transport http litellm_proxy http://0.0.0.0:4000/github_mcp/mcp --header "Authorization: Bearer sk-LITELLM_VIRTUAL_KEY"
+```
+
+For MCP servers that require dynamic client registration (such as Atlassian), please set `x-litellm-api-key: Bearer sk-LITELLM_VIRTUAL_KEY` instead of using `Authorization: Bearer LITELLM_VIRTUAL_KEY`.
+
+4. Authenticate via Claude Code
+
+a. Start Claude Code
+
+```bash
+claude
+```
+
+b. Authenticate via Claude Code
+
+```bash
+/mcp
+```
+
+c. Select the MCP server
+
+```bash
+> litellm_proxy
+```
+
+d. Start Oauth flow via Claude Code
+
+```bash
+> 1. Authenticate
+ 2. Reconnect
+ 3. Disable
+```
+
+e. Once completed, you should see this success message:
+
+.default})
diff --git a/docs/my-website/docs/tutorials/claude_non_anthropic_models.md b/docs/my-website/docs/tutorials/claude_non_anthropic_models.md
new file mode 100644
index 00000000000..75ac08e3094
--- /dev/null
+++ b/docs/my-website/docs/tutorials/claude_non_anthropic_models.md
@@ -0,0 +1,316 @@
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Use Claude Code with Non-Anthropic Models
+
+This tutorial shows how to use Claude Code with non-Anthropic models like OpenAI, Gemini, and other LLM providers through LiteLLM proxy.
+
+:::info
+
+LiteLLM automatically translates between different provider formats, allowing you to use any supported LLM provider with Claude Code while maintaining the Anthropic Messages API format.
+
+:::
+
+## Prerequisites
+
+- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed
+- API keys for your chosen providers (OpenAI, Vertex AI, etc.)
+
+## Installation
+
+First, install LiteLLM with proxy support:
+
+```bash
+pip install 'litellm[proxy]'
+```
+
+## Configuration
+
+### 1. Setup config.yaml
+
+Create a configuration file with your preferred non-Anthropic models:
+
+
+
+
+```yaml
+model_list:
+ # OpenAI GPT-4o
+ - model_name: gpt-4o
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: os.environ/OPENAI_API_KEY
+
+ # OpenAI GPT-4o-mini
+ - model_name: gpt-4o-mini
+ litellm_params:
+ model: openai/gpt-4o-mini
+ api_key: os.environ/OPENAI_API_KEY
+```
+
+Set your environment variables:
+
+```bash
+export OPENAI_API_KEY="your-openai-api-key"
+export LITELLM_MASTER_KEY="sk-1234567890" # Generate a secure key
+```
+
+
+
+
+```yaml
+model_list:
+ # Google Gemini
+ - model_name: gemini-3.0-flash-exp
+ litellm_params:
+ model: gemini/gemini-3.0-flash-exp
+ api_key: os.environ/GEMINI_API_KEY
+```
+
+Set your environment variables:
+
+```bash
+export GEMINI_API_KEY="your-gemini-api-key"
+export LITELLM_MASTER_KEY="sk-1234567890" # Generate a secure key
+```
+
+
+
+
+```yaml
+model_list:
+ # Google Gemini
+ - model_name: vertex-gemini-3-flash-preview
+ litellm_params:
+ model: vertex_ai/gemini-3-flash-preview
+ vertex_credentials: os.environ/VERTEX_FILE_PATH_ENV_VAR # os.environ["VERTEX_FILE_PATH_ENV_VAR"] = "/path/to/service_account.json"
+ vertex_project: "my-test-project"
+ vertex_location: "us-east-1"
+
+ # Anthropic Claude
+ - model_name: anthropic-vertex
+ litellm_params:
+ model: vertex_ai/claude-3-sonnet@20240229
+ vertex_ai_project: "my-test-project"
+ vertex_ai_location: "us-east-1"
+ vertex_credentials: os.environ/VERTEX_FILE_PATH_ENV_VAR # os.environ["VERTEX_FILE_PATH_ENV_VAR"] = "/path/to/service_account.json"
+```
+
+Set your environment variables:
+
+```bash
+export VERTEX_FILE_PATH_ENV_VAR="/path/to/service_account.json"
+export LITELLM_MASTER_KEY="sk-1234567890"
+```
+
+
+
+
+```yaml
+model_list:
+ # Azure OpenAI
+ - model_name: azure-gpt-4
+ litellm_params:
+ model: azure/gpt-4
+ api_key: os.environ/AZURE_API_KEY
+ api_base: os.environ/AZURE_API_BASE
+ api_version: "2024-02-01"
+```
+
+Set your environment variables:
+
+```bash
+export AZURE_API_KEY="your-azure-api-key"
+export AZURE_API_BASE="https://your-resource.openai.azure.com"
+export LITELLM_MASTER_KEY="sk-1234567890"
+```
+
+
+
+
+### 2. Start LiteLLM Proxy
+
+```bash
+litellm --config /path/to/config.yaml
+
+# RUNNING on http://0.0.0.0:4000
+```
+
+### 3. Verify Setup
+
+Test that your proxy is working correctly:
+
+
+
+
+```bash
+curl -X POST http://0.0.0.0:4000/v1/messages \
+-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+-H "Content-Type: application/json" \
+-d '{
+ "model": "gpt-4o",
+ "max_tokens": 1000,
+ "messages": [{"role": "user", "content": "What is the capital of France?"}]
+}'
+```
+
+
+
+
+```bash
+curl -X POST http://0.0.0.0:4000/v1/messages \
+-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+-H "Content-Type: application/json" \
+-d '{
+ "model": "gemini-3.0-flash-exp",
+ "max_tokens": 1000,
+ "messages": [{"role": "user", "content": "What is the capital of France?"}]
+}'
+```
+
+
+
+
+```bash
+curl -X POST http://0.0.0.0:4000/v1/messages \
+-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+-H "Content-Type: application/json" \
+-d '{
+ "model": "gemini-3.0-flash-exp",
+ "max_tokens": 1000,
+ "messages": [{"role": "user", "content": "What is the capital of France?"}]
+}'
+```
+
+
+
+
+```bash
+curl -X POST http://0.0.0.0:4000/v1/messages \
+-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+-H "Content-Type: application/json" \
+-d '{
+ "model": "azure-gpt-4",
+ "max_tokens": 1000,
+ "messages": [{"role": "user", "content": "What is the capital of France?"}]
+}'
+```
+
+
+
+
+### 4. Configure Claude Code
+
+Configure Claude Code to use your LiteLLM proxy:
+
+```bash
+export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
+export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
+```
+
+:::tip
+The `LITELLM_MASTER_KEY` gives Claude Code access to all proxy models. You can also create virtual keys in the LiteLLM UI to limit access to specific models.
+:::
+
+### 5. Use Claude Code with Non-Anthropic Models
+
+Start Claude Code and specify which model to use:
+
+```bash
+# Use OpenAI GPT-4o
+claude --model gpt-4o
+
+# Use OpenAI GPT-4o-mini for faster responses
+claude --model gpt-4o-mini
+
+# Use Google Gemini
+claude --model gemini-3.0-flash-exp
+
+# Use Vertex AI Gemini
+claude --model vertex-gemini-3-flash-preview
+
+# Use Vertex AI Anthropic Claude
+claude --model anthropic-vertex
+
+# Use Azure OpenAI
+claude --model azure-gpt-4
+```
+
+## How It Works
+
+LiteLLM acts as a unified interface that:
+
+1. **Receives requests** from Claude Code in Anthropic Messages API format
+2. **Translates** the request to the target provider's format (OpenAI, Gemini, etc.)
+3. **Forwards** the request to the actual provider
+4. **Translates** the response back to Anthropic Messages API format
+5. **Returns** the response to Claude Code
+
+This allows you to use Claude Code's interface with any LLM provider supported by LiteLLM.
+
+## Advanced Features
+
+### Load Balancing and Fallbacks
+
+Configure multiple deployments with automatic fallback:
+
+```yaml
+model_list:
+ - model_name: gpt-4o # virtual model name
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: os.environ/OPENAI_API_KEY
+
+ - model_name: gpt-4o # same virtual name
+ litellm_params:
+ model: azure/gpt-4o
+ api_key: os.environ/AZURE_API_KEY
+ api_base: os.environ/AZURE_API_BASE
+
+router_settings:
+ routing_strategy: simple-shuffle # Load balance between deployments
+ num_retries: 2
+ timeout: 30
+```
+
+### Usage Tracking and Budgets
+
+Track usage and set budgets through the LiteLLM UI:
+
+```yaml
+litellm_settings:
+ master_key: os.environ/LITELLM_MASTER_KEY
+ database_url: "postgresql://..." # Enable database for tracking
+
+general_settings:
+ store_model_in_db: true
+```
+
+Start the proxy with the UI:
+
+```bash
+litellm --config /path/to/config.yaml --detailed_debug
+```
+
+Access the UI at `http://0.0.0.0:4000/ui` to:
+- View usage analytics
+- Set budget limits per user/key
+- Monitor costs across different providers
+- Create virtual keys with specific permissions
+
+
+## Supported Providers
+
+LiteLLM supports 100+ providers. Here are some popular ones for use with Claude Code:
+
+- **OpenAI**: GPT-4o, GPT-4o-mini, o1, o3-mini
+- **Google**: Gemini 2.0 Flash, Gemini 1.5 Pro/Flash
+- **Azure OpenAI**: All OpenAI models via Azure
+- **AWS Bedrock**: Llama, Mistral, and other models
+- **Vertex AI**: Gemini, Claude, and other models on Google Cloud
+- **Groq**: Fast inference for Llama and Mixtral
+- **Together AI**: Llama, Mixtral, and other open source models
+- **Deepseek**: Deepseek-chat, Deepseek-coder
+
+[View full list of supported providers →](https://docs.litellm.ai/docs/providers)
diff --git a/docs/my-website/docs/tutorials/claude_responses_api.md b/docs/my-website/docs/tutorials/claude_responses_api.md
index aafeccceaf5..6b681d93a83 100644
--- a/docs/my-website/docs/tutorials/claude_responses_api.md
+++ b/docs/my-website/docs/tutorials/claude_responses_api.md
@@ -2,7 +2,7 @@ import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Claude Code
+# Claude Code Quickstart
This tutorial shows how to call Claude models through LiteLLM proxy from Claude Code.
@@ -142,7 +142,7 @@ Common issues and solutions:
- Ensure the model name in Claude Code matches exactly with your `config.yaml`
- Check LiteLLM logs for detailed error messages
-## Using Multiple Models
+## Using Bedrock/Vertex AI/Azure Foundry Models
Expand your configuration to support multiple providers and models:
@@ -151,25 +151,6 @@ Expand your configuration to support multiple providers and models:
```yaml
model_list:
- # OpenAI models
- - model_name: codex-mini
- litellm_params:
- model: openai/codex-mini
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
-
- - model_name: o3-pro
- litellm_params:
- model: openai/o3-pro
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
-
- - model_name: gpt-4o
- litellm_params:
- model: openai/gpt-4o
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
-
# Anthropic models
- model_name: claude-3-5-sonnet-20241022
litellm_params:
@@ -189,6 +170,24 @@ model_list:
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_region_name: us-east-1
+ # Azure Foundry
+ - model_name: claude-4-azure
+ litellm_params:
+ model: azure_ai/claude-opus-4-1
+ api_key: os.environ/AZURE_AI_API_KEY
+ api_base: os.environ/AZURE_AI_API_BASE # https://my-resource.services.ai.azure.com/anthropic
+
+ # Google Vertex AI
+ - model_name: anthropic-vertex
+ litellm_params:
+ model: vertex_ai/claude-haiku-4-5@20251001
+ vertex_ai_project: "my-test-project"
+ vertex_ai_location: "us-east-1"
+ vertex_credentials: os.environ/VERTEX_FILE_PATH_ENV_VAR # os.environ["VERTEX_FILE_PATH_ENV_VAR"] = "/path/to/service_account.json"
+
+
+
+
litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
```
@@ -204,6 +203,12 @@ claude --model claude-3-5-haiku-20241022
# Use Bedrock deployment
claude --model claude-bedrock
+
+# Use Azure Foundry deployment
+claude --model claude-4-azure
+
+# Use Vertex AI deployment
+claude --model anthropic-vertex
```
@@ -211,96 +216,3 @@ claude --model claude-bedrock
-
-## Connecting MCP Servers
-
-You can also connect MCP servers to Claude Code via LiteLLM Proxy.
-
-:::note
-
-Limitations:
-
-- Currently, only HTTP MCP servers are supported
-
-:::
-
-1. Add the MCP server to your `config.yaml`
-
-
-
-
-In this example, we'll add the Github MCP server to our `config.yaml`
-
-```yaml title="config.yaml" showLineNumbers
-mcp_servers:
- github_mcp:
- url: "https://api.githubcopilot.com/mcp"
- auth_type: oauth2
- client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
- client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
-```
-
-
-
-
-In this example, we'll add the Atlassian MCP server to our `config.yaml`
-
-```yaml title="config.yaml" showLineNumbers
-atlassian_mcp:
- server_id: atlassian_mcp_id
- url: "https://mcp.atlassian.com/v1/sse"
- transport: "sse"
- auth_type: oauth2
-```
-
-
-
-
-2. Start LiteLLM Proxy
-
-```bash
-litellm --config /path/to/config.yaml
-
-# RUNNING on http://0.0.0.0:4000
-```
-
-3. Use the MCP server in Claude Code
-
-```bash
-claude mcp add --transport http litellm_proxy http://0.0.0.0:4000/github_mcp/mcp --header "Authorization: Bearer sk-LITELLM_VIRTUAL_KEY"
-```
-
-For MCP servers that require dynamic client registration (such as Atlassian), please set `x-litellm-api-key: Bearer sk-LITELLM_VIRTUAL_KEY` instead of using `Authorization: Bearer LITELLM_VIRTUAL_KEY`.
-
-4. Authenticate via Claude Code
-
-a. Start Claude Code
-
-```bash
-claude
-```
-
-b. Authenticate via Claude Code
-
-```bash
-/mcp
-```
-
-c. Select the MCP server
-
-```bash
-> litellm_proxy
-```
-
-d. Start Oauth flow via Claude Code
-
-```bash
-> 1. Authenticate
- 2. Reconnect
- 3. Disable
-```
-
-e. Once completed, you should see this success message:
-
-
-
diff --git a/docs/my-website/img/mcp_playground.png b/docs/my-website/img/mcp_playground.png
new file mode 100644
index 00000000000..dac88544363
Binary files /dev/null and b/docs/my-website/img/mcp_playground.png differ
diff --git a/docs/my-website/img/mcp_tool_testing_playground.png b/docs/my-website/img/mcp_tool_testing_playground.png
new file mode 100644
index 00000000000..56b526a20cd
Binary files /dev/null and b/docs/my-website/img/mcp_tool_testing_playground.png differ
diff --git a/docs/my-website/img/ui_deleted_keys_table.png b/docs/my-website/img/ui_deleted_keys_table.png
new file mode 100644
index 00000000000..9d7cf8455b3
Binary files /dev/null and b/docs/my-website/img/ui_deleted_keys_table.png differ
diff --git a/docs/my-website/img/ui_endpoint_activity.png b/docs/my-website/img/ui_endpoint_activity.png
index e2550066a07..fc0a90ca444 100644
Binary files a/docs/my-website/img/ui_endpoint_activity.png and b/docs/my-website/img/ui_endpoint_activity.png differ
diff --git a/docs/my-website/release_notes/v1.80.15/index.md b/docs/my-website/release_notes/v1.80.15/index.md
index 33c01973b18..4037a0d9b5d 100644
--- a/docs/my-website/release_notes/v1.80.15/index.md
+++ b/docs/my-website/release_notes/v1.80.15/index.md
@@ -1,5 +1,5 @@
---
-title: "[Preview] v1.80.15.rc.1 - Manus API Support"
+title: "v1.80.15-stable - Manus API Support"
slug: "v1-80-15"
date: 2026-01-10T10:00:00
authors:
@@ -27,7 +27,7 @@ import TabItem from '@theme/TabItem';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
-docker.litellm.ai/berriai/litellm:v1.80.15.rc.1
+docker.litellm.ai/berriai/litellm:v1.80.15-stable.1
```
@@ -638,6 +638,6 @@ Users can now see Endpoint Activity Metrics in the UI.
## Full Changelog
-**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.80.11.rc.1...v1.80.14.rc.1)**
+**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.80.11.rc.1...v1.80.15-stable.1)**
diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js
index 2e8ea07ab75..102e3dfe1c5 100644
--- a/docs/my-website/sidebars.js
+++ b/docs/my-website/sidebars.js
@@ -108,15 +108,31 @@ const sidebars = {
{
type: "category",
label: "AI Tools (OpenWebUI, Claude Code, etc.)",
+ link: {
+ type: "generated-index",
+ title: "AI Tools",
+ description: "Integrate LiteLLM with AI tools like OpenWebUI, Claude Code, and more",
+ slug: "/ai_tools"
+ },
items: [
- "tutorials/claude_responses_api",
+ "tutorials/openweb_ui",
+ {
+ type: "category",
+ label: "Claude Code",
+ items: [
+ "tutorials/claude_responses_api",
+ "tutorials/claude_code_customer_tracking",
+ "tutorials/claude_code_websearch",
+ "tutorials/claude_mcp",
+ "tutorials/claude_non_anthropic_models",
+ ]
+ },
"tutorials/cost_tracking_coding",
"tutorials/cursor_integration",
"tutorials/github_copilot_integration",
"tutorials/litellm_gemini_cli",
"tutorials/litellm_qwen_code_cli",
- "tutorials/openai_codex",
- "tutorials/openweb_ui"
+ "tutorials/openai_codex"
]
},
@@ -259,12 +275,21 @@ const sidebars = {
"proxy/ui/bulk_edit_users",
"proxy/ui_credentials",
"tutorials/scim_litellm",
+ {
+ type: "category",
+ label: "UI Usage Tracking",
+ items: [
+ "proxy/customer_usage",
+ "proxy/endpoint_activity"
+ ]
+ },
{
type: "category",
label: "UI Logs",
items: [
"proxy/ui_logs",
- "proxy/ui_logs_sessions"
+ "proxy/ui_logs_sessions",
+ "proxy/deleted_keys_teams"
]
}
],
@@ -314,7 +339,6 @@ const sidebars = {
"proxy/team_budgets",
"proxy/tag_budgets",
"proxy/customers",
- "proxy/customer_usage",
"proxy/dynamic_rate_limit",
"proxy/rate_limit_tiers",
"proxy/temporary_budget_increase",
@@ -488,6 +512,7 @@ const sidebars = {
"mcp_control",
"mcp_cost",
"mcp_guardrail",
+ "mcp_troubleshoot",
]
},
"anthropic_unified",
@@ -602,6 +627,7 @@ const sidebars = {
label: "Azure AI",
items: [
"providers/azure_ai",
+ "providers/azure_ai/azure_model_router",
"providers/azure_ai_agents",
"providers/azure_ocr",
"providers/azure_document_intelligence",
@@ -841,6 +867,7 @@ const sidebars = {
"proxy/load_balancing",
"proxy/provider_budget_routing",
"proxy/reliability",
+ "proxy/fallback_management",
"proxy/tag_routing",
"proxy/timeout",
"wildcard_routing"
@@ -860,10 +887,11 @@ const sidebars = {
type: "category",
label: "Tutorials",
items: [
- "tutorials/openweb_ui",
- "tutorials/openai_codex",
- "tutorials/litellm_gemini_cli",
- "tutorials/litellm_qwen_code_cli",
+ {
+ type: "link",
+ label: "AI Coding Tools (OpenWebUI, Claude Code, Gemini CLI, OpenAI Codex, etc.)",
+ href: "/docs/ai_tools",
+ },
"tutorials/anthropic_file_usage",
"tutorials/default_team_self_serve",
"tutorials/msft_sso",
@@ -873,7 +901,6 @@ const sidebars = {
"tutorials/presidio_pii_masking",
"tutorials/elasticsearch_logging",
"tutorials/gemini_realtime_with_audio",
- "tutorials/claude_responses_api",
{
type: "category",
label: "LiteLLM Python SDK Tutorials",
@@ -970,6 +997,14 @@ const sidebars = {
],
},
"troubleshoot",
+ {
+ type: "category",
+ label: "Issue Reporting",
+ items: [
+ "troubleshoot/cpu_issues",
+ "troubleshoot/memory_issues",
+ ],
+ },
],
};
diff --git a/document.txt b/document.txt
deleted file mode 100644
index 4a91207970a..00000000000
--- a/document.txt
+++ /dev/null
@@ -1,19 +0,0 @@
-LiteLLM provides a unified interface for calling 100+ different LLM providers.
-
-Key capabilities:
-- Translate requests to provider-specific formats
-- Consistent OpenAI-compatible responses
-- Retry and fallback logic across deployments
-- Proxy server with authentication and rate limiting
-- Support for streaming, function calling, and embeddings
-
-Popular providers supported:
-- OpenAI (GPT-4, GPT-3.5)
-- Anthropic (Claude)
-- AWS Bedrock
-- Azure OpenAI
-- Google Vertex AI
-- Cohere
-- And 95+ more
-
-This allows developers to easily switch between providers without code changes.
diff --git a/enterprise/litellm_enterprise/proxy/__init__.py b/enterprise/litellm_enterprise/proxy/__init__.py
new file mode 100644
index 00000000000..52b74882bc9
--- /dev/null
+++ b/enterprise/litellm_enterprise/proxy/__init__.py
@@ -0,0 +1 @@
+# Package marker for enterprise proxy components.
diff --git a/enterprise/litellm_enterprise/proxy/common_utils/__init__.py b/enterprise/litellm_enterprise/proxy/common_utils/__init__.py
new file mode 100644
index 00000000000..fe8384c8925
--- /dev/null
+++ b/enterprise/litellm_enterprise/proxy/common_utils/__init__.py
@@ -0,0 +1 @@
+# Package marker for enterprise proxy common utilities.
diff --git a/enterprise/litellm_enterprise/proxy/hooks/managed_files.py b/enterprise/litellm_enterprise/proxy/hooks/managed_files.py
index a83d7e224b5..445d2b242b4 100644
--- a/enterprise/litellm_enterprise/proxy/hooks/managed_files.py
+++ b/enterprise/litellm_enterprise/proxy/hooks/managed_files.py
@@ -8,6 +8,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Union, cas
from fastapi import HTTPException
+import litellm
from litellm import Router, verbose_logger
from litellm._uuid import uuid
from litellm.caching.caching import DualCache
@@ -836,15 +837,36 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
return response
async def afile_retrieve(
- self, file_id: str, litellm_parent_otel_span: Optional[Span]
+ self, file_id: str, litellm_parent_otel_span: Optional[Span], llm_router=None
) -> OpenAIFileObject:
stored_file_object = await self.get_unified_file_id(
file_id, litellm_parent_otel_span
)
- if stored_file_object:
- return stored_file_object.file_object
- else:
+
+ # Case 1 : This is not a managed file
+ if not stored_file_object:
raise Exception(f"LiteLLM Managed File object with id={file_id} not found")
+
+ # Case 2: Managed file and the file object exists in the database
+ if stored_file_object and stored_file_object.file_object:
+ return stored_file_object.file_object
+
+ # Case 3: Managed file exists in the database but not the file object (for. e.g the batch task might not have run)
+ # So we fetch the file object from the provider. We deliberately do not store the result to avoid interfering with batch cost tracking code.
+ if not llm_router:
+ raise Exception(
+ f"LiteLLM Managed File object with id={file_id} has no file_object "
+ f"and llm_router is required to fetch from provider"
+ )
+
+ try:
+ model_id, model_file_id = next(iter(stored_file_object.model_mappings.items()))
+ credentials = llm_router.get_deployment_credentials_with_provider(model_id) or {}
+ response = await litellm.afile_retrieve(file_id=model_file_id, **credentials)
+ response.id = file_id # Replace with unified ID
+ return response
+ except Exception as e:
+ raise Exception(f"Failed to retrieve file {file_id} from provider: {str(e)}") from e
async def afile_list(
self,
@@ -868,10 +890,11 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
[file_id], litellm_parent_otel_span
)
+ delete_response = None
specific_model_file_id_mapping = model_file_id_mapping.get(file_id)
if specific_model_file_id_mapping:
for model_id, model_file_id in specific_model_file_id_mapping.items():
- await llm_router.afile_delete(model=model_id, file_id=model_file_id, **data) # type: ignore
+ delete_response = await llm_router.afile_delete(model=model_id, file_id=model_file_id, **data) # type: ignore
stored_file_object = await self.delete_unified_file_id(
file_id, litellm_parent_otel_span
@@ -879,6 +902,9 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
if stored_file_object:
return stored_file_object
+ elif delete_response:
+ delete_response.id = file_id
+ return delete_response
else:
raise Exception(f"LiteLLM Managed File object with id={file_id} not found")
diff --git a/enterprise/pyproject.toml b/enterprise/pyproject.toml
index 1f3da432574..0d86460a649 100644
--- a/enterprise/pyproject.toml
+++ b/enterprise/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-enterprise"
-version = "0.1.27"
+version = "0.1.28"
description = "Package for LiteLLM Enterprise features"
authors = ["BerriAI"]
readme = "README.md"
@@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
-version = "0.1.27"
+version = "0.1.28"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-enterprise==",
diff --git a/flux2_test_image.png b/flux2_test_image.png
deleted file mode 100644
index d40fa1a65f2..00000000000
Binary files a/flux2_test_image.png and /dev/null differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.15-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.15-py3-none-any.whl
new file mode 100644
index 00000000000..ba2e5e5fce5
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.15-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.15.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.15.tar.gz
new file mode 100644
index 00000000000..7d01b3de6ff
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.15.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.22-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.22-py3-none-any.whl
new file mode 100644
index 00000000000..1e2f6967dc7
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.22-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.22.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.22.tar.gz
new file mode 100644
index 00000000000..1864c77ddda
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.22.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.23-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.23-py3-none-any.whl
new file mode 100644
index 00000000000..54fb2d23cdd
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.23-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.23.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.23.tar.gz
new file mode 100644
index 00000000000..9c1a2625f71
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.4.23.tar.gz differ
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20251219110931_add_deleted_keys_and_deleted_teams_tables/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251219110931_add_deleted_keys_and_deleted_teams_tables/migration.sql
new file mode 100644
index 00000000000..6ca66ddaad2
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251219110931_add_deleted_keys_and_deleted_teams_tables/migration.sql
@@ -0,0 +1,117 @@
+-- CreateTable
+CREATE TABLE "LiteLLM_DeletedTeamTable" (
+ "id" TEXT NOT NULL,
+ "team_id" TEXT NOT NULL,
+ "team_alias" TEXT,
+ "organization_id" TEXT,
+ "object_permission_id" TEXT,
+ "admins" TEXT[],
+ "members" TEXT[],
+ "members_with_roles" JSONB NOT NULL DEFAULT '{}',
+ "metadata" JSONB NOT NULL DEFAULT '{}',
+ "max_budget" DOUBLE PRECISION,
+ "spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0,
+ "models" TEXT[],
+ "max_parallel_requests" INTEGER,
+ "tpm_limit" BIGINT,
+ "rpm_limit" BIGINT,
+ "budget_duration" TEXT,
+ "budget_reset_at" TIMESTAMP(3),
+ "blocked" BOOLEAN NOT NULL DEFAULT false,
+ "model_spend" JSONB NOT NULL DEFAULT '{}',
+ "model_max_budget" JSONB NOT NULL DEFAULT '{}',
+ "team_member_permissions" TEXT[] DEFAULT ARRAY[]::TEXT[],
+ "model_id" INTEGER,
+ "created_at" TIMESTAMP(3),
+ "updated_at" TIMESTAMP(3),
+ "deleted_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ "deleted_by" TEXT,
+ "deleted_by_api_key" TEXT,
+ "litellm_changed_by" TEXT,
+
+ CONSTRAINT "LiteLLM_DeletedTeamTable_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateTable
+CREATE TABLE "LiteLLM_DeletedVerificationToken" (
+ "id" TEXT NOT NULL,
+ "token" TEXT NOT NULL,
+ "key_name" TEXT,
+ "key_alias" TEXT,
+ "soft_budget_cooldown" BOOLEAN NOT NULL DEFAULT false,
+ "spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0,
+ "expires" TIMESTAMP(3),
+ "models" TEXT[],
+ "aliases" JSONB NOT NULL DEFAULT '{}',
+ "config" JSONB NOT NULL DEFAULT '{}',
+ "user_id" TEXT,
+ "team_id" TEXT,
+ "permissions" JSONB NOT NULL DEFAULT '{}',
+ "max_parallel_requests" INTEGER,
+ "metadata" JSONB NOT NULL DEFAULT '{}',
+ "blocked" BOOLEAN,
+ "tpm_limit" BIGINT,
+ "rpm_limit" BIGINT,
+ "max_budget" DOUBLE PRECISION,
+ "budget_duration" TEXT,
+ "budget_reset_at" TIMESTAMP(3),
+ "allowed_cache_controls" TEXT[] DEFAULT ARRAY[]::TEXT[],
+ "allowed_routes" TEXT[] DEFAULT ARRAY[]::TEXT[],
+ "model_spend" JSONB NOT NULL DEFAULT '{}',
+ "model_max_budget" JSONB NOT NULL DEFAULT '{}',
+ "budget_id" TEXT,
+ "organization_id" TEXT,
+ "object_permission_id" TEXT,
+ "created_at" TIMESTAMP(3),
+ "created_by" TEXT,
+ "updated_at" TIMESTAMP(3),
+ "updated_by" TEXT,
+ "rotation_count" INTEGER DEFAULT 0,
+ "auto_rotate" BOOLEAN DEFAULT false,
+ "rotation_interval" TEXT,
+ "last_rotation_at" TIMESTAMP(3),
+ "key_rotation_at" TIMESTAMP(3),
+ "deleted_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ "deleted_by" TEXT,
+ "deleted_by_api_key" TEXT,
+ "litellm_changed_by" TEXT,
+
+ CONSTRAINT "LiteLLM_DeletedVerificationToken_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedTeamTable_team_id_idx" ON "LiteLLM_DeletedTeamTable"("team_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedTeamTable_deleted_at_idx" ON "LiteLLM_DeletedTeamTable"("deleted_at");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedTeamTable_organization_id_idx" ON "LiteLLM_DeletedTeamTable"("organization_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedTeamTable_team_alias_idx" ON "LiteLLM_DeletedTeamTable"("team_alias");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedTeamTable_created_at_idx" ON "LiteLLM_DeletedTeamTable"("created_at");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_token_idx" ON "LiteLLM_DeletedVerificationToken"("token");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_deleted_at_idx" ON "LiteLLM_DeletedVerificationToken"("deleted_at");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_user_id_idx" ON "LiteLLM_DeletedVerificationToken"("user_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_team_id_idx" ON "LiteLLM_DeletedVerificationToken"("team_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_organization_id_idx" ON "LiteLLM_DeletedVerificationToken"("organization_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_key_alias_idx" ON "LiteLLM_DeletedVerificationToken"("key_alias");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DeletedVerificationToken_created_at_idx" ON "LiteLLM_DeletedVerificationToken"("created_at");
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20260116142756_update_deleted_keys_teams_table_routing_settings/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20260116142756_update_deleted_keys_teams_table_routing_settings/migration.sql
new file mode 100644
index 00000000000..9426bed0da2
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20260116142756_update_deleted_keys_teams_table_routing_settings/migration.sql
@@ -0,0 +1,6 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_DeletedTeamTable" ADD COLUMN "router_settings" JSONB DEFAULT '{}';
+
+-- AlterTable
+ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN "router_settings" JSONB DEFAULT '{}';
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/schema.prisma b/litellm-proxy-extras/litellm_proxy_extras/schema.prisma
index 56fe093a8bc..71b398c59a4 100644
--- a/litellm-proxy-extras/litellm_proxy_extras/schema.prisma
+++ b/litellm-proxy-extras/litellm_proxy_extras/schema.prisma
@@ -132,6 +132,49 @@ model LiteLLM_TeamTable {
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [object_permission_id], references: [object_permission_id])
}
+// Audit table for deleted teams - preserves spend and team information for historical tracking
+model LiteLLM_DeletedTeamTable {
+ id String @id @default(uuid())
+ team_id String // Original team_id
+ team_alias String?
+ organization_id String?
+ object_permission_id String?
+ admins String[]
+ members String[]
+ members_with_roles Json @default("{}")
+ metadata Json @default("{}")
+ max_budget Float?
+ spend Float @default(0.0)
+ models String[]
+ max_parallel_requests Int?
+ tpm_limit BigInt?
+ rpm_limit BigInt?
+ budget_duration String?
+ budget_reset_at DateTime?
+ blocked Boolean @default(false)
+ model_spend Json @default("{}")
+ model_max_budget Json @default("{}")
+ router_settings Json? @default("{}")
+ team_member_permissions String[] @default([])
+ model_id Int? // id for LiteLLM_ModelTable -> stores team-level model aliases
+
+ // Original timestamps from team creation/updates
+ created_at DateTime? @map("created_at")
+ updated_at DateTime? @map("updated_at")
+
+ // Deletion metadata
+ deleted_at DateTime @default(now()) @map("deleted_at")
+ deleted_by String? @map("deleted_by") // User who deleted the team
+ deleted_by_api_key String? @map("deleted_by_api_key") // API key hash that performed the deletion
+ litellm_changed_by String? @map("litellm_changed_by") // From litellm-changed-by header if provided
+
+ @@index([team_id])
+ @@index([deleted_at])
+ @@index([organization_id])
+ @@index([team_alias])
+ @@index([created_at])
+}
+
// Track spend, rate limit, budget Users
model LiteLLM_UserTable {
user_id String @id
@@ -259,6 +302,62 @@ model LiteLLM_VerificationToken {
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [object_permission_id], references: [object_permission_id])
}
+// Audit table for deleted keys - preserves spend and key information for historical tracking
+model LiteLLM_DeletedVerificationToken {
+ id String @id @default(uuid())
+ token String // Original token (hashed)
+ key_name String?
+ key_alias String?
+ soft_budget_cooldown Boolean @default(false)
+ spend Float @default(0.0)
+ expires DateTime?
+ models String[]
+ aliases Json @default("{}")
+ config Json @default("{}")
+ user_id String?
+ team_id String?
+ permissions Json @default("{}")
+ max_parallel_requests Int?
+ metadata Json @default("{}")
+ blocked Boolean?
+ tpm_limit BigInt?
+ rpm_limit BigInt?
+ max_budget Float?
+ budget_duration String?
+ budget_reset_at DateTime?
+ allowed_cache_controls String[] @default([])
+ allowed_routes String[] @default([])
+ model_spend Json @default("{}")
+ model_max_budget Json @default("{}")
+ router_settings Json? @default("{}")
+ budget_id String?
+ organization_id String?
+ object_permission_id String?
+ created_at DateTime? // Original creation timestamp
+ created_by String? // Original creator
+ updated_at DateTime? // Last update timestamp before deletion
+ updated_by String? // Last user who updated before deletion
+ rotation_count Int? @default(0)
+ auto_rotate Boolean? @default(false)
+ rotation_interval String?
+ last_rotation_at DateTime?
+ key_rotation_at DateTime?
+
+ // Deletion metadata
+ deleted_at DateTime @default(now()) @map("deleted_at")
+ deleted_by String? @map("deleted_by") // User who deleted the key
+ deleted_by_api_key String? @map("deleted_by_api_key") // API key hash that performed the deletion
+ litellm_changed_by String? @map("litellm_changed_by") // From litellm-changed-by header if provided
+
+ @@index([token])
+ @@index([deleted_at])
+ @@index([user_id])
+ @@index([team_id])
+ @@index([organization_id])
+ @@index([key_alias])
+ @@index([created_at])
+}
+
model LiteLLM_EndUserTable {
user_id String @id
alias String? // admin-facing alias
diff --git a/litellm-proxy-extras/pyproject.toml b/litellm-proxy-extras/pyproject.toml
index 2952aa6c979..52258ebe2e4 100644
--- a/litellm-proxy-extras/pyproject.toml
+++ b/litellm-proxy-extras/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-proxy-extras"
-version = "0.4.21"
+version = "0.4.23"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
authors = ["BerriAI"]
readme = "README.md"
@@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
-version = "0.4.21"
+version = "0.4.23"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-proxy-extras==",
diff --git a/litellm/__init__.py b/litellm/__init__.py
index 2b7d26de129..9eb3f075d5e 100644
--- a/litellm/__init__.py
+++ b/litellm/__init__.py
@@ -9,7 +9,7 @@ warnings.filterwarnings("ignore", message=".*conflict with protected namespace.*
warnings.filterwarnings(
"ignore", message=".*Accessing the.*attribute on the instance is deprecated.*"
)
-### INIT VARIABLES ########################
+### INIT VARIABLES #########################
import threading
import os
from typing import (
diff --git a/litellm/_logging.py b/litellm/_logging.py
index 73902d2fc5a..b3156b15ba7 100644
--- a/litellm/_logging.py
+++ b/litellm/_logging.py
@@ -133,6 +133,26 @@ ALL_LOGGERS = [
]
+def _get_loggers_to_initialize():
+ """
+ Get all loggers that should be initialized with the JSON handler.
+
+ Includes third-party integration loggers (like langfuse) if they are
+ configured as callbacks.
+ """
+ import litellm
+
+ loggers = list(ALL_LOGGERS)
+
+ # Add langfuse logger if langfuse is being used as a callback
+ langfuse_callbacks = {"langfuse", "langfuse_otel"}
+ all_callbacks = set(litellm.success_callback + litellm.failure_callback)
+ if langfuse_callbacks & all_callbacks:
+ loggers.append(logging.getLogger("langfuse"))
+
+ return loggers
+
+
def _initialize_loggers_with_handler(handler: logging.Handler):
"""
Initialize all loggers with a handler
@@ -140,7 +160,7 @@ def _initialize_loggers_with_handler(handler: logging.Handler):
- Adds a handler to each logger
- Prevents bubbling to parent/root (critical to prevent duplicate JSON logs)
"""
- for lg in ALL_LOGGERS:
+ for lg in _get_loggers_to_initialize():
lg.handlers.clear() # remove any existing handlers
lg.addHandler(handler) # add JSON formatter handler
lg.propagate = False # prevent bubbling to parent/root
diff --git a/litellm/anthropic_interface/exceptions/__init__.py b/litellm/anthropic_interface/exceptions/__init__.py
new file mode 100644
index 00000000000..875b09e3da3
--- /dev/null
+++ b/litellm/anthropic_interface/exceptions/__init__.py
@@ -0,0 +1,19 @@
+"""Anthropic error format utilities."""
+
+from .exception_mapping_utils import (
+ ANTHROPIC_ERROR_TYPE_MAP,
+ AnthropicExceptionMapping,
+)
+from .exceptions import (
+ AnthropicErrorDetail,
+ AnthropicErrorResponse,
+ AnthropicErrorType,
+)
+
+__all__ = [
+ "AnthropicErrorType",
+ "AnthropicErrorDetail",
+ "AnthropicErrorResponse",
+ "ANTHROPIC_ERROR_TYPE_MAP",
+ "AnthropicExceptionMapping",
+]
diff --git a/litellm/anthropic_interface/exceptions/exception_mapping_utils.py b/litellm/anthropic_interface/exceptions/exception_mapping_utils.py
new file mode 100644
index 00000000000..b8a5079a4eb
--- /dev/null
+++ b/litellm/anthropic_interface/exceptions/exception_mapping_utils.py
@@ -0,0 +1,168 @@
+"""
+Utilities for mapping exceptions to Anthropic error format.
+
+Similar to litellm/litellm_core_utils/exception_mapping_utils.py but for Anthropic response format.
+"""
+
+from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
+from typing import Dict, Optional
+
+from .exceptions import AnthropicErrorResponse, AnthropicErrorType
+
+
+# HTTP status code -> Anthropic error type
+# Source: https://docs.anthropic.com/en/api/errors
+ANTHROPIC_ERROR_TYPE_MAP: Dict[int, AnthropicErrorType] = {
+ 400: "invalid_request_error",
+ 401: "authentication_error",
+ 403: "permission_error",
+ 404: "not_found_error",
+ 413: "request_too_large",
+ 429: "rate_limit_error",
+ 500: "api_error",
+ 529: "overloaded_error",
+}
+
+
+class AnthropicExceptionMapping:
+ """
+ Helper class for mapping exceptions to Anthropic error format.
+
+ Similar pattern to ExceptionCheckers in litellm_core_utils/exception_mapping_utils.py
+ """
+
+ @staticmethod
+ def get_error_type(status_code: int) -> AnthropicErrorType:
+ """Map HTTP status code to Anthropic error type."""
+ return ANTHROPIC_ERROR_TYPE_MAP.get(status_code, "api_error")
+
+ @staticmethod
+ def create_error_response(
+ status_code: int,
+ message: str,
+ request_id: Optional[str] = None,
+ ) -> AnthropicErrorResponse:
+ """
+ Create an Anthropic-formatted error response dict.
+
+ Anthropic error format:
+ {
+ "type": "error",
+ "error": {"type": "...", "message": "..."},
+ "request_id": "req_..."
+ }
+ """
+ error_type = AnthropicExceptionMapping.get_error_type(status_code)
+
+ response: AnthropicErrorResponse = {
+ "type": "error",
+ "error": {
+ "type": error_type,
+ "message": message,
+ },
+ }
+
+ if request_id:
+ response["request_id"] = request_id
+
+ return response
+
+ @staticmethod
+ def extract_error_message(raw_message: str) -> str:
+ """
+ Extract error message from various provider response formats.
+
+ Handles:
+ - Bedrock: {"detail": {"message": "..."}}
+ - AWS: {"Message": "..."}
+ - Generic: {"message": "..."}
+ - Plain strings
+ """
+ parsed = safe_json_loads(raw_message)
+ if isinstance(parsed, dict):
+ # Bedrock format
+ if "detail" in parsed and isinstance(parsed["detail"], dict):
+ return parsed["detail"].get("message", raw_message)
+ # AWS/generic format
+ return parsed.get("Message") or parsed.get("message") or raw_message
+ return raw_message
+
+ @staticmethod
+ def _is_anthropic_error_dict(parsed: dict) -> bool:
+ """
+ Check if a parsed dict is in Anthropic error format.
+
+ Anthropic error format:
+ {
+ "type": "error",
+ "error": {"type": "...", "message": "..."}
+ }
+ """
+ return (
+ parsed.get("type") == "error"
+ and isinstance(parsed.get("error"), dict)
+ and "type" in parsed["error"]
+ and "message" in parsed["error"]
+ )
+
+ @staticmethod
+ def _extract_message_from_dict(parsed: dict, raw_message: str) -> str:
+ """
+ Extract error message from a parsed provider-specific dict.
+
+ Handles:
+ - Bedrock: {"detail": {"message": "..."}}
+ - AWS: {"Message": "..."}
+ - Generic: {"message": "..."}
+ """
+ # Bedrock format
+ if "detail" in parsed and isinstance(parsed["detail"], dict):
+ return parsed["detail"].get("message", raw_message)
+ # AWS/generic format
+ return parsed.get("Message") or parsed.get("message") or raw_message
+
+ @staticmethod
+ def transform_to_anthropic_error(
+ status_code: int,
+ raw_message: str,
+ request_id: Optional[str] = None,
+ ) -> AnthropicErrorResponse:
+ """
+ Transform an error message to Anthropic format.
+
+ - If already in Anthropic format: passthrough unchanged
+ - Otherwise: extract message and create Anthropic error
+
+ Parses JSON only once for efficiency.
+
+ Args:
+ status_code: HTTP status code
+ raw_message: Raw error message (may be JSON string or plain text)
+ request_id: Optional request ID to include
+
+ Returns:
+ AnthropicErrorResponse dict
+ """
+ # Try to parse as JSON once
+ parsed: Optional[dict] = safe_json_loads(raw_message)
+ if not isinstance(parsed, dict):
+ parsed = None
+
+ # If parsed and already in Anthropic format - passthrough
+ if parsed is not None and AnthropicExceptionMapping._is_anthropic_error_dict(parsed):
+ # Optionally add request_id if provided and not present
+ if request_id and "request_id" not in parsed:
+ parsed["request_id"] = request_id
+ return parsed # type: ignore
+
+ # Extract message - use parsed dict if available, otherwise raw string
+ if parsed is not None:
+ message = AnthropicExceptionMapping._extract_message_from_dict(parsed, raw_message)
+ else:
+ message = raw_message
+
+ return AnthropicExceptionMapping.create_error_response(
+ status_code=status_code,
+ message=message,
+ request_id=request_id,
+ )
diff --git a/litellm/anthropic_interface/exceptions/exceptions.py b/litellm/anthropic_interface/exceptions/exceptions.py
new file mode 100644
index 00000000000..984390fa702
--- /dev/null
+++ b/litellm/anthropic_interface/exceptions/exceptions.py
@@ -0,0 +1,41 @@
+"""Anthropic error format type definitions."""
+
+from typing_extensions import Literal, Required, TypedDict
+
+
+# Known Anthropic error types
+# Source: https://docs.anthropic.com/en/api/errors
+AnthropicErrorType = Literal[
+ "invalid_request_error",
+ "authentication_error",
+ "permission_error",
+ "not_found_error",
+ "request_too_large",
+ "rate_limit_error",
+ "api_error",
+ "overloaded_error",
+]
+
+
+class AnthropicErrorDetail(TypedDict):
+ """Inner error detail in Anthropic format."""
+
+ type: AnthropicErrorType
+ message: str
+
+
+class AnthropicErrorResponse(TypedDict, total=False):
+ """
+ Anthropic-formatted error response.
+
+ Format:
+ {
+ "type": "error",
+ "error": {"type": "...", "message": "..."},
+ "request_id": "req_..." # optional
+ }
+ """
+
+ type: Required[Literal["error"]]
+ error: Required[AnthropicErrorDetail]
+ request_id: str
diff --git a/litellm/caching/caching.py b/litellm/caching/caching.py
index 82fc37e0cb4..a03bff60686 100644
--- a/litellm/caching/caching.py
+++ b/litellm/caching/caching.py
@@ -78,6 +78,8 @@ class Cache:
"text_completion",
"arerank",
"rerank",
+ "responses",
+ "aresponses",
],
# s3 Bucket, boto3 configuration
azure_account_url: Optional[str] = None,
@@ -796,6 +798,8 @@ def enable_cache(
"text_completion",
"arerank",
"rerank",
+ "responses",
+ "aresponses",
],
**kwargs,
):
@@ -854,6 +858,8 @@ def update_cache(
"text_completion",
"arerank",
"rerank",
+ "responses",
+ "aresponses",
],
**kwargs,
):
diff --git a/litellm/caching/caching_handler.py b/litellm/caching/caching_handler.py
index 628ee118e9c..4e97197a9de 100644
--- a/litellm/caching/caching_handler.py
+++ b/litellm/caching/caching_handler.py
@@ -44,6 +44,7 @@ from litellm.litellm_core_utils.logging_utils import (
_assemble_complete_response_from_streaming_chunks,
)
from litellm.types.caching import CachedEmbedding
+from litellm.types.llms.openai import ResponsesAPIResponse
from litellm.types.rerank import RerankResponse
from litellm.types.utils import (
CachingDetails,
@@ -727,6 +728,12 @@ class LLMCachingHandler:
response_type="audio_transcription",
hidden_params=hidden_params,
)
+ elif (
+ call_type == "aresponses"
+ or call_type == "responses"
+ ) and isinstance(cached_result, dict):
+ # Convert cached dict back to ResponsesAPIResponse object
+ cached_result = ResponsesAPIResponse(**cached_result)
if (
hasattr(cached_result, "_hidden_params")
@@ -826,6 +833,7 @@ class LLMCachingHandler:
or isinstance(result, litellm.EmbeddingResponse)
or isinstance(result, TranscriptionResponse)
or isinstance(result, RerankResponse)
+ or isinstance(result, ResponsesAPIResponse)
):
if (
isinstance(result, EmbeddingResponse)
diff --git a/litellm/constants.py b/litellm/constants.py
index e24186d567e..3bdd943481e 100644
--- a/litellm/constants.py
+++ b/litellm/constants.py
@@ -48,6 +48,11 @@ DEFAULT_REPLICATE_POLLING_DELAY_SECONDS = int(
DEFAULT_IMAGE_TOKEN_COUNT = int(os.getenv("DEFAULT_IMAGE_TOKEN_COUNT", 250))
DEFAULT_IMAGE_WIDTH = int(os.getenv("DEFAULT_IMAGE_WIDTH", 300))
DEFAULT_IMAGE_HEIGHT = int(os.getenv("DEFAULT_IMAGE_HEIGHT", 300))
+# Maximum size for image URL downloads in MB (default 50MB, set to 0 to disable limit)
+# This prevents memory issues from downloading very large images
+# Maps to OpenAI's 50 MB payload limit - requests with images exceeding this size will be rejected
+# Set MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=0 to disable image URL handling entirely
+MAX_IMAGE_URL_DOWNLOAD_SIZE_MB = float(os.getenv("MAX_IMAGE_URL_DOWNLOAD_SIZE_MB", 50))
MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB = int(
os.getenv("MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB", 1024)
) # 1MB = 1024KB
@@ -324,6 +329,11 @@ ANTHROPIC_WEB_SEARCH_TOOL_MAX_USES = {
"medium": 5,
"high": 10,
}
+
+# LiteLLM standard web search tool name
+# Used for web search interception across providers
+LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"
+
DEFAULT_IMAGE_ENDPOINT_MODEL = "dall-e-2"
DEFAULT_VIDEO_ENDPOINT_MODEL = "sora-2"
@@ -1073,6 +1083,13 @@ LITELLM_TRUNCATED_PAYLOAD_FIELD = "litellm_truncated"
########################### LiteLLM Proxy Specific Constants ###########################
########################################################################################
+
+# Standard headers that are always checked for customer/end-user ID (no configuration required)
+# These headers work out-of-the-box for tools like Claude Code that support custom headers
+STANDARD_CUSTOMER_ID_HEADERS = [
+ "x-litellm-customer-id",
+ "x-litellm-end-user-id",
+]
MAX_SPENDLOG_ROWS_TO_QUERY = int(
os.getenv("MAX_SPENDLOG_ROWS_TO_QUERY", 1_000_000)
) # if spendLogs has more than 1M rows, do not query the DB
@@ -1285,3 +1302,20 @@ COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY = int(
########################### RAG Text Splitter Constants ###########################
DEFAULT_CHUNK_SIZE = int(os.getenv("DEFAULT_CHUNK_SIZE", 1000))
DEFAULT_CHUNK_OVERLAP = int(os.getenv("DEFAULT_CHUNK_OVERLAP", 200))
+
+########################### Microsoft SSO Constants ###########################
+MICROSOFT_USER_EMAIL_ATTRIBUTE = str(
+ os.getenv("MICROSOFT_USER_EMAIL_ATTRIBUTE", "userPrincipalName")
+)
+MICROSOFT_USER_DISPLAY_NAME_ATTRIBUTE = str(
+ os.getenv("MICROSOFT_USER_DISPLAY_NAME_ATTRIBUTE", "displayName")
+)
+MICROSOFT_USER_ID_ATTRIBUTE = str(
+ os.getenv("MICROSOFT_USER_ID_ATTRIBUTE", "id")
+)
+MICROSOFT_USER_FIRST_NAME_ATTRIBUTE = str(
+ os.getenv("MICROSOFT_USER_FIRST_NAME_ATTRIBUTE", "givenName")
+)
+MICROSOFT_USER_LAST_NAME_ATTRIBUTE = str(
+ os.getenv("MICROSOFT_USER_LAST_NAME_ATTRIBUTE", "surname")
+)
diff --git a/litellm/containers/main.py b/litellm/containers/main.py
index 625a291fb55..105e999ffe8 100644
--- a/litellm/containers/main.py
+++ b/litellm/containers/main.py
@@ -199,7 +199,13 @@ def create_container(
return response
# get llm provider logic
- litellm_params = GenericLiteLLMParams(**kwargs)
+ # Pass credential params explicitly since they're named args, not in kwargs
+ litellm_params = GenericLiteLLMParams(
+ api_key=api_key,
+ api_base=api_base,
+ api_version=api_version,
+ **kwargs,
+ )
# get provider config
container_provider_config: Optional[BaseContainerConfig] = (
ProviderConfigManager.get_provider_container_config(
@@ -406,7 +412,13 @@ def list_containers(
return response
# get llm provider logic
- litellm_params = GenericLiteLLMParams(**kwargs)
+ # Pass credential params explicitly since they're named args, not in kwargs
+ litellm_params = GenericLiteLLMParams(
+ api_key=api_key,
+ api_base=api_base,
+ api_version=api_version,
+ **kwargs,
+ )
# get provider config
container_provider_config: Optional[BaseContainerConfig] = (
ProviderConfigManager.get_provider_container_config(
@@ -594,7 +606,13 @@ def retrieve_container(
return response
# get llm provider logic
- litellm_params = GenericLiteLLMParams(**kwargs)
+ # Pass credential params explicitly since they're named args, not in kwargs
+ litellm_params = GenericLiteLLMParams(
+ api_key=api_key,
+ api_base=api_base,
+ api_version=api_version,
+ **kwargs,
+ )
# get provider config
container_provider_config: Optional[BaseContainerConfig] = (
ProviderConfigManager.get_provider_container_config(
@@ -774,7 +792,13 @@ def delete_container(
return response
# get llm provider logic
- litellm_params = GenericLiteLLMParams(**kwargs)
+ # Pass credential params explicitly since they're named args, not in kwargs
+ litellm_params = GenericLiteLLMParams(
+ api_key=api_key,
+ api_base=api_base,
+ api_version=api_version,
+ **kwargs,
+ )
# get provider config
container_provider_config: Optional[BaseContainerConfig] = (
ProviderConfigManager.get_provider_container_config(
@@ -968,7 +992,13 @@ def list_container_files(
return response
# get llm provider logic
- litellm_params = GenericLiteLLMParams(**kwargs)
+ # Pass credential params explicitly since they're named args, not in kwargs
+ litellm_params = GenericLiteLLMParams(
+ api_key=api_key,
+ api_base=api_base,
+ api_version=api_version,
+ **kwargs,
+ )
# get provider config
container_provider_config: Optional[BaseContainerConfig] = (
ProviderConfigManager.get_provider_container_config(
@@ -1203,7 +1233,13 @@ def upload_container_file(
return response
# get llm provider logic
- litellm_params = GenericLiteLLMParams(**kwargs)
+ # Pass credential params explicitly since they're named args, not in kwargs
+ litellm_params = GenericLiteLLMParams(
+ api_key=api_key,
+ api_base=api_base,
+ api_version=api_version,
+ **kwargs,
+ )
# get provider config
container_provider_config: Optional[BaseContainerConfig] = (
ProviderConfigManager.get_provider_container_config(
diff --git a/litellm/cost_calculator.py b/litellm/cost_calculator.py
index af7dd078107..f18e8d62aa9 100644
--- a/litellm/cost_calculator.py
+++ b/litellm/cost_calculator.py
@@ -587,6 +587,24 @@ def _model_contains_known_llm_provider(model: str) -> bool:
return _provider_prefix in LlmProvidersSet
+def _get_response_model(completion_response: Any) -> Optional[str]:
+ """
+ Extract the model name from a completion response object.
+
+ Used as a fallback for cost calculation when the input model name
+ doesn't exist in model_cost (e.g., Azure Model Router).
+ """
+ if completion_response is None:
+ return None
+
+ if isinstance(completion_response, BaseModel):
+ return getattr(completion_response, "model", None)
+ elif isinstance(completion_response, dict):
+ return completion_response.get("model", None)
+
+ return None
+
+
def _get_usage_object(
completion_response: Any,
) -> Optional[Usage]:
@@ -933,7 +951,7 @@ def completion_cost( # noqa: PLR0915
router_model_id=router_model_id,
)
- potential_model_names = [selected_model]
+ potential_model_names = [selected_model, _get_response_model(completion_response)]
if model is not None:
potential_model_names.append(model)
diff --git a/litellm/exceptions.py b/litellm/exceptions.py
index d963cac754c..c2443626b8d 100644
--- a/litellm/exceptions.py
+++ b/litellm/exceptions.py
@@ -125,16 +125,20 @@ class BadRequestError(openai.BadRequestError): # type: ignore
self.model = model
self.llm_provider = llm_provider
self.litellm_debug_info = litellm_debug_info
- response = httpx.Response(
+ self.max_retries = max_retries
+ self.num_retries = num_retries
+ _response_headers = (
+ getattr(response, "headers", None) if response is not None else None
+ )
+ self.response = httpx.Response(
status_code=self.status_code,
+ headers=_response_headers,
request=httpx.Request(
method="GET", url="https://litellm.ai"
), # mock request object
)
- self.max_retries = max_retries
- self.num_retries = num_retries
super().__init__(
- self.message, response=response, body=body
+ self.message, response=self.response, body=body
) # Call the base class constructor with the parameters it needs
def __str__(self):
@@ -368,13 +372,11 @@ class ContextWindowExceededError(BadRequestError): # type: ignore
self.model = model
self.llm_provider = llm_provider
self.litellm_debug_info = litellm_debug_info
- request = httpx.Request(method="POST", url="https://api.openai.com/v1")
- self.response = httpx.Response(status_code=400, request=request)
super().__init__(
message=message,
model=self.model, # type: ignore
llm_provider=self.llm_provider, # type: ignore
- response=self.response,
+ response=response,
litellm_debug_info=self.litellm_debug_info,
) # Call the base class constructor with the parameters it needs
@@ -457,18 +459,14 @@ class ContentPolicyViolationError(BadRequestError): # type: ignore
self.model = model
self.llm_provider = llm_provider
self.litellm_debug_info = litellm_debug_info
- request = httpx.Request(method="POST", url="https://api.openai.com/v1")
- self.response = httpx.Response(status_code=400, request=request)
self.provider_specific_fields = provider_specific_fields
-
super().__init__(
message=self.message,
model=self.model, # type: ignore
llm_provider=self.llm_provider, # type: ignore
- response=self.response,
+ response=response,
litellm_debug_info=self.litellm_debug_info,
) # Call the base class constructor with the parameters it needs
-
def __str__(self):
return self._transform_error_to_string()
@@ -898,9 +896,15 @@ class LiteLLMUnknownProvider(BadRequestError):
class GuardrailRaisedException(Exception):
- def __init__(self, guardrail_name: Optional[str] = None, message: str = ""):
+ def __init__(
+ self,
+ guardrail_name: Optional[str] = None,
+ message: str = "",
+ should_wrap_with_default_message: bool = True,
+ ):
+ default_message = f"Guardrail raised an exception, Guardrail: {guardrail_name}, Message: {message}"
self.guardrail_name = guardrail_name
- self.message = f"Guardrail raised an exception, Guardrail: {guardrail_name}, Message: {message}"
+ self.message = default_message if should_wrap_with_default_message else message
super().__init__(self.message)
diff --git a/litellm/images/main.py b/litellm/images/main.py
index cf588cbcf0f..1b09c20d350 100644
--- a/litellm/images/main.py
+++ b/litellm/images/main.py
@@ -404,6 +404,7 @@ def image_generation( # noqa: PLR0915
litellm.LlmProviders.STABILITY,
litellm.LlmProviders.RUNWAYML,
litellm.LlmProviders.VERTEX_AI,
+ litellm.LlmProviders.OPENROUTER
):
if image_generation_config is None:
raise ValueError(
diff --git a/litellm/integrations/azure_storage/azure_storage.py b/litellm/integrations/azure_storage/azure_storage.py
index b4362665a4c..85f91199c1c 100644
--- a/litellm/integrations/azure_storage/azure_storage.py
+++ b/litellm/integrations/azure_storage/azure_storage.py
@@ -1,5 +1,4 @@
import asyncio
-import json
import os
import time
from litellm._uuid import uuid
@@ -15,6 +14,7 @@ from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
+from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.types.utils import StandardLoggingPayload
@@ -168,7 +168,7 @@ class AzureBlobStorageLogger(CustomBatchLogger):
llm_provider=httpxSpecialProvider.LoggingCallback
)
json_payload = (
- json.dumps(payload) + "\n"
+ safe_dumps(payload) + "\n"
) # Add newline for each log entry
payload_bytes = json_payload.encode("utf-8")
filename = f"{payload.get('id') or str(uuid.uuid4())}.json"
@@ -384,7 +384,7 @@ class AzureBlobStorageLogger(CustomBatchLogger):
await file_client.create_file()
# Content to append
- content = json.dumps(payload).encode("utf-8")
+ content = safe_dumps(payload).encode("utf-8")
# Append content to the file
await file_client.append_data(data=content, offset=0, length=len(content))
diff --git a/litellm/integrations/custom_logger.py b/litellm/integrations/custom_logger.py
index 4c4e6fa6342..12243a19184 100644
--- a/litellm/integrations/custom_logger.py
+++ b/litellm/integrations/custom_logger.py
@@ -33,10 +33,9 @@ from litellm.types.utils import (
if TYPE_CHECKING:
from fastapi import HTTPException
-
- from litellm.caching.caching import DualCache
from opentelemetry.trace import Span as _Span
+ from litellm.caching.caching import DualCache
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.mcp import (
@@ -144,6 +143,34 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
async def async_log_pre_api_call(self, model, messages, kwargs):
pass
+ async def async_pre_request_hook(
+ self, model: str, messages: List, kwargs: Dict
+ ) -> Optional[Dict]:
+ """
+ Hook called before making the API request to allow modifying request parameters.
+
+ This is specifically designed for modifying the request before it's sent to the provider.
+ Unlike async_log_pre_api_call (which is for logging), this hook is meant for transformations.
+
+ Args:
+ model: The model name
+ messages: The messages list
+ kwargs: The request parameters (tools, stream, temperature, etc.)
+
+ Returns:
+ Optional[Dict]: Modified kwargs to use for the request, or None if no modifications
+
+ Example:
+ ```python
+ async def async_pre_request_hook(self, model, messages, kwargs):
+ # Convert native tools to standard format
+ if kwargs.get("tools"):
+ kwargs["tools"] = convert_tools(kwargs["tools"])
+ return kwargs
+ ```
+ """
+ pass
+
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
pass
@@ -484,6 +511,138 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
"""
return None
+ #########################################################
+ # AGENTIC LOOP HOOKS (for litellm.messages + future completion support)
+ #########################################################
+
+ async def async_should_run_agentic_loop(
+ self,
+ response: Any,
+ model: str,
+ messages: List[Dict],
+ tools: Optional[List[Dict]],
+ stream: bool,
+ custom_llm_provider: str,
+ kwargs: Dict,
+ ) -> Tuple[bool, Dict]:
+ """
+ Hook to determine if agentic loop should be executed.
+
+ Called after receiving response from model, before returning to user.
+
+ USE CASE: Enables transparent server-side tool execution for models that
+ don't natively support server-side tools. User makes ONE API call and gets
+ back the final answer - the agentic loop happens transparently on the server.
+
+ Example use cases:
+ - WebSearch: Intercept WebSearch tool calls for Bedrock/Claude, execute
+ litellm.search(), return final answer with search results
+ - Code execution: Execute code in sandboxed environment, return results
+ - Database queries: Execute queries server-side, return data to model
+ - API calls: Make external API calls and inject responses back into context
+
+ Flow:
+ 1. User calls litellm.messages.acreate(tools=[...])
+ 2. Model responds with tool_use
+ 3. THIS HOOK checks if tool should run server-side
+ 4. If True, async_run_agentic_loop executes the tool
+ 5. User receives final answer (never sees intermediate tool_use)
+
+ Args:
+ response: Response from model (AnthropicMessagesResponse or AsyncIterator)
+ model: Model name
+ messages: Original messages sent to model
+ tools: List of tool definitions from request
+ stream: Whether response is streaming
+ custom_llm_provider: Provider name (e.g., "bedrock", "anthropic")
+ kwargs: Additional request parameters
+
+ Returns:
+ (should_run, tools):
+ should_run: True if agentic loop should execute
+ tools: Dict with tool_calls and metadata for execution
+
+ Example:
+ # Detect WebSearch tool call
+ if has_websearch_tool_use(response):
+ return True, {
+ "tool_calls": extract_tool_calls(response),
+ "tool_type": "websearch"
+ }
+ return False, {}
+ """
+ return False, {}
+
+ async def async_run_agentic_loop(
+ self,
+ tools: Dict,
+ model: str,
+ messages: List[Dict],
+ response: Any,
+ anthropic_messages_provider_config: Any,
+ anthropic_messages_optional_request_params: Dict,
+ logging_obj: "LiteLLMLoggingObj",
+ stream: bool,
+ kwargs: Dict,
+ ) -> Any:
+ """
+ Hook to execute agentic loop based on context from should_run hook.
+
+ Called only if async_messages_should_run_agentic_loop returns True.
+
+ USE CASE: Execute server-side tools and orchestrate the agentic loop to
+ return a complete answer to the user in a single API call.
+
+ What to do here:
+ 1. Extract tool calls from tools dict
+ 2. Execute the tools (litellm.search, code execution, DB queries, etc.)
+ 3. Build assistant message with tool_use blocks
+ 4. Build user message with tool_result blocks containing results
+ 5. Make follow-up litellm.messages.acreate() call with results
+ 6. Return the final response
+
+ Args:
+ tools: Dict from async_should_run_agentic_loop
+ Contains tool_calls and metadata
+ model: Model name
+ messages: Original messages sent to model
+ response: Original response from model (with tool_use)
+ anthropic_messages_provider_config: Provider config for making requests
+ anthropic_messages_optional_request_params: Request parameters (tools, etc.)
+ logging_obj: LiteLLM logging object
+ stream: Whether response is streaming
+ kwargs: Additional request parameters
+
+ Returns:
+ Final response after executing agentic loop
+ (AnthropicMessagesResponse with final answer)
+
+ Example:
+ # Extract tool calls
+ tool_calls = agentic_context["tool_calls"]
+
+ # Execute searches in parallel
+ search_results = await asyncio.gather(
+ *[litellm.asearch(tc["input"]["query"]) for tc in tool_calls]
+ )
+
+ # Build messages with tool results
+ assistant_msg = {"role": "assistant", "content": [...tool_use blocks...]}
+ user_msg = {"role": "user", "content": [...tool_result blocks...]}
+
+ # Make follow-up request
+ from litellm.anthropic_interface import messages
+ final_response = await messages.acreate(
+ model=model,
+ messages=messages + [assistant_msg, user_msg],
+ max_tokens=anthropic_messages_optional_request_params.get("max_tokens"),
+ **anthropic_messages_optional_request_params
+ )
+
+ return final_response
+ """
+ pass
+
# Useful helpers for custom logger classes
def truncate_standard_logging_payload_content(
diff --git a/litellm/integrations/langsmith.py b/litellm/integrations/langsmith.py
index 570b78f2927..5893f14105d 100644
--- a/litellm/integrations/langsmith.py
+++ b/litellm/integrations/langsmith.py
@@ -134,6 +134,13 @@ class LangsmithLogger(CustomBatchLogger):
"metadata"
] # ensure logged metadata is json serializable
+ extra_metadata = dict(metadata)
+ requester_metadata = extra_metadata.get("requester_metadata")
+ if requester_metadata and isinstance(requester_metadata, dict):
+ for key in ("session_id", "thread_id", "conversation_id"):
+ if key in requester_metadata and key not in extra_metadata:
+ extra_metadata[key] = requester_metadata[key]
+
data = {
"name": run_name,
"run_type": "llm", # this should always be llm, since litellm always logs llm calls. Langsmith allow us to log "chain"
@@ -143,7 +150,7 @@ class LangsmithLogger(CustomBatchLogger):
"start_time": payload["startTime"],
"end_time": payload["endTime"],
"tags": payload["request_tags"],
- "extra": metadata,
+ "extra": extra_metadata,
}
if payload["error_str"] is not None and payload["status"] == "failure":
@@ -439,9 +446,9 @@ class LangsmithLogger(CustomBatchLogger):
return log_queue_by_credentials
def _get_sampling_rate_to_use_for_request(self, kwargs: Dict[str, Any]) -> float:
- standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
- kwargs.get("standard_callback_dynamic_params", None)
- )
+ standard_callback_dynamic_params: Optional[
+ StandardCallbackDynamicParams
+ ] = kwargs.get("standard_callback_dynamic_params", None)
sampling_rate: float = self.sampling_rate
if standard_callback_dynamic_params is not None:
_sampling_rate = standard_callback_dynamic_params.get(
@@ -461,9 +468,9 @@ class LangsmithLogger(CustomBatchLogger):
Otherwise, use the default credentials.
"""
- standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
- kwargs.get("standard_callback_dynamic_params", None)
- )
+ standard_callback_dynamic_params: Optional[
+ StandardCallbackDynamicParams
+ ] = kwargs.get("standard_callback_dynamic_params", None)
if standard_callback_dynamic_params is not None:
credentials = self.get_credentials_from_env(
langsmith_api_key=standard_callback_dynamic_params.get(
diff --git a/litellm/integrations/opentelemetry.py b/litellm/integrations/opentelemetry.py
index 7e0cfab617b..a223925d59a 100644
--- a/litellm/integrations/opentelemetry.py
+++ b/litellm/integrations/opentelemetry.py
@@ -594,9 +594,9 @@ class OpenTelemetry(CustomLogger):
def _get_dynamic_otel_headers_from_kwargs(self, kwargs) -> Optional[dict]:
"""Extract dynamic headers from kwargs if available."""
- standard_callback_dynamic_params: Optional[
- StandardCallbackDynamicParams
- ] = kwargs.get("standard_callback_dynamic_params")
+ standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
+ kwargs.get("standard_callback_dynamic_params")
+ )
if not standard_callback_dynamic_params:
return None
@@ -797,7 +797,7 @@ class OpenTelemetry(CustomLogger):
and self._token_usage_histogram
):
in_attrs = {**common_attrs, "gen_ai.token.type": "input"}
- out_attrs = {**common_attrs, "gen_ai.token.type": "completion"}
+ out_attrs = {**common_attrs, "gen_ai.token.type": "output"}
self._token_usage_histogram.record(
usage.get("prompt_tokens", 0), attributes=in_attrs
)
@@ -1488,21 +1488,21 @@ class OpenTelemetry(CustomLogger):
if usage:
self.safe_set_attribute(
span=span,
- key=SpanAttributes.LLM_USAGE_TOTAL_TOKENS.value,
+ key=SpanAttributes.GEN_AI_USAGE_TOTAL_TOKENS.value,
value=usage.get("total_tokens"),
)
# The number of tokens used in the LLM response (completion).
self.safe_set_attribute(
span=span,
- key=SpanAttributes.LLM_USAGE_COMPLETION_TOKENS.value,
+ key=SpanAttributes.GEN_AI_USAGE_OUTPUT_TOKENS.value,
value=usage.get("completion_tokens"),
)
# The number of tokens used in the LLM prompt.
self.safe_set_attribute(
span=span,
- key=SpanAttributes.LLM_USAGE_PROMPT_TOKENS.value,
+ key=SpanAttributes.GEN_AI_USAGE_INPUT_TOKENS.value,
value=usage.get("prompt_tokens"),
)
@@ -1520,53 +1520,75 @@ class OpenTelemetry(CustomLogger):
self.set_tools_attributes(span, tools)
if kwargs.get("messages"):
- for idx, prompt in enumerate(kwargs.get("messages")):
- if prompt.get("role"):
- self.safe_set_attribute(
- span=span,
- key=f"{SpanAttributes.LLM_PROMPTS.value}.{idx}.role",
- value=prompt.get("role"),
- )
+ transformed_messages = (
+ self._transform_messages_to_otel_semantic_conventions(
+ kwargs.get("messages")
+ )
+ )
+ self.safe_set_attribute(
+ span=span,
+ key=SpanAttributes.GEN_AI_INPUT_MESSAGES.value,
+ value=safe_dumps(transformed_messages),
+ )
- if prompt.get("content"):
- if not isinstance(prompt.get("content"), str):
- prompt["content"] = str(prompt.get("content"))
- self.safe_set_attribute(
- span=span,
- key=f"{SpanAttributes.LLM_PROMPTS.value}.{idx}.content",
- value=prompt.get("content"),
- )
+ if kwargs.get("system_instructions"):
+ transformed_system_instructions = (
+ self._transform_messages_to_otel_semantic_conventions(
+ kwargs.get("system_instructions")
+ )
+ )
+ self.safe_set_attribute(
+ span=span,
+ key=SpanAttributes.GEN_AI_SYSTEM_INSTRUCTIONS.value,
+ value=safe_dumps(transformed_system_instructions),
+ )
+
+ self.safe_set_attribute(
+ span=span,
+ key=SpanAttributes.GEN_AI_OPERATION_NAME.value,
+ value=(
+ "chat"
+ if standard_logging_payload.get("call_type") == "completion"
+ else standard_logging_payload.get("call_type") or "chat"
+ ),
+ )
+
+ if standard_logging_payload.get("request_id"):
+ self.safe_set_attribute(
+ span=span,
+ key=SpanAttributes.GEN_AI_REQUEST_ID.value,
+ value=standard_logging_payload.get("request_id"),
+ )
#############################################
########## LLM Response Attributes ##########
#############################################
if response_obj is not None:
if response_obj.get("choices"):
+ transformed_choices = (
+ self._transform_choices_to_otel_semantic_conventions(
+ response_obj.get("choices")
+ )
+ )
+ self.safe_set_attribute(
+ span=span,
+ key=SpanAttributes.GEN_AI_OUTPUT_MESSAGES.value,
+ value=safe_dumps(transformed_choices),
+ )
+
+ finish_reasons = []
+ for idx, choice in enumerate(response_obj.get("choices")):
+ if choice.get("finish_reason"):
+ finish_reasons.append(choice.get("finish_reason"))
+
+ if finish_reasons:
+ self.safe_set_attribute(
+ span=span,
+ key=SpanAttributes.GEN_AI_RESPONSE_FINISH_REASONS.value,
+ value=safe_dumps(finish_reasons),
+ )
+
for idx, choice in enumerate(response_obj.get("choices")):
if choice.get("finish_reason"):
- self.safe_set_attribute(
- span=span,
- key=f"{SpanAttributes.LLM_COMPLETIONS.value}.{idx}.finish_reason",
- value=choice.get("finish_reason"),
- )
- if choice.get("message"):
- if choice.get("message").get("role"):
- self.safe_set_attribute(
- span=span,
- key=f"{SpanAttributes.LLM_COMPLETIONS.value}.{idx}.role",
- value=choice.get("message").get("role"),
- )
- if choice.get("message").get("content"):
- if not isinstance(
- choice.get("message").get("content"), str
- ):
- choice["message"]["content"] = str(
- choice.get("message").get("content")
- )
- self.safe_set_attribute(
- span=span,
- key=f"{SpanAttributes.LLM_COMPLETIONS.value}.{idx}.content",
- value=choice.get("message").get("content"),
- )
message = choice.get("message")
tool_calls = message.get("tool_calls")
@@ -1608,6 +1630,66 @@ class OpenTelemetry(CustomLogger):
primitive_value = self._cast_as_primitive_value_type(value)
span.set_attribute(key, primitive_value)
+ def _transform_messages_to_otel_semantic_conventions(
+ self, messages: Union[List[dict], str]
+ ) -> List[dict]:
+ """
+ Transforms LiteLLM/OpenAI style messages into OTEL GenAI 1.38 compliant format.
+ OTEL expects a 'parts' array instead of a single 'content' string.
+ """
+ if isinstance(messages, str):
+ # Handle system_instructions passed as a string
+ return [
+ {"role": "system", "parts": [{"type": "text", "content": messages}]}
+ ]
+
+ transformed = []
+ for msg in messages:
+ role = msg.get("role", "user")
+ content = msg.get("content", "")
+ parts = []
+
+ if isinstance(content, str):
+ parts.append({"type": "text", "content": content})
+ elif isinstance(content, list):
+ # Handle multi-modal content if necessary
+ for part in content:
+ if isinstance(part, dict):
+ parts.append(part)
+ else:
+ parts.append({"type": "text", "content": str(part)})
+
+ transformed_msg = {"role": role, "parts": parts}
+ if "id" in msg:
+ transformed_msg["id"] = msg["id"]
+ if "tool_calls" in msg:
+ transformed_msg["tool_calls"] = msg["tool_calls"]
+ if "tool_call_id" in msg:
+ transformed_msg["tool_call_id"] = msg["tool_call_id"]
+ transformed.append(transformed_msg)
+
+ return transformed
+
+ def _transform_choices_to_otel_semantic_conventions(
+ self, choices: List[dict]
+ ) -> List[dict]:
+ """
+ Transforms choices into OTEL GenAI 1.38 compliant format for output.messages.
+ """
+ transformed = []
+ for choice in choices:
+ message = choice.get("message") or {}
+ finish_reason = choice.get("finish_reason")
+
+ transformed_msg = self._transform_messages_to_otel_semantic_conventions(
+ [message]
+ )[0]
+ if finish_reason:
+ transformed_msg["finish_reason"] = finish_reason
+
+ transformed.append(transformed_msg)
+ return transformed
+
def set_raw_request_attributes(self, span: Span, kwargs, response_obj):
try:
kwargs.get("optional_params", {})
diff --git a/litellm/integrations/prometheus.py b/litellm/integrations/prometheus.py
index 2385edc5297..b490c21174f 100644
--- a/litellm/integrations/prometheus.py
+++ b/litellm/integrations/prometheus.py
@@ -21,7 +21,12 @@ from typing import (
import litellm
from litellm._logging import print_verbose, verbose_logger
from litellm.integrations.custom_logger import CustomLogger
-from litellm.proxy._types import LiteLLM_TeamTable, UserAPIKeyAuth
+from litellm.proxy._types import (
+ LiteLLM_DeletedVerificationToken,
+ LiteLLM_TeamTable,
+ LiteLLM_UserTable,
+ UserAPIKeyAuth,
+)
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name
from litellm.types.utils import StandardLoggingPayload
@@ -52,7 +57,7 @@ def _get_cached_end_user_id_for_cost_tracking():
class PrometheusLogger(CustomLogger):
# Class variables or attributes
- def __init__(
+ def __init__( # noqa: PLR0915
self,
**kwargs,
):
@@ -193,6 +198,30 @@ class PrometheusLogger(CustomLogger):
),
)
+ # Remaining Budget for User
+ self.litellm_remaining_user_budget_metric = self._gauge_factory(
+ "litellm_remaining_user_budget_metric",
+ "Remaining budget for user",
+ labelnames=self.get_labels_for_metric(
+ "litellm_remaining_user_budget_metric"
+ ),
+ )
+
+ # Max Budget for User
+ self.litellm_user_max_budget_metric = self._gauge_factory(
+ "litellm_user_max_budget_metric",
+ "Maximum budget set for user",
+ labelnames=self.get_labels_for_metric("litellm_user_max_budget_metric"),
+ )
+
+ self.litellm_user_budget_remaining_hours_metric = self._gauge_factory(
+ "litellm_user_budget_remaining_hours_metric",
+ "Remaining hours for user budget to be reset",
+ labelnames=self.get_labels_for_metric(
+ "litellm_user_budget_remaining_hours_metric"
+ ),
+ )
+
########################################
# LiteLLM Virtual API KEY metrics
########################################
@@ -960,6 +989,7 @@ class PrometheusLogger(CustomLogger):
user_api_key_alias=user_api_key_alias,
litellm_params=litellm_params,
response_cost=response_cost,
+ user_id=user_id,
)
# set proxy virtual key rpm/tpm metrics
@@ -1120,6 +1150,7 @@ class PrometheusLogger(CustomLogger):
user_api_key_alias: Optional[str],
litellm_params: dict,
response_cost: float,
+ user_id: Optional[str] = None,
):
_team_spend = litellm_params.get("metadata", {}).get(
"user_api_key_team_spend", None
@@ -1134,6 +1165,14 @@ class PrometheusLogger(CustomLogger):
_api_key_max_budget = litellm_params.get("metadata", {}).get(
"user_api_key_max_budget", None
)
+
+ _user_spend = litellm_params.get("metadata", {}).get(
+ "user_api_key_user_spend", None
+ )
+ _user_max_budget = litellm_params.get("metadata", {}).get(
+ "user_api_key_user_max_budget", None
+ )
+
await self._set_api_key_budget_metrics_after_api_request(
user_api_key=user_api_key,
user_api_key_alias=user_api_key_alias,
@@ -1150,6 +1189,13 @@ class PrometheusLogger(CustomLogger):
response_cost=response_cost,
)
+ await self._set_user_budget_metrics_after_api_request(
+ user_id=user_id,
+ user_spend=_user_spend,
+ user_max_budget=_user_max_budget,
+ response_cost=response_cost,
+ )
+
def _increment_top_level_request_and_spend_metrics(
self,
end_user_id: Optional[str],
@@ -2112,7 +2158,7 @@ class PrometheusLogger(CustomLogger):
self,
data_fetch_function: Callable[..., Awaitable[Tuple[List[Any], Optional[int]]]],
set_metrics_function: Callable[[List[Any]], Awaitable[None]],
- data_type: Literal["teams", "keys"],
+ data_type: Literal["teams", "keys", "users"],
):
"""
Generic method to initialize budget metrics for teams or API keys.
@@ -2204,7 +2250,7 @@ class PrometheusLogger(CustomLogger):
async def fetch_keys(
page_size: int, page: int
- ) -> Tuple[List[Union[str, UserAPIKeyAuth]], Optional[int]]:
+ ) -> Tuple[List[Union[str, UserAPIKeyAuth, LiteLLM_DeletedVerificationToken]], Optional[int]]:
key_list_response = await _list_key_helper(
prisma_client=prisma_client,
page=page,
@@ -2229,6 +2275,37 @@ class PrometheusLogger(CustomLogger):
data_type="keys",
)
+ async def _initialize_user_budget_metrics(self):
+ """
+ Initialize user budget metrics by reusing the generic pagination logic.
+ """
+ from litellm.proxy._types import LiteLLM_UserTable
+ from litellm.proxy.proxy_server import prisma_client
+
+ if prisma_client is None:
+ verbose_logger.debug(
+ "Prometheus: skipping user metrics initialization, DB not initialized"
+ )
+ return
+
+ async def fetch_users(
+ page_size: int, page: int
+ ) -> Tuple[List[LiteLLM_UserTable], Optional[int]]:
+ skip = (page - 1) * page_size
+ users = await prisma_client.db.litellm_usertable.find_many(
+ skip=skip,
+ take=page_size,
+ order={"created_at": "desc"},
+ )
+ total_count = await prisma_client.db.litellm_usertable.count()
+ return users, total_count
+
+ await self._initialize_budget_metrics(
+ data_fetch_function=fetch_users,
+ set_metrics_function=self._set_user_list_budget_metrics,
+ data_type="users",
+ )
+
async def initialize_remaining_budget_metrics(self):
"""
Handler for initializing remaining budget metrics for all teams to avoid metric discrepancies.
@@ -2261,11 +2338,12 @@ class PrometheusLogger(CustomLogger):
async def _initialize_remaining_budget_metrics(self):
"""
- Helper to initialize remaining budget metrics for all teams and API keys.
+ Helper to initialize remaining budget metrics for all teams, API keys, and users.
"""
- verbose_logger.debug("Emitting key, team budget metrics....")
+ verbose_logger.debug("Emitting key, team, user budget metrics....")
await self._initialize_team_budget_metrics()
await self._initialize_api_key_budget_metrics()
+ await self._initialize_user_budget_metrics()
async def _set_key_list_budget_metrics(
self, keys: List[Union[str, UserAPIKeyAuth]]
@@ -2280,6 +2358,11 @@ class PrometheusLogger(CustomLogger):
for team in teams:
self._set_team_budget_metrics(team)
+ async def _set_user_list_budget_metrics(self, users: List[LiteLLM_UserTable]):
+ """Helper function to set budget metrics for a list of users"""
+ for user in users:
+ self._set_user_budget_metrics(user)
+
async def _set_team_budget_metrics_after_api_request(
self,
user_api_team: Optional[str],
@@ -2497,6 +2580,122 @@ class PrometheusLogger(CustomLogger):
return user_api_key_dict
+ async def _set_user_budget_metrics_after_api_request(
+ self,
+ user_id: Optional[str],
+ user_spend: Optional[float],
+ user_max_budget: Optional[float],
+ response_cost: float,
+ ):
+ """
+ Set user budget metrics after an LLM API request
+
+ - Assemble a LiteLLM_UserTable object
+ - looks up user info from db if not available in metadata
+ - Set user budget metrics
+ """
+ if user_id:
+ user_object = await self._assemble_user_object(
+ user_id=user_id,
+ spend=user_spend,
+ max_budget=user_max_budget,
+ response_cost=response_cost,
+ )
+
+ self._set_user_budget_metrics(user_object)
+
+ async def _assemble_user_object(
+ self,
+ user_id: str,
+ spend: Optional[float],
+ max_budget: Optional[float],
+ response_cost: float,
+ ) -> LiteLLM_UserTable:
+ """
+ Assemble a LiteLLM_UserTable object
+
+ for fields not available in metadata, we fetch from db
+ Fields not available in metadata:
+ - `budget_reset_at`
+ """
+ from litellm.proxy.auth.auth_checks import get_user_object
+ from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
+
+ _total_user_spend = (spend or 0) + response_cost
+ user_object = LiteLLM_UserTable(
+ user_id=user_id,
+ spend=_total_user_spend,
+ max_budget=max_budget,
+ )
+ try:
+ user_info = await get_user_object(
+ user_id=user_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ user_id_upsert=False,
+ check_db_only=True,
+ )
+ except Exception as e:
+ verbose_logger.debug(
+ f"[Non-Blocking] Prometheus: Error getting user info: {str(e)}"
+ )
+ return user_object
+
+ if user_info:
+ user_object.budget_reset_at = user_info.budget_reset_at
+
+ return user_object
+
+ def _set_user_budget_metrics(
+ self,
+ user: LiteLLM_UserTable,
+ ):
+ """
+ Set user budget metrics for a single user
+
+ - Remaining Budget
+ - Max Budget
+ - Budget Reset At
+ """
+ enum_values = UserAPIKeyLabelValues(
+ user=user.user_id,
+ )
+
+ _labels = prometheus_label_factory(
+ supported_enum_labels=self.get_labels_for_metric(
+ metric_name="litellm_remaining_user_budget_metric"
+ ),
+ enum_values=enum_values,
+ )
+ self.litellm_remaining_user_budget_metric.labels(**_labels).set(
+ self._safe_get_remaining_budget(
+ max_budget=user.max_budget,
+ spend=user.spend,
+ )
+ )
+
+ if user.max_budget is not None:
+ _labels = prometheus_label_factory(
+ supported_enum_labels=self.get_labels_for_metric(
+ metric_name="litellm_user_max_budget_metric"
+ ),
+ enum_values=enum_values,
+ )
+ self.litellm_user_max_budget_metric.labels(**_labels).set(user.max_budget)
+
+ if user.budget_reset_at is not None:
+ _labels = prometheus_label_factory(
+ supported_enum_labels=self.get_labels_for_metric(
+ metric_name="litellm_user_budget_remaining_hours_metric"
+ ),
+ enum_values=enum_values,
+ )
+ self.litellm_user_budget_remaining_hours_metric.labels(**_labels).set(
+ self._get_remaining_hours_for_budget_reset(
+ budget_reset_at=user.budget_reset_at
+ )
+ )
+
def _get_remaining_hours_for_budget_reset(self, budget_reset_at: datetime) -> float:
"""
Get remaining hours for budget reset
diff --git a/litellm/integrations/websearch_interception/ARCHITECTURE.md b/litellm/integrations/websearch_interception/ARCHITECTURE.md
new file mode 100644
index 00000000000..3aa0a1558d7
--- /dev/null
+++ b/litellm/integrations/websearch_interception/ARCHITECTURE.md
@@ -0,0 +1,292 @@
+# WebSearch Interception Architecture
+
+Server-side WebSearch tool execution for models that don't natively support it (e.g., Bedrock/Claude).
+
+## How It Works
+
+User makes **ONE** `litellm.messages.acreate()` call → Gets final answer with search results.
+The agentic loop happens transparently on the server.
+
+## LiteLLM Standard Web Search Tool
+
+LiteLLM defines a standard web search tool format (`litellm_web_search`) that all native provider tools are converted to. This enables consistent interception across providers.
+
+**Standard Tool Definition** (defined in `tools.py`):
+```python
+{
+ "name": "litellm_web_search",
+ "description": "Search the web for information...",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "query": {"type": "string", "description": "The search query"}
+ },
+ "required": ["query"]
+ }
+}
+```
+
+**Tool Name Constant**: `LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"` (defined in `litellm/constants.py`)
+
+### Supported Tool Formats
+
+The interception system automatically detects and handles:
+
+| Tool Format | Example | Provider | Detection Method | Future-Proof |
+|-------------|---------|----------|------------------|-------------|
+| **LiteLLM Standard** | `name="litellm_web_search"` | Any | Direct name match | N/A |
+| **Anthropic Native** | `type="web_search_20250305"` | Bedrock, Claude API | Type prefix: `startswith("web_search_")` | ✅ Yes (web_search_2026, etc.) |
+| **Claude Code CLI** | `name="web_search"`, `type="web_search_20250305"` | Claude Code | Name + type check | ✅ Yes (version-agnostic) |
+| **Legacy** | `name="WebSearch"` | Custom | Name match | N/A (backwards compat) |
+
+**Future Compatibility**: The `startswith("web_search_")` check in `tools.py` automatically supports future Anthropic web search versions.
+
+### Claude Code CLI Integration
+
+Claude Code (Anthropic's official CLI) sends web search requests using Anthropic's native tool format:
+
+```python
+{
+ "type": "web_search_20250305",
+ "name": "web_search",
+ "max_uses": 8
+}
+```
+
+**What Happens:**
+1. Claude Code sends native `web_search_20250305` tool to LiteLLM proxy
+2. LiteLLM intercepts and converts to `litellm_web_search` standard format
+3. Bedrock receives converted tool (NOT native format)
+4. Model returns `tool_use` block for `litellm_web_search` (not `server_tool_use`)
+5. LiteLLM's agentic loop intercepts the `tool_use`
+6. Executes `litellm.asearch()` using configured provider (Perplexity, Tavily, etc.)
+7. Returns final answer to Claude Code user
+
+**Without Interception**: Bedrock would receive native tool → try to execute natively → return `web_search_tool_result_error` with `invalid_tool_input`
+
+**With Interception**: LiteLLM converts → Bedrock returns tool_use → LiteLLM executes search → Returns final answer ✅
+
+### Native Tool Conversion
+
+Native tools are converted to LiteLLM standard format **before** sending to the provider:
+
+1. **Conversion Point** (`litellm/llms/anthropic/experimental_pass_through/messages/handler.py`):
+ - In `anthropic_messages()` function (lines 60-127)
+ - Runs BEFORE the API request is made
+ - Detects native web search tools using `is_web_search_tool()`
+ - Converts to `litellm_web_search` format using `get_litellm_web_search_tool()`
+ - Prevents provider from executing search natively (avoids `web_search_tool_result_error`)
+
+2. **Response Detection** (`transformation.py`):
+ - Detects `tool_use` blocks with any web search tool name
+ - Handles: `litellm_web_search`, `WebSearch`, `web_search`
+ - Extracts search queries for execution
+
+**Example Conversion**:
+```python
+# Input (Claude Code's native tool)
+{
+ "type": "web_search_20250305",
+ "name": "web_search",
+ "max_uses": 8
+}
+
+# Output (LiteLLM standard)
+{
+ "name": "litellm_web_search",
+ "description": "Search the web for information...",
+ "input_schema": {...}
+}
+```
+
+---
+
+## Request Flow
+
+### Without Interception (Client-Side)
+User manually handles tool execution:
+1. User calls `litellm.messages.acreate()` → Gets `tool_use` response
+2. User executes `litellm.asearch()`
+3. User calls `litellm.messages.acreate()` again with results
+4. User gets final answer
+
+**Result**: 2 API calls, manual tool execution
+
+### With Interception (Server-Side)
+Server handles tool execution automatically:
+
+```mermaid
+sequenceDiagram
+ participant User
+ participant Messages as litellm.messages.acreate()
+ participant Handler as llm_http_handler.py
+ participant Logger as WebSearchInterceptionLogger
+ participant Router as proxy_server.llm_router
+ participant Search as litellm.asearch()
+ participant Provider as Bedrock API
+
+ User->>Messages: acreate(tools=[WebSearch])
+ Messages->>Handler: async_anthropic_messages_handler()
+ Handler->>Provider: Request
+ Provider-->>Handler: Response (tool_use)
+ Handler->>Logger: async_should_run_agentic_loop()
+ Logger->>Logger: Detect WebSearch tool_use
+ Logger-->>Handler: (True, tools)
+ Handler->>Logger: async_run_agentic_loop(tools)
+ Logger->>Router: Get search_provider from search_tools
+ Router-->>Logger: search_provider
+ Logger->>Search: asearch(query, provider)
+ Search-->>Logger: Search results
+ Logger->>Logger: Build tool_result message
+ Logger->>Messages: acreate() with results
+ Messages->>Provider: Request with search results
+ Provider-->>Messages: Final answer
+ Messages-->>Logger: Final response
+ Logger-->>Handler: Final response
+ Handler-->>User: Final answer (with search results)
+```
+
+**Result**: 1 API call from user, server handles agentic loop
+
+---
+
+## Key Components
+
+| Component | File | Purpose |
+|-----------|------|---------|
+| **WebSearchInterceptionLogger** | `handler.py` | CustomLogger that implements agentic loop hooks |
+| **Tool Standardization** | `tools.py` | Standard tool definition, detection, and utilities |
+| **Tool Name Constant** | `constants.py` | `LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"` |
+| **Tool Conversion** | `anthropic/.../ handler.py` | Converts native tools to LiteLLM standard before API call |
+| **Transformation Logic** | `transformation.py` | Detect tool_use, build tool_result messages, format search responses |
+| **Agentic Loop Hooks** | `integrations/custom_logger.py` | Base hooks: `async_should_run_agentic_loop()`, `async_run_agentic_loop()` |
+| **Hook Orchestration** | `llms/custom_httpx/llm_http_handler.py` | `_call_agentic_completion_hooks()` - calls hooks after response |
+| **Router Search Tools** | `proxy/proxy_server.py` | `llm_router.search_tools` - configured search providers |
+| **Search Endpoints** | `proxy/search_endpoints/endpoints.py` | Router logic for selecting search provider |
+
+---
+
+## Configuration
+
+```python
+from litellm.integrations.websearch_interception import (
+ WebSearchInterceptionLogger,
+ get_litellm_web_search_tool,
+)
+from litellm.types.utils import LlmProviders
+
+# Enable for Bedrock with specific search tool
+litellm.callbacks = [
+ WebSearchInterceptionLogger(
+ enabled_providers=[LlmProviders.BEDROCK],
+ search_tool_name="my-perplexity-tool" # Optional: uses router's first tool if None
+ )
+]
+
+# Make request with LiteLLM standard tool (recommended)
+response = await litellm.messages.acreate(
+ model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
+ messages=[{"role": "user", "content": "What is LiteLLM?"}],
+ tools=[get_litellm_web_search_tool()], # LiteLLM standard
+ max_tokens=1024,
+ stream=True # Auto-converted to non-streaming
+)
+
+# OR send native tools - they're auto-converted to LiteLLM standard
+response = await litellm.messages.acreate(
+ model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
+ messages=[{"role": "user", "content": "What is LiteLLM?"}],
+ tools=[{
+ "type": "web_search_20250305", # Native Anthropic format
+ "name": "web_search",
+ "max_uses": 8
+ }],
+ max_tokens=1024,
+)
+```
+
+---
+
+## Streaming Support
+
+WebSearch interception works transparently with both streaming and non-streaming requests.
+
+**How streaming is handled:**
+1. User makes request with `stream=True` and WebSearch tool
+2. Before API call, `anthropic_messages()` detects WebSearch + interception enabled
+3. Converts `stream=True` → `stream=False` internally
+4. Agentic loop executes with non-streaming responses
+5. Final response returned to user (non-streaming)
+
+**Why this approach:**
+- Server-side agentic loops require consuming full responses to detect tool_use
+- User opts into this behavior by enabling WebSearch interception
+- Provides seamless experience without client changes
+
+**Testing:**
+- **Non-streaming**: `test_websearch_interception_e2e.py`
+- **Streaming**: `test_websearch_interception_streaming_e2e.py`
+
+---
+
+## Search Provider Selection
+
+1. If `search_tool_name` specified → Look up in `llm_router.search_tools`
+2. If not found or None → Use first available search tool
+3. If no router or no tools → Fallback to `perplexity`
+
+Example router config:
+```yaml
+search_tools:
+ - search_tool_name: "my-perplexity-tool"
+ litellm_params:
+ search_provider: "perplexity"
+ - search_tool_name: "my-tavily-tool"
+ litellm_params:
+ search_provider: "tavily"
+```
+
+---
+
+## Message Flow
+
+### Initial Request
+```python
+messages = [{"role": "user", "content": "What is LiteLLM?"}]
+tools = [{"name": "WebSearch", ...}]
+```
+
+### First API Call (Internal)
+**Response**: `tool_use` with `name="WebSearch"`, `input={"query": "what is litellm"}`
+
+### Server Processing
+1. Logger detects WebSearch tool_use
+2. Looks up search provider from router
+3. Executes `litellm.asearch(query="what is litellm", search_provider="perplexity")`
+4. Gets results: `"Title: LiteLLM Docs\nURL: docs.litellm.ai\n..."`
+
+### Follow-Up Request (Internal)
+```python
+messages = [
+ {"role": "user", "content": "What is LiteLLM?"},
+ {"role": "assistant", "content": [{"type": "tool_use", ...}]},
+ {"role": "user", "content": [{"type": "tool_result", "content": "search results..."}]}
+]
+```
+
+### User Receives
+```python
+response.content[0].text
+# "Based on the search results, LiteLLM is a unified interface..."
+```
+
+---
+
+## Testing
+
+**E2E Tests**:
+- `test_websearch_interception_e2e.py` - Non-streaming real API calls to Bedrock
+- `test_websearch_interception_streaming_e2e.py` - Streaming real API calls to Bedrock
+
+**Unit Tests**: `test_websearch_interception.py`
+Mocked tests for tool detection, provider filtering, edge cases.
diff --git a/litellm/integrations/websearch_interception/__init__.py b/litellm/integrations/websearch_interception/__init__.py
new file mode 100644
index 00000000000..f5b1963c1cf
--- /dev/null
+++ b/litellm/integrations/websearch_interception/__init__.py
@@ -0,0 +1,20 @@
+"""
+WebSearch Interception Module
+
+Provides server-side WebSearch tool execution for models that don't natively
+support server-side tool calling (e.g., Bedrock/Claude).
+"""
+
+from litellm.integrations.websearch_interception.handler import (
+ WebSearchInterceptionLogger,
+)
+from litellm.integrations.websearch_interception.tools import (
+ get_litellm_web_search_tool,
+ is_web_search_tool,
+)
+
+__all__ = [
+ "WebSearchInterceptionLogger",
+ "get_litellm_web_search_tool",
+ "is_web_search_tool",
+]
diff --git a/litellm/integrations/websearch_interception/handler.py b/litellm/integrations/websearch_interception/handler.py
new file mode 100644
index 00000000000..943a2bb4f36
--- /dev/null
+++ b/litellm/integrations/websearch_interception/handler.py
@@ -0,0 +1,553 @@
+"""
+WebSearch Interception Handler
+
+CustomLogger that intercepts WebSearch tool calls for models that don't
+natively support web search (e.g., Bedrock/Claude) and executes them
+server-side using litellm router's search tools.
+"""
+
+import asyncio
+from typing import Any, Dict, List, Optional, Tuple, Union, cast
+
+import litellm
+from litellm._logging import verbose_logger
+from litellm.anthropic_interface import messages as anthropic_messages
+from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
+from litellm.integrations.custom_logger import CustomLogger
+from litellm.integrations.websearch_interception.tools import (
+ get_litellm_web_search_tool,
+ is_web_search_tool,
+)
+from litellm.integrations.websearch_interception.transformation import (
+ WebSearchTransformation,
+)
+from litellm.types.integrations.websearch_interception import (
+ WebSearchInterceptionConfig,
+)
+from litellm.types.utils import LlmProviders
+
+
+class WebSearchInterceptionLogger(CustomLogger):
+ """
+ CustomLogger that intercepts WebSearch tool calls for models that don't
+ natively support web search.
+
+ Implements agentic loop:
+ 1. Detects WebSearch tool_use in model response
+ 2. Executes litellm.asearch() for each query using router's search tools
+ 3. Makes follow-up request with search results
+ 4. Returns final response
+ """
+
+ def __init__(
+ self,
+ enabled_providers: Optional[List[Union[LlmProviders, str]]] = None,
+ search_tool_name: Optional[str] = None,
+ ):
+ """
+ Args:
+ enabled_providers: List of LLM providers to enable interception for.
+ Use LlmProviders enum values (e.g., [LlmProviders.BEDROCK])
+ Default: [LlmProviders.BEDROCK]
+ search_tool_name: Name of search tool configured in router's search_tools.
+ If None, will attempt to use first available search tool.
+ """
+ super().__init__()
+ # Convert enum values to strings for comparison
+ if enabled_providers is None:
+ self.enabled_providers = [LlmProviders.BEDROCK.value]
+ else:
+ self.enabled_providers = [
+ p.value if isinstance(p, LlmProviders) else p
+ for p in enabled_providers
+ ]
+ self.search_tool_name = search_tool_name
+ self._request_has_websearch = False # Track if current request has web search
+
+ async def async_pre_call_deployment_hook(
+ self, kwargs: Dict[str, Any], call_type: Optional[Any]
+ ) -> Optional[dict]:
+ """
+ Pre-call hook to convert native Anthropic web_search tools to regular tools.
+
+ This prevents Bedrock from trying to execute web search server-side (which fails).
+ Instead, we convert it to a regular tool so the model returns tool_use blocks
+ that we can intercept and execute ourselves.
+ """
+ # Check if this is for an enabled provider
+ custom_llm_provider = kwargs.get("litellm_params", {}).get("custom_llm_provider", "")
+ if custom_llm_provider not in self.enabled_providers:
+ return None
+
+ # Check if request has tools with native web_search
+ tools = kwargs.get("tools")
+ if not tools:
+ return None
+
+ # Check if any tool is a web search tool (native or already LiteLLM standard)
+ has_websearch = any(is_web_search_tool(t) for t in tools)
+
+ if not has_websearch:
+ return None
+
+ verbose_logger.debug(
+ "WebSearchInterception: Converting native web_search tools to LiteLLM standard"
+ )
+
+ # Convert native/custom web_search tools to LiteLLM standard
+ converted_tools = []
+ for tool in tools:
+ if is_web_search_tool(tool):
+ # Convert to LiteLLM standard web search tool
+ converted_tool = get_litellm_web_search_tool()
+ converted_tools.append(converted_tool)
+ verbose_logger.debug(
+ f"WebSearchInterception: Converted {tool.get('name', 'unknown')} "
+ f"(type={tool.get('type', 'none')}) to {LITELLM_WEB_SEARCH_TOOL_NAME}"
+ )
+ else:
+ # Keep other tools as-is
+ converted_tools.append(tool)
+
+ # Return modified kwargs with converted tools
+ return {"tools": converted_tools}
+
+ @classmethod
+ def from_config_yaml(
+ cls, config: WebSearchInterceptionConfig
+ ) -> "WebSearchInterceptionLogger":
+ """
+ Initialize WebSearchInterceptionLogger from proxy config.yaml parameters.
+
+ Args:
+ config: Configuration dictionary from litellm_settings.websearch_interception_params
+
+ Returns:
+ Configured WebSearchInterceptionLogger instance
+
+ Example:
+ From proxy_config.yaml:
+ litellm_settings:
+ websearch_interception_params:
+ enabled_providers: ["bedrock"]
+ search_tool_name: "my-perplexity-search"
+
+ Usage:
+ config = litellm_settings.get("websearch_interception_params", {})
+ logger = WebSearchInterceptionLogger.from_config_yaml(config)
+ """
+ # Extract parameters from config
+ enabled_providers_str = config.get("enabled_providers", None)
+ search_tool_name = config.get("search_tool_name", None)
+
+ # Convert string provider names to LlmProviders enum values
+ enabled_providers: Optional[List[Union[LlmProviders, str]]] = None
+ if enabled_providers_str is not None:
+ enabled_providers = []
+ for provider in enabled_providers_str:
+ try:
+ # Try to convert string to LlmProviders enum
+ provider_enum = LlmProviders(provider)
+ enabled_providers.append(provider_enum)
+ except ValueError:
+ # If conversion fails, keep as string
+ enabled_providers.append(provider)
+
+ return cls(
+ enabled_providers=enabled_providers,
+ search_tool_name=search_tool_name,
+ )
+
+ async def async_pre_request_hook(
+ self, model: str, messages: List[Dict], kwargs: Dict
+ ) -> Optional[Dict]:
+ """
+ Pre-request hook to convert native web search tools to LiteLLM standard.
+
+ This hook is called before the API request is made, allowing us to:
+ 1. Detect native web search tools (web_search_20250305, etc.)
+ 2. Convert them to LiteLLM standard format (litellm_web_search)
+ 3. Convert stream=True to stream=False for interception
+
+ This prevents providers like Bedrock from trying to execute web search
+ natively (which fails), and ensures our agentic loop can intercept tool_use.
+
+ Returns:
+ Modified kwargs dict with converted tools, or None if no modifications needed
+ """
+ # Check if this request is for an enabled provider
+ custom_llm_provider = kwargs.get("litellm_params", {}).get(
+ "custom_llm_provider", ""
+ )
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Pre-request hook called"
+ f" - custom_llm_provider={custom_llm_provider}"
+ f" - enabled_providers={self.enabled_providers}"
+ )
+
+ if custom_llm_provider not in self.enabled_providers:
+ verbose_logger.debug(
+ f"WebSearchInterception: Skipping - provider {custom_llm_provider} not in {self.enabled_providers}"
+ )
+ return None
+
+ # Check if request has tools
+ tools = kwargs.get("tools")
+ if not tools:
+ return None
+
+ # Check if any tool is a web search tool
+ has_websearch = any(is_web_search_tool(t) for t in tools)
+ if not has_websearch:
+ return None
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Pre-request hook triggered for provider={custom_llm_provider}"
+ )
+
+ # Convert native web search tools to LiteLLM standard
+ converted_tools = []
+ for tool in tools:
+ if is_web_search_tool(tool):
+ standard_tool = get_litellm_web_search_tool()
+ converted_tools.append(standard_tool)
+ verbose_logger.debug(
+ f"WebSearchInterception: Converted {tool.get('name', 'unknown')} "
+ f"(type={tool.get('type', 'none')}) to {LITELLM_WEB_SEARCH_TOOL_NAME}"
+ )
+ else:
+ converted_tools.append(tool)
+
+ # Update kwargs with converted tools
+ kwargs["tools"] = converted_tools
+ verbose_logger.debug(
+ f"WebSearchInterception: Tools after conversion: {[t.get('name') for t in converted_tools]}"
+ )
+
+ # Convert stream=True to stream=False for WebSearch interception
+ if kwargs.get("stream"):
+ verbose_logger.debug(
+ "WebSearchInterception: Converting stream=True to stream=False"
+ )
+ kwargs["stream"] = False
+ kwargs["_websearch_interception_converted_stream"] = True
+
+ return kwargs
+
+ async def async_should_run_agentic_loop(
+ self,
+ response: Any,
+ model: str,
+ messages: List[Dict],
+ tools: Optional[List[Dict]],
+ stream: bool,
+ custom_llm_provider: str,
+ kwargs: Dict,
+ ) -> Tuple[bool, Dict]:
+ """Check if WebSearch tool interception is needed"""
+
+ verbose_logger.debug(f"WebSearchInterception: Hook called! provider={custom_llm_provider}, stream={stream}")
+ verbose_logger.debug(f"WebSearchInterception: Response type: {type(response)}")
+
+ # Check if provider should be intercepted
+ # Note: custom_llm_provider is already normalized by get_llm_provider()
+ # (e.g., "bedrock/invoke/..." -> "bedrock")
+ if custom_llm_provider not in self.enabled_providers:
+ verbose_logger.debug(
+ f"WebSearchInterception: Skipping provider {custom_llm_provider} (not in enabled list: {self.enabled_providers})"
+ )
+ return False, {}
+
+ # Check if tools include any web search tool (LiteLLM standard or native)
+ has_websearch_tool = any(is_web_search_tool(t) for t in (tools or []))
+ if not has_websearch_tool:
+ verbose_logger.debug(
+ "WebSearchInterception: No web search tool in request"
+ )
+ return False, {}
+
+ # Detect WebSearch tool_use in response
+ should_intercept, tool_calls = WebSearchTransformation.transform_request(
+ response=response,
+ stream=stream,
+ )
+
+ if not should_intercept:
+ verbose_logger.debug(
+ "WebSearchInterception: No WebSearch tool_use detected in response"
+ )
+ return False, {}
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Detected {len(tool_calls)} WebSearch tool call(s), executing agentic loop"
+ )
+
+ # Return tools dict with tool calls
+ tools_dict = {
+ "tool_calls": tool_calls,
+ "tool_type": "websearch",
+ "provider": custom_llm_provider,
+ }
+ return True, tools_dict
+
+ async def async_run_agentic_loop(
+ self,
+ tools: Dict,
+ model: str,
+ messages: List[Dict],
+ response: Any,
+ anthropic_messages_provider_config: Any,
+ anthropic_messages_optional_request_params: Dict,
+ logging_obj: Any,
+ stream: bool,
+ kwargs: Dict,
+ ) -> Any:
+ """Execute agentic loop with WebSearch execution"""
+
+ tool_calls = tools["tool_calls"]
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Executing agentic loop for {len(tool_calls)} search(es)"
+ )
+
+ return await self._execute_agentic_loop(
+ model=model,
+ messages=messages,
+ tool_calls=tool_calls,
+ anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
+ logging_obj=logging_obj,
+ stream=stream,
+ kwargs=kwargs,
+ )
+
+ async def _execute_agentic_loop(
+ self,
+ model: str,
+ messages: List[Dict],
+ tool_calls: List[Dict],
+ anthropic_messages_optional_request_params: Dict,
+ logging_obj: Any,
+ stream: bool,
+ kwargs: Dict,
+ ) -> Any:
+ """Execute litellm.search() and make follow-up request"""
+
+ # Extract search queries from tool_use blocks
+ search_tasks = []
+ for tool_call in tool_calls:
+ query = tool_call["input"].get("query")
+ if query:
+ verbose_logger.debug(
+ f"WebSearchInterception: Queuing search for query='{query}'"
+ )
+ search_tasks.append(self._execute_search(query))
+ else:
+ verbose_logger.warning(
+ f"WebSearchInterception: Tool call {tool_call['id']} has no query"
+ )
+ # Add empty result for tools without query
+ search_tasks.append(self._create_empty_search_result())
+
+ # Execute searches in parallel
+ verbose_logger.debug(
+ f"WebSearchInterception: Executing {len(search_tasks)} search(es) in parallel"
+ )
+ search_results = await asyncio.gather(*search_tasks, return_exceptions=True)
+
+ # Handle any exceptions in search results
+ final_search_results: List[str] = []
+ for i, result in enumerate(search_results):
+ if isinstance(result, Exception):
+ verbose_logger.error(
+ f"WebSearchInterception: Search {i} failed with error: {str(result)}"
+ )
+ final_search_results.append(
+ f"Search failed: {str(result)}"
+ )
+ elif isinstance(result, str):
+ # Explicitly cast to str for type checker
+ final_search_results.append(cast(str, result))
+ else:
+ # Should never happen, but handle for type safety
+ verbose_logger.warning(
+ f"WebSearchInterception: Unexpected result type {type(result)} at index {i}"
+ )
+ final_search_results.append(str(result))
+
+ # Build assistant and user messages using transformation
+ assistant_message, user_message = WebSearchTransformation.transform_response(
+ tool_calls=tool_calls,
+ search_results=final_search_results,
+ )
+
+ # Make follow-up request with search results
+ follow_up_messages = messages + [assistant_message, user_message]
+
+ verbose_logger.debug(
+ "WebSearchInterception: Making follow-up request with search results"
+ )
+ verbose_logger.debug(
+ f"WebSearchInterception: Follow-up messages count: {len(follow_up_messages)}"
+ )
+ verbose_logger.debug(
+ f"WebSearchInterception: Last message (tool_result): {user_message}"
+ )
+
+ # Use anthropic_messages.acreate for follow-up request
+ try:
+ # Extract max_tokens from optional params or kwargs
+ # max_tokens is a required parameter for anthropic_messages.acreate()
+ max_tokens = anthropic_messages_optional_request_params.get(
+ "max_tokens",
+ kwargs.get("max_tokens", 1024) # Default to 1024 if not found
+ )
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Using max_tokens={max_tokens} for follow-up request"
+ )
+
+ # Create a copy of optional params without max_tokens (since we pass it explicitly)
+ optional_params_without_max_tokens = {
+ k: v for k, v in anthropic_messages_optional_request_params.items()
+ if k != 'max_tokens'
+ }
+
+ # Get model from logging_obj.model_call_details["agentic_loop_params"]
+ # This preserves the full model name with provider prefix (e.g., "bedrock/invoke/...")
+ full_model_name = model
+ if logging_obj is not None:
+ agentic_params = logging_obj.model_call_details.get("agentic_loop_params", {})
+ full_model_name = agentic_params.get("model", model)
+ verbose_logger.debug(
+ f"WebSearchInterception: Using model name: {full_model_name}"
+ )
+
+ final_response = await anthropic_messages.acreate(
+ max_tokens=max_tokens,
+ messages=follow_up_messages,
+ model=full_model_name,
+ **optional_params_without_max_tokens,
+ **kwargs,
+ )
+ verbose_logger.debug(
+ f"WebSearchInterception: Follow-up request completed, response type: {type(final_response)}"
+ )
+ verbose_logger.debug(
+ f"WebSearchInterception: Final response: {final_response}"
+ )
+ return final_response
+ except Exception as e:
+ verbose_logger.exception(
+ f"WebSearchInterception: Follow-up request failed: {str(e)}"
+ )
+ raise
+
+ async def _execute_search(self, query: str) -> str:
+ """Execute a single web search using router's search tools"""
+ try:
+ # Import router from proxy_server
+ try:
+ from litellm.proxy.proxy_server import llm_router
+ except ImportError:
+ verbose_logger.warning(
+ "WebSearchInterception: Could not import llm_router from proxy_server, "
+ "falling back to direct litellm.asearch() with perplexity"
+ )
+ llm_router = None
+
+ # Determine search provider from router's search_tools
+ search_provider: Optional[str] = None
+ if llm_router is not None and hasattr(llm_router, "search_tools"):
+ if self.search_tool_name:
+ # Find specific search tool by name
+ matching_tools = [
+ tool for tool in llm_router.search_tools
+ if tool.get("search_tool_name") == self.search_tool_name
+ ]
+ if matching_tools:
+ search_tool = matching_tools[0]
+ search_provider = search_tool.get("litellm_params", {}).get("search_provider")
+ verbose_logger.debug(
+ f"WebSearchInterception: Found search tool '{self.search_tool_name}' "
+ f"with provider '{search_provider}'"
+ )
+ else:
+ verbose_logger.warning(
+ f"WebSearchInterception: Search tool '{self.search_tool_name}' not found in router, "
+ "falling back to first available or perplexity"
+ )
+
+ # If no specific tool or not found, use first available
+ if not search_provider and llm_router.search_tools:
+ first_tool = llm_router.search_tools[0]
+ search_provider = first_tool.get("litellm_params", {}).get("search_provider")
+ verbose_logger.debug(
+ f"WebSearchInterception: Using first available search tool with provider '{search_provider}'"
+ )
+
+ # Fallback to perplexity if no router or no search tools configured
+ if not search_provider:
+ search_provider = "perplexity"
+ verbose_logger.debug(
+ "WebSearchInterception: No search tools configured in router, "
+ f"using default provider '{search_provider}'"
+ )
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Executing search for '{query}' using provider '{search_provider}'"
+ )
+ result = await litellm.asearch(
+ query=query, search_provider=search_provider
+ )
+
+ # Format using transformation function
+ search_result_text = WebSearchTransformation.format_search_response(result)
+
+ verbose_logger.debug(
+ f"WebSearchInterception: Search completed for '{query}', got {len(search_result_text)} chars"
+ )
+ return search_result_text
+ except Exception as e:
+ verbose_logger.error(
+ f"WebSearchInterception: Search failed for '{query}': {str(e)}"
+ )
+ raise
+
+ async def _create_empty_search_result(self) -> str:
+ """Create an empty search result for tool calls without queries"""
+ return "No search query provided"
+
+ @staticmethod
+ def initialize_from_proxy_config(
+ litellm_settings: Dict[str, Any],
+ callback_specific_params: Dict[str, Any],
+ ) -> "WebSearchInterceptionLogger":
+ """
+ Static method to initialize WebSearchInterceptionLogger from proxy config.
+
+ Used in callback_utils.py to simplify initialization logic.
+
+ Args:
+ litellm_settings: Dictionary containing litellm_settings from proxy_config.yaml
+ callback_specific_params: Dictionary containing callback-specific parameters
+
+ Returns:
+ Configured WebSearchInterceptionLogger instance
+
+ Example:
+ From callback_utils.py:
+ websearch_obj = WebSearchInterceptionLogger.initialize_from_proxy_config(
+ litellm_settings=litellm_settings,
+ callback_specific_params=callback_specific_params
+ )
+ """
+ # Get websearch_interception_params from litellm_settings or callback_specific_params
+ websearch_params: WebSearchInterceptionConfig = {}
+ if "websearch_interception_params" in litellm_settings:
+ websearch_params = litellm_settings["websearch_interception_params"]
+ elif "websearch_interception" in callback_specific_params:
+ websearch_params = callback_specific_params["websearch_interception"]
+
+ # Use classmethod to initialize from config
+ return WebSearchInterceptionLogger.from_config_yaml(websearch_params)
diff --git a/litellm/integrations/websearch_interception/tools.py b/litellm/integrations/websearch_interception/tools.py
new file mode 100644
index 00000000000..4f8b7372fe3
--- /dev/null
+++ b/litellm/integrations/websearch_interception/tools.py
@@ -0,0 +1,95 @@
+"""
+LiteLLM Web Search Tool Definition
+
+This module defines the standard web search tool used across LiteLLM.
+Native provider tools (like Anthropic's web_search_20250305) are converted
+to this format for consistent interception and execution.
+"""
+
+from typing import Any, Dict
+
+from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
+
+
+def get_litellm_web_search_tool() -> Dict[str, Any]:
+ """
+ Get the standard LiteLLM web search tool definition.
+
+ This is the canonical tool definition that all native web search tools
+ (like Anthropic's web_search_20250305, Claude Code's web_search, etc.)
+ are converted to for interception.
+
+ Returns:
+ Dict containing the Anthropic-style tool definition with:
+ - name: Tool name
+ - description: What the tool does
+ - input_schema: JSON schema for tool parameters
+
+ Example:
+ >>> tool = get_litellm_web_search_tool()
+ >>> tool['name']
+ 'litellm_web_search'
+ """
+ return {
+ "name": LITELLM_WEB_SEARCH_TOOL_NAME,
+ "description": (
+ "Search the web for information. Use this when you need current "
+ "information or answers to questions that require up-to-date data."
+ ),
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "query": {
+ "type": "string",
+ "description": "The search query to execute"
+ }
+ },
+ "required": ["query"]
+ }
+ }
+
+
+def is_web_search_tool(tool: Dict[str, Any]) -> bool:
+ """
+ Check if a tool is a web search tool (native or LiteLLM standard).
+
+ Detects:
+ - LiteLLM standard: name == "litellm_web_search"
+ - Anthropic native: type starts with "web_search_" (e.g., "web_search_20250305")
+ - Claude Code: name == "web_search" with a type field
+ - Custom: name == "WebSearch" (legacy format)
+
+ Args:
+ tool: Tool dictionary to check
+
+ Returns:
+ True if tool is a web search tool
+
+ Example:
+ >>> is_web_search_tool({"name": "litellm_web_search"})
+ True
+ >>> is_web_search_tool({"type": "web_search_20250305", "name": "web_search"})
+ True
+ >>> is_web_search_tool({"name": "calculator"})
+ False
+ """
+ tool_name = tool.get("name", "")
+ tool_type = tool.get("type", "")
+
+ # Check for LiteLLM standard tool
+ if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME:
+ return True
+
+ # Check for native Anthropic web_search_* types
+ if tool_type.startswith("web_search_"):
+ return True
+
+ # Check for Claude Code's web_search with a type field
+ if tool_name == "web_search" and tool_type:
+ return True
+
+ # Check for legacy WebSearch format
+ if tool_name == "WebSearch":
+ return True
+
+ return False
diff --git a/litellm/integrations/websearch_interception/transformation.py b/litellm/integrations/websearch_interception/transformation.py
new file mode 100644
index 00000000000..313358822a5
--- /dev/null
+++ b/litellm/integrations/websearch_interception/transformation.py
@@ -0,0 +1,189 @@
+"""
+WebSearch Tool Transformation
+
+Transforms between Anthropic tool_use format and LiteLLM search format.
+"""
+
+from typing import Any, Dict, List, Tuple
+
+from litellm._logging import verbose_logger
+from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
+from litellm.llms.base_llm.search.transformation import SearchResponse
+
+
+class WebSearchTransformation:
+ """
+ Transformation class for WebSearch tool interception.
+
+ Handles transformation between:
+ - Anthropic tool_use format → LiteLLM search requests
+ - LiteLLM SearchResponse → Anthropic tool_result format
+ """
+
+ @staticmethod
+ def transform_request(
+ response: Any,
+ stream: bool,
+ ) -> Tuple[bool, List[Dict]]:
+ """
+ Transform Anthropic response to extract WebSearch tool calls.
+
+ Detects if response contains WebSearch tool_use blocks and extracts
+ the search queries for execution.
+
+ Args:
+ response: Model response (dict or AnthropicMessagesResponse)
+ stream: Whether response is streaming
+
+ Returns:
+ (has_websearch, tool_calls):
+ has_websearch: True if WebSearch tool_use found
+ tool_calls: List of tool_use dicts with id, name, input
+
+ Note:
+ Streaming requests are handled by converting stream=True to stream=False
+ in the WebSearchInterceptionLogger.async_log_pre_api_call hook before
+ the API request is made. This means by the time this method is called,
+ streaming requests have already been converted to non-streaming.
+ """
+ if stream:
+ # This should not happen in practice since we convert streaming to non-streaming
+ # in async_log_pre_api_call, but keep this check for safety
+ verbose_logger.warning(
+ "WebSearchInterception: Unexpected streaming response, skipping interception"
+ )
+ return False, []
+
+ # Parse non-streaming response
+ return WebSearchTransformation._detect_from_non_streaming_response(response)
+
+ @staticmethod
+ def _detect_from_non_streaming_response(
+ response: Any,
+ ) -> Tuple[bool, List[Dict]]:
+ """Parse non-streaming response for WebSearch tool_use"""
+
+ # Handle both dict and object responses
+ if isinstance(response, dict):
+ content = response.get("content", [])
+ else:
+ if not hasattr(response, "content"):
+ verbose_logger.debug(
+ "WebSearchInterception: Response has no content attribute"
+ )
+ return False, []
+ content = response.content or []
+
+ if not content:
+ verbose_logger.debug(
+ "WebSearchInterception: Response has empty content"
+ )
+ return False, []
+
+ # Find all WebSearch tool_use blocks
+ tool_calls = []
+ for block in content:
+ # Handle both dict and object blocks
+ if isinstance(block, dict):
+ block_type = block.get("type")
+ block_name = block.get("name")
+ block_id = block.get("id")
+ block_input = block.get("input", {})
+ else:
+ block_type = getattr(block, "type", None)
+ block_name = getattr(block, "name", None)
+ block_id = getattr(block, "id", None)
+ block_input = getattr(block, "input", {})
+
+ # Check for LiteLLM standard or legacy web search tools
+ # Handles: litellm_web_search, WebSearch, web_search
+ if block_type == "tool_use" and block_name in (
+ LITELLM_WEB_SEARCH_TOOL_NAME, "WebSearch", "web_search"
+ ):
+ # Convert to dict for easier handling
+ tool_call = {
+ "id": block_id,
+ "type": "tool_use",
+ "name": block_name, # Preserve original name
+ "input": block_input,
+ }
+ tool_calls.append(tool_call)
+ verbose_logger.debug(
+ f"WebSearchInterception: Found {block_name} tool_use with id={tool_call['id']}"
+ )
+
+ return len(tool_calls) > 0, tool_calls
+
+ @staticmethod
+ def transform_response(
+ tool_calls: List[Dict],
+ search_results: List[str],
+ ) -> Tuple[Dict, Dict]:
+ """
+ Transform LiteLLM search results to Anthropic tool_result format.
+
+ Builds the assistant and user messages needed for the agentic loop
+ follow-up request.
+
+ Args:
+ tool_calls: List of tool_use dicts from transform_request
+ search_results: List of search result strings (one per tool_call)
+
+ Returns:
+ (assistant_message, user_message):
+ assistant_message: Message with tool_use blocks
+ user_message: Message with tool_result blocks
+ """
+ # Build assistant message with tool_use blocks
+ assistant_message = {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_use",
+ "id": tc["id"],
+ "name": tc["name"],
+ "input": tc["input"],
+ }
+ for tc in tool_calls
+ ],
+ }
+
+ # Build user message with tool_result blocks
+ user_message = {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": tool_calls[i]["id"],
+ "content": search_results[i],
+ }
+ for i in range(len(tool_calls))
+ ],
+ }
+
+ return assistant_message, user_message
+
+ @staticmethod
+ def format_search_response(result: SearchResponse) -> str:
+ """
+ Format SearchResponse as text for tool_result content.
+
+ Args:
+ result: SearchResponse from litellm.asearch()
+
+ Returns:
+ Formatted text with Title, URL, Snippet for each result
+ """
+ # Convert SearchResponse to string
+ if hasattr(result, "results") and result.results:
+ # Format results as text
+ search_result_text = "\n\n".join(
+ [
+ f"Title: {r.title}\nURL: {r.url}\nSnippet: {r.snippet}"
+ for r in result.results
+ ]
+ )
+ else:
+ search_result_text = str(result)
+
+ return search_result_text
diff --git a/litellm/litellm_core_utils/exception_mapping_utils.py b/litellm/litellm_core_utils/exception_mapping_utils.py
index 1517d1e776d..107cdf39bfa 100644
--- a/litellm/litellm_core_utils/exception_mapping_utils.py
+++ b/litellm/litellm_core_utils/exception_mapping_utils.py
@@ -197,12 +197,22 @@ def extract_and_raise_litellm_exception(
exception_name = exception_name.strip().replace("litellm.", "")
raised_exception_obj = getattr(litellm, exception_name, None)
if raised_exception_obj:
- raise raised_exception_obj(
- message=error_str,
- llm_provider=custom_llm_provider,
- model=model,
- response=response,
- )
+ # Try with response parameter first, fall back to without it
+ # Some exceptions (e.g., APIConnectionError) don't accept response param
+ try:
+ raise raised_exception_obj(
+ message=error_str,
+ llm_provider=custom_llm_provider,
+ model=model,
+ response=response,
+ )
+ except TypeError:
+ # Exception doesn't accept response parameter
+ raise raised_exception_obj(
+ message=error_str,
+ llm_provider=custom_llm_provider,
+ model=model,
+ )
def exception_type( # type: ignore # noqa: PLR0915
diff --git a/litellm/litellm_core_utils/litellm_logging.py b/litellm/litellm_core_utils/litellm_logging.py
index e0a799d8e5c..bc5faf962c2 100644
--- a/litellm/litellm_core_utils/litellm_logging.py
+++ b/litellm/litellm_core_utils/litellm_logging.py
@@ -1853,6 +1853,14 @@ class Logging(LiteLLMLoggingBaseClass):
cache_hit=cache_hit,
standard_logging_object=kwargs.get("standard_logging_object", None),
)
+ litellm_params = self.model_call_details.get("litellm_params", {})
+ is_sync_request = (
+ litellm_params.get(CallTypes.acompletion.value, False) is not True
+ and litellm_params.get(CallTypes.aresponses.value, False) is not True
+ and litellm_params.get(CallTypes.aembedding.value, False) is not True
+ and litellm_params.get(CallTypes.aimage_generation.value, False) is not True
+ and litellm_params.get(CallTypes.atranscription.value, False) is not True
+ )
try:
## BUILD COMPLETE STREAMED RESPONSE
complete_streaming_response: Optional[
@@ -1915,7 +1923,6 @@ class Logging(LiteLLMLoggingBaseClass):
self.has_run_logging(event_type="sync_success")
for callback in callbacks:
try:
- litellm_params = self.model_call_details.get("litellm_params", {})
should_run = self.should_run_callback(
callback=callback,
litellm_params=litellm_params,
@@ -2185,22 +2192,7 @@ class Logging(LiteLLMLoggingBaseClass):
if (
callback == "openmeter"
- and self.model_call_details.get("litellm_params", {}).get(
- "acompletion", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aembedding", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aimage_generation", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "atranscription", False
- )
- is not True
+ and is_sync_request
):
global openMeterLogger
if openMeterLogger is None:
@@ -2229,22 +2221,7 @@ class Logging(LiteLLMLoggingBaseClass):
)
if (
isinstance(callback, CustomLogger)
- and self.model_call_details.get("litellm_params", {}).get(
- "acompletion", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aembedding", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aimage_generation", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "atranscription", False
- )
- is not True
+ and is_sync_request
and self.call_type
!= CallTypes.pass_through.value # pass-through endpoints call async_log_success_event
): # custom logger class
@@ -2272,22 +2249,7 @@ class Logging(LiteLLMLoggingBaseClass):
)
if (
callable(callback) is True
- and self.model_call_details.get("litellm_params", {}).get(
- "acompletion", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aembedding", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aimage_generation", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "atranscription", False
- )
- is not True
+ and is_sync_request
and customLogger is not None
): # custom logger functions
print_verbose(
@@ -2737,6 +2699,15 @@ class Logging(LiteLLMLoggingBaseClass):
event_type="sync_failure"
): # prevent double logging
return
+ litellm_params = self.model_call_details.get("litellm_params", {})
+ is_sync_request = (
+ litellm_params.get(CallTypes.acompletion.value, False) is not True
+ and litellm_params.get(CallTypes.aresponses.value, False) is not True
+ and litellm_params.get(CallTypes.aembedding.value, False) is not True
+ and litellm_params.get(CallTypes.aimage_generation.value, False) is not True
+ and litellm_params.get(CallTypes.atranscription.value, False) is not True
+ )
+
try:
start_time, end_time = self._failure_handler_helper_fn(
exception=exception,
@@ -2762,7 +2733,6 @@ class Logging(LiteLLMLoggingBaseClass):
self.has_run_logging(event_type="sync_failure")
for callback in callbacks:
try:
- litellm_params = self.model_call_details.get("litellm_params", {})
should_run = self.should_run_callback(
callback=callback,
litellm_params=litellm_params,
@@ -2830,14 +2800,7 @@ class Logging(LiteLLMLoggingBaseClass):
)
if (
isinstance(callback, CustomLogger)
- and self.model_call_details.get("litellm_params", {}).get(
- "acompletion", False
- )
- is not True
- and self.model_call_details.get("litellm_params", {}).get(
- "aembedding", False
- )
- is not True
+ and is_sync_request
): # custom logger class
callback.log_failure_event(
start_time=start_time,
@@ -3780,10 +3743,10 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
OpenTelemetry,
OpenTelemetryConfig,
)
-
+ logfire_base_url = os.getenv("LOGFIRE_BASE_URL", "https://logfire-api.pydantic.dev")
otel_config = OpenTelemetryConfig(
exporter="otlp_http",
- endpoint="https://logfire-api.pydantic.dev/v1/traces",
+ endpoint = f"{logfire_base_url.rstrip('/')}/v1/traces",
headers=f"Authorization={os.getenv('LOGFIRE_TOKEN')}",
)
for callback in _in_memory_loggers:
@@ -4375,6 +4338,38 @@ class StandardLoggingPayloadSetup:
return messages
+ @staticmethod
+ def merge_litellm_metadata(litellm_params: dict) -> dict:
+ """
+ Merge both litellm_metadata and metadata from litellm_params.
+
+ litellm_metadata contains model-related fields, metadata contains user API key fields.
+ We need both for complete standard logging payload.
+
+ Args:
+ litellm_params: Dictionary containing metadata and litellm_metadata
+
+ Returns:
+ dict: Merged metadata with user API key fields taking precedence
+ """
+ merged_metadata: dict = {}
+
+ # Start with metadata (user API key fields) - but skip non-serializable objects
+ if litellm_params.get("metadata") and isinstance(litellm_params.get("metadata"), dict):
+ for key, value in litellm_params["metadata"].items():
+ # Skip non-serializable objects like UserAPIKeyAuth
+ if key == "user_api_key_auth":
+ continue
+ merged_metadata[key] = value
+
+ # Then merge litellm_metadata (model-related fields) - this will NOT overwrite existing keys
+ if litellm_params.get("litellm_metadata") and isinstance(litellm_params.get("litellm_metadata"), dict):
+ for key, value in litellm_params["litellm_metadata"].items():
+ if key not in merged_metadata: # Don't overwrite existing keys from metadata
+ merged_metadata[key] = value
+
+ return merged_metadata
+
@staticmethod
def get_standard_logging_metadata(
metadata: Optional[Dict[str, Any]],
@@ -4963,11 +4958,8 @@ def get_standard_logging_object_payload(
litellm_params = kwargs.get("litellm_params", {}) or {}
proxy_server_request = litellm_params.get("proxy_server_request") or {}
- metadata: dict = (
- litellm_params.get("litellm_metadata")
- or litellm_params.get("metadata", None)
- or {}
- )
+ # Merge both litellm_metadata and metadata to get complete metadata
+ metadata: dict = StandardLoggingPayloadSetup.merge_litellm_metadata(litellm_params)
completion_start_time = kwargs.get("completion_start_time", end_time)
call_type = kwargs.get("call_type")
diff --git a/litellm/litellm_core_utils/prompt_templates/factory.py b/litellm/litellm_core_utils/prompt_templates/factory.py
index 4320f756454..43ed23587d8 100644
--- a/litellm/litellm_core_utils/prompt_templates/factory.py
+++ b/litellm/litellm_core_utils/prompt_templates/factory.py
@@ -903,11 +903,11 @@ def convert_to_anthropic_image_obj(
media_type=media_type,
data=base64_data,
)
+ except litellm.ImageFetchError:
+ raise
except Exception as e:
- if "Error: Unable to fetch image from URL" in str(e):
- raise e
raise Exception(
- """Image url not in expected format. Example Expected input - "image_url": "data:image/jpeg;base64,{base64_image}". Supported formats - ['image/jpeg', 'image/png', 'image/gif', 'image/webp']."""
+ f"""Image url not in expected format. Example Expected input - "image_url": "data:image/jpeg;base64,{{base64_image}}". Supported formats - ['image/jpeg', 'image/png', 'image/gif', 'image/webp']. Error: {str(e)}"""
)
@@ -1555,8 +1555,6 @@ def convert_to_gemini_tool_call_result(
# For Computer Use, the response should contain structured data like {"url": "..."}
response_data: dict
try:
- import json
-
if content_str.strip().startswith("{") or content_str.strip().startswith("["):
# Try to parse as JSON (for Computer Use structured responses)
parsed = json.loads(content_str)
@@ -1672,7 +1670,7 @@ def convert_to_anthropic_tool_result(
anthropic_content_element=_anthropic_image_param,
original_content_element=content,
)
- anthropic_content_list.append(_anthropic_image_param)
+ anthropic_content_list.append(cast(AnthropicMessagesImageParam, _anthropic_image_param))
anthropic_content = anthropic_content_list
anthropic_tool_result: Optional[AnthropicMessagesToolResultParam] = None
diff --git a/litellm/litellm_core_utils/prompt_templates/image_handling.py b/litellm/litellm_core_utils/prompt_templates/image_handling.py
index 4fa10e42111..5d0bedb776d 100644
--- a/litellm/litellm_core_utils/prompt_templates/image_handling.py
+++ b/litellm/litellm_core_utils/prompt_templates/image_handling.py
@@ -9,6 +9,7 @@ from httpx import Response
import litellm
from litellm import verbose_logger
from litellm.caching.caching import InMemoryCache
+from litellm.constants import MAX_IMAGE_URL_DOWNLOAD_SIZE_MB
MAX_IMGS_IN_MEMORY = 10
@@ -21,7 +22,25 @@ def _process_image_response(response: Response, url: str) -> str:
f"Error: Unable to fetch image from URL. Status code: {response.status_code}, url={url}"
)
+ # Check size before downloading if Content-Length header is present
+ content_length = response.headers.get("Content-Length")
+ if content_length is not None:
+ size_mb = int(content_length) / (1024 * 1024)
+ if size_mb > MAX_IMAGE_URL_DOWNLOAD_SIZE_MB:
+ raise litellm.ImageFetchError(
+ f"Error: Image size ({size_mb:.2f}MB) exceeds maximum allowed size ({MAX_IMAGE_URL_DOWNLOAD_SIZE_MB}MB). url={url}"
+ )
+
image_bytes = response.content
+
+ # Check actual size after download if Content-Length was not available
+ if content_length is None:
+ size_mb = len(image_bytes) / (1024 * 1024)
+ if size_mb > MAX_IMAGE_URL_DOWNLOAD_SIZE_MB:
+ raise litellm.ImageFetchError(
+ f"Error: Image size ({size_mb:.2f}MB) exceeds maximum allowed size ({MAX_IMAGE_URL_DOWNLOAD_SIZE_MB}MB). url={url}"
+ )
+
base64_image = base64.b64encode(image_bytes).decode("utf-8")
image_type = response.headers.get("Content-Type")
@@ -48,6 +67,12 @@ def _process_image_response(response: Response, url: str) -> str:
async def async_convert_url_to_base64(url: str) -> str:
+ # If MAX_IMAGE_URL_DOWNLOAD_SIZE_MB is 0, block all image downloads
+ if MAX_IMAGE_URL_DOWNLOAD_SIZE_MB == 0:
+ raise litellm.ImageFetchError(
+ f"Error: Image URL download is disabled (MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=0). url={url}"
+ )
+
cached_result = in_memory_cache.get_cache(url)
if cached_result:
return cached_result
@@ -67,6 +92,12 @@ async def async_convert_url_to_base64(url: str) -> str:
def convert_url_to_base64(url: str) -> str:
+ # If MAX_IMAGE_URL_DOWNLOAD_SIZE_MB is 0, block all image downloads
+ if MAX_IMAGE_URL_DOWNLOAD_SIZE_MB == 0:
+ raise litellm.ImageFetchError(
+ f"Error: Image URL download is disabled (MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=0). url={url}"
+ )
+
cached_result = in_memory_cache.get_cache(url)
if cached_result:
return cached_result
diff --git a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
index c332e5f88f7..53252df0a28 100644
--- a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
+++ b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
@@ -17,8 +17,8 @@ from litellm.types.utils import (
ModelResponse,
ModelResponseStream,
PromptTokensDetailsWrapper,
+ ServerToolUse,
Usage,
- ServerToolUse
)
from litellm.utils import print_verbose, token_counter
@@ -68,12 +68,31 @@ class ChunkProcessor:
return chunk["id"]
return ""
+ @staticmethod
+ def _get_model_from_chunks(chunks: List[Dict[str, Any]], first_chunk_model: str) -> str:
+ """
+ Get the actual model from chunks, preferring a model that differs from the first chunk.
+
+ For Azure Model Router, the first chunk may have the request model (e.g., 'azure-model-router')
+ while subsequent chunks have the actual model (e.g., 'gpt-4.1-nano-2025-04-14').
+ This method finds the actual model for accurate cost calculation.
+ """
+ # Look for a model in chunks that differs from the first chunk's model
+ for chunk in chunks:
+ chunk_model = chunk.get("model")
+ if chunk_model and chunk_model != first_chunk_model:
+ return chunk_model
+ # Fall back to first chunk's model if no different model found
+ return first_chunk_model
+
def build_base_response(self, chunks: List[Dict[str, Any]]) -> ModelResponse:
chunk = self.first_chunk
id = ChunkProcessor._get_chunk_id(chunks)
object = chunk["object"]
created = chunk["created"]
- model = chunk["model"]
+ first_chunk_model = chunk["model"]
+ # Get the actual model - for Azure Model Router, this finds the real model from later chunks
+ model = ChunkProcessor._get_model_from_chunks(chunks, first_chunk_model)
system_fingerprint = chunk.get("system_fingerprint", None)
role = chunk["choices"][0]["delta"]["role"]
@@ -113,7 +132,7 @@ class ChunkProcessor:
)
return response
- def get_combined_tool_content(
+ def get_combined_tool_content( # noqa: PLR0915
self, tool_call_chunks: List[Dict[str, Any]]
) -> List[ChatCompletionMessageToolCall]:
tool_calls_list: List[ChatCompletionMessageToolCall] = []
@@ -128,10 +147,26 @@ class ChunkProcessor:
tool_calls = delta.get("tool_calls", [])
for tool_call in tool_calls:
- if not tool_call or not hasattr(tool_call, "function"):
+ # Handle both dict and object formats
+ if not tool_call:
+ continue
+
+ # Check if tool_call has function (either as attribute or dict key)
+ has_function = False
+ if isinstance(tool_call, dict):
+ has_function = "function" in tool_call and tool_call["function"] is not None
+ else:
+ has_function = hasattr(tool_call, "function") and tool_call.function is not None
+
+ if not has_function:
continue
- index = getattr(tool_call, "index", 0)
+ # Get index (handle both dict and object)
+ if isinstance(tool_call, dict):
+ index = tool_call.get("index", 0)
+ else:
+ index = getattr(tool_call, "index", 0)
+
if index not in tool_call_map:
tool_call_map[index] = {
"id": None,
@@ -141,30 +176,56 @@ class ChunkProcessor:
"provider_specific_fields": None,
}
- if hasattr(tool_call, "id") and tool_call.id:
- tool_call_map[index]["id"] = tool_call.id
- if hasattr(tool_call, "type") and tool_call.type:
- tool_call_map[index]["type"] = tool_call.type
- if hasattr(tool_call, "function"):
- if (
- hasattr(tool_call.function, "name")
- and tool_call.function.name
- ):
- tool_call_map[index]["name"] = tool_call.function.name
- if (
- hasattr(tool_call.function, "arguments")
- and tool_call.function.arguments
- ):
- tool_call_map[index]["arguments"].append(
- tool_call.function.arguments
- )
+ # Extract id, type, and function data (handle both dict and object)
+ if isinstance(tool_call, dict):
+ if tool_call.get("id"):
+ tool_call_map[index]["id"] = tool_call["id"]
+ if tool_call.get("type"):
+ tool_call_map[index]["type"] = tool_call["type"]
+
+ function = tool_call.get("function", {})
+ if isinstance(function, dict):
+ if function.get("name"):
+ tool_call_map[index]["name"] = function["name"]
+ if function.get("arguments"):
+ tool_call_map[index]["arguments"].append(function["arguments"])
+ else:
+ # function is an object
+ if hasattr(function, "name") and function.name:
+ tool_call_map[index]["name"] = function.name
+ if hasattr(function, "arguments") and function.arguments:
+ tool_call_map[index]["arguments"].append(function.arguments)
+ else:
+ # tool_call is an object
+ if hasattr(tool_call, "id") and tool_call.id:
+ tool_call_map[index]["id"] = tool_call.id
+ if hasattr(tool_call, "type") and tool_call.type:
+ tool_call_map[index]["type"] = tool_call.type
+ if hasattr(tool_call, "function"):
+ if (
+ hasattr(tool_call.function, "name")
+ and tool_call.function.name
+ ):
+ tool_call_map[index]["name"] = tool_call.function.name
+ if (
+ hasattr(tool_call.function, "arguments")
+ and tool_call.function.arguments
+ ):
+ tool_call_map[index]["arguments"].append(
+ tool_call.function.arguments
+ )
# Preserve provider_specific_fields from streaming chunks
provider_fields = None
- if hasattr(tool_call, "provider_specific_fields") and tool_call.provider_specific_fields:
- provider_fields = tool_call.provider_specific_fields
- elif hasattr(tool_call, "function") and hasattr(tool_call.function, "provider_specific_fields") and tool_call.function.provider_specific_fields:
- provider_fields = tool_call.function.provider_specific_fields
+ if isinstance(tool_call, dict):
+ provider_fields = tool_call.get("provider_specific_fields")
+ if not provider_fields and isinstance(tool_call.get("function"), dict):
+ provider_fields = tool_call["function"].get("provider_specific_fields")
+ else:
+ if hasattr(tool_call, "provider_specific_fields") and tool_call.provider_specific_fields:
+ provider_fields = tool_call.provider_specific_fields
+ elif hasattr(tool_call, "function") and hasattr(tool_call.function, "provider_specific_fields") and tool_call.function.provider_specific_fields:
+ provider_fields = tool_call.function.provider_specific_fields
if provider_fields:
# Merge provider_specific_fields if multiple chunks have them
@@ -203,6 +264,7 @@ class ChunkProcessor:
return tool_calls_list
+
def get_combined_function_call_content(
self, function_call_chunks: List[Dict[str, Any]]
) -> FunctionCall:
diff --git a/litellm/litellm_core_utils/streaming_handler.py b/litellm/litellm_core_utils/streaming_handler.py
index 6baaae7ae3f..3093a37c26a 100644
--- a/litellm/litellm_core_utils/streaming_handler.py
+++ b/litellm/litellm_core_utils/streaming_handler.py
@@ -25,6 +25,7 @@ from litellm.types.utils import (
)
from litellm.types.utils import GenericStreamingChunk as GChunk
from litellm.types.utils import (
+ LlmProviders,
ModelResponse,
ModelResponseStream,
StreamingChoices,
@@ -1301,7 +1302,7 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
else: # openai / azure chat model
- if self.custom_llm_provider == "azure":
+ if self.custom_llm_provider in [LlmProviders.AZURE.value, LlmProviders.AZURE_AI.value]:
if isinstance(chunk, BaseModel) and hasattr(chunk, "model"):
# for azure, we need to pass the model from the original chunk
self.model = getattr(chunk, "model", self.model)
diff --git a/litellm/litellm_core_utils/token_counter.py b/litellm/litellm_core_utils/token_counter.py
index a21ebd56f60..a99bd1cd0f3 100644
--- a/litellm/litellm_core_utils/token_counter.py
+++ b/litellm/litellm_core_utils/token_counter.py
@@ -719,6 +719,12 @@ def _count_content_list(
use_default_image_token_count,
default_token_count,
)
+ elif c["type"] == "thinking":
+ # Claude extended thinking content block
+ # Count the thinking text and skip signature (opaque signature blob)
+ thinking_text = c.get("thinking", "")
+ if thinking_text:
+ num_tokens += count_function(thinking_text)
else:
raise ValueError(
f"Invalid content item type: {type(c).__name__}. "
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py b/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
index ecad7a50011..24524233ddf 100644
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
@@ -2,11 +2,11 @@
## Translates OpenAI call to Anthropic `/v1/messages` format
import json
import traceback
-from litellm._uuid import uuid
from collections import deque
from typing import TYPE_CHECKING, Any, AsyncIterator, Iterator, Literal, Optional
from litellm import verbose_logger
+from litellm._uuid import uuid
from litellm.types.llms.anthropic import UsageDelta
from litellm.types.utils import AdapterCompletionStreamWrapper
@@ -48,6 +48,27 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
super().__init__(completion_stream)
self.model = model
+ def _create_initial_usage_delta(self) -> UsageDelta:
+ """
+ Create the initial UsageDelta for the message_start event.
+
+ Initializes cache token fields (cache_creation_input_tokens, cache_read_input_tokens)
+ to 0 to indicate to clients (like Claude Code) that prompt caching is supported.
+
+ The actual cache token values will be provided in the message_delta event at the
+ end of the stream, since Bedrock Converse API only returns usage data in the final
+ response chunk.
+
+ Returns:
+ UsageDelta with all token counts initialized to 0.
+ """
+ return UsageDelta(
+ input_tokens=0,
+ output_tokens=0,
+ cache_creation_input_tokens=0,
+ cache_read_input_tokens=0,
+ )
+
def __next__(self):
from .transformation import LiteLLMAnthropicMessagesAdapter
@@ -64,7 +85,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
"model": self.model,
"stop_reason": None,
"stop_sequence": None,
- "usage": UsageDelta(input_tokens=0, output_tokens=0),
+ "usage": self._create_initial_usage_delta(),
},
}
if self.sent_content_block_start is False:
@@ -169,7 +190,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
"model": self.model,
"stop_reason": None,
"stop_sequence": None,
- "usage": UsageDelta(input_tokens=0, output_tokens=0),
+ "usage": self._create_initial_usage_delta(),
},
}
)
@@ -211,10 +232,16 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
merged_chunk["delta"] = {}
# Add usage to the held chunk
- merged_chunk["usage"] = {
+ usage_dict: UsageDelta = {
"input_tokens": chunk.usage.prompt_tokens or 0,
"output_tokens": chunk.usage.completion_tokens or 0,
}
+ # Add cache tokens if available (for prompt caching support)
+ if hasattr(chunk.usage, "_cache_creation_input_tokens") and chunk.usage._cache_creation_input_tokens > 0:
+ usage_dict["cache_creation_input_tokens"] = chunk.usage._cache_creation_input_tokens
+ if hasattr(chunk.usage, "_cache_read_input_tokens") and chunk.usage._cache_read_input_tokens > 0:
+ usage_dict["cache_read_input_tokens"] = chunk.usage._cache_read_input_tokens
+ merged_chunk["usage"] = usage_dict
# Queue the merged chunk and reset
self.chunk_queue.append(merged_chunk)
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
index 06092755b17..877e47a9aea 100644
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
@@ -17,7 +17,6 @@ from openai.types.chat.chat_completion_chunk import Choice as OpenAIStreamingCho
from litellm.litellm_core_utils.prompt_templates.common_utils import (
parse_tool_call_arguments,
)
-
from litellm.types.llms.anthropic import (
AllAnthropicToolsValues,
AnthopicMessagesAssistantMessageParam,
@@ -183,6 +182,7 @@ class LiteLLMAnthropicMessagesAdapter:
AnthopicMessagesAssistantMessageParam,
]
],
+ model: Optional[str] = None,
) -> List:
new_messages: List[AllMessageValues] = []
for m in messages:
@@ -205,12 +205,17 @@ class LiteLLMAnthropicMessagesAdapter:
text_obj = ChatCompletionTextObject(
type="text", text=content.get("text", "")
)
+ # Preserve cache_control if present (for prompt caching)
+ # Only for Anthropic models that support prompt caching
+ cache_control = content.get("cache_control")
+ if cache_control and model and self.is_anthropic_claude_model(model):
+ text_obj["cache_control"] = cache_control # type: ignore
new_user_content_list.append(text_obj)
elif content.get("type") == "image":
# Convert Anthropic image format to OpenAI format
source = content.get("source", {})
openai_image_url = (
- self._translate_anthropic_image_to_openai(source)
+ self._translate_anthropic_image_to_openai(cast(dict, source))
)
if openai_image_url:
@@ -240,7 +245,7 @@ class LiteLLMAnthropicMessagesAdapter:
# Combine all content items into a single tool message
# to avoid creating multiple tool_result blocks with the same ID
# (each tool_use must have exactly one tool_result)
- content_items = content.get("content", [])
+ content_items = list(content.get("content", []))
# For single-item content, maintain backward compatibility with string/url format
if len(content_items) == 1:
@@ -266,7 +271,7 @@ class LiteLLMAnthropicMessagesAdapter:
source = c.get("source", {})
openai_image_url = (
self._translate_anthropic_image_to_openai(
- source
+ cast(dict, source)
)
or ""
)
@@ -306,7 +311,7 @@ class LiteLLMAnthropicMessagesAdapter:
source = c.get("source", {})
openai_image_url = (
self._translate_anthropic_image_to_openai(
- source
+ cast(dict, source)
)
or ""
)
@@ -363,7 +368,7 @@ class LiteLLMAnthropicMessagesAdapter:
}
signature = (
self._extract_signature_from_tool_use_content(
- content
+ cast(Dict[str, Any], content)
)
)
@@ -424,14 +429,21 @@ class LiteLLMAnthropicMessagesAdapter:
return new_messages
- def translate_anthropic_thinking_to_openai(
- self, thinking: Dict[str, Any]
+ @staticmethod
+ def translate_anthropic_thinking_to_reasoning_effort(
+ thinking: Dict[str, Any]
) -> Optional[str]:
"""
Translate Anthropic's thinking parameter to OpenAI's reasoning_effort.
Anthropic thinking format: {'type': 'enabled'|'disabled', 'budget_tokens': int}
OpenAI reasoning_effort: 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'default'
+
+ Mapping:
+ - budget_tokens >= 10000 -> 'high'
+ - budget_tokens >= 5000 -> 'medium'
+ - budget_tokens >= 2000 -> 'low'
+ - budget_tokens < 2000 -> 'minimal'
"""
if not isinstance(thinking, dict):
return None
@@ -453,6 +465,53 @@ class LiteLLMAnthropicMessagesAdapter:
return None
+ @staticmethod
+ def is_anthropic_claude_model(model: str) -> bool:
+ """
+ Check if the model is an Anthropic Claude model that supports the thinking parameter.
+
+ Returns True for:
+ - anthropic/* models
+ - bedrock/*anthropic* models (including converse)
+ - vertex_ai/*claude* models
+ """
+ model_lower = model.lower()
+ return (
+ "anthropic" in model_lower
+ or "claude" in model_lower
+ )
+
+ @staticmethod
+ def translate_thinking_for_model(
+ thinking: Dict[str, Any],
+ model: str,
+ ) -> Dict[str, Any]:
+ """
+ Translate Anthropic thinking parameter based on the target model.
+
+ For Claude/Anthropic models: returns {'thinking': }
+ - Preserves exact budget_tokens value
+
+ For non-Claude models: returns {'reasoning_effort': }
+ - Converts thinking to reasoning_effort to avoid UnsupportedParamsError
+
+ Args:
+ thinking: Anthropic thinking dict with 'type' and 'budget_tokens'
+ model: The target model name
+
+ Returns:
+ Dict with either 'thinking' or 'reasoning_effort' key
+ """
+ if LiteLLMAnthropicMessagesAdapter.is_anthropic_claude_model(model):
+ return {"thinking": thinking}
+ else:
+ reasoning_effort = LiteLLMAnthropicMessagesAdapter.translate_anthropic_thinking_to_reasoning_effort(
+ thinking
+ )
+ if reasoning_effort:
+ return {"reasoning_effort": reasoning_effort}
+ return {}
+
def translate_anthropic_tool_choice_to_openai(
self, tool_choice: AnthropicMessagesToolChoice
) -> ChatCompletionToolChoiceValues:
@@ -519,7 +578,8 @@ class LiteLLMAnthropicMessagesAdapter:
anthropic_message_request["messages"],
)
new_messages = self.translate_anthropic_messages_to_openai(
- messages=messages_list
+ messages=messages_list,
+ model=anthropic_message_request.get("model"),
)
## ADD SYSTEM MESSAGE TO MESSAGES
if "system" in anthropic_message_request:
@@ -566,11 +626,15 @@ class LiteLLMAnthropicMessagesAdapter:
if "thinking" in anthropic_message_request:
thinking = anthropic_message_request["thinking"]
if thinking:
- reasoning_effort = self.translate_anthropic_thinking_to_openai(
- thinking=cast(Dict[str, Any], thinking)
- )
- if reasoning_effort:
- new_kwargs["reasoning_effort"] = reasoning_effort
+ model = new_kwargs.get("model", "")
+ if self.is_anthropic_claude_model(model):
+ new_kwargs["thinking"] = thinking # type: ignore
+ else:
+ reasoning_effort = self.translate_anthropic_thinking_to_reasoning_effort(
+ cast(Dict[str, Any], thinking)
+ )
+ if reasoning_effort:
+ new_kwargs["reasoning_effort"] = reasoning_effort
translatable_params = self.translatable_anthropic_params()
for k, v in anthropic_message_request.items():
@@ -721,6 +785,12 @@ class LiteLLMAnthropicMessagesAdapter:
input_tokens=usage.prompt_tokens or 0,
output_tokens=usage.completion_tokens or 0,
)
+ # Add cache tokens if available (for prompt caching support)
+ if hasattr(usage, "_cache_creation_input_tokens") and usage._cache_creation_input_tokens > 0:
+ anthropic_usage["cache_creation_input_tokens"] = usage._cache_creation_input_tokens
+ if hasattr(usage, "_cache_read_input_tokens") and usage._cache_read_input_tokens > 0:
+ anthropic_usage["cache_read_input_tokens"] = usage._cache_read_input_tokens
+
translated_obj = AnthropicMessagesResponse(
id=response.id,
type="message",
@@ -868,6 +938,11 @@ class LiteLLMAnthropicMessagesAdapter:
input_tokens=litellm_usage_chunk.prompt_tokens or 0,
output_tokens=litellm_usage_chunk.completion_tokens or 0,
)
+ # Add cache tokens if available (for prompt caching support)
+ if hasattr(litellm_usage_chunk, "_cache_creation_input_tokens") and litellm_usage_chunk._cache_creation_input_tokens > 0:
+ usage_delta["cache_creation_input_tokens"] = litellm_usage_chunk._cache_creation_input_tokens
+ if hasattr(litellm_usage_chunk, "_cache_read_input_tokens") and litellm_usage_chunk._cache_read_input_tokens > 0:
+ usage_delta["cache_read_input_tokens"] = litellm_usage_chunk._cache_read_input_tokens
else:
usage_delta = UsageDelta(input_tokens=0, output_tokens=0)
return MessageBlockDelta(
diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py b/litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py
new file mode 100644
index 00000000000..542ae20b602
--- /dev/null
+++ b/litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py
@@ -0,0 +1,246 @@
+"""
+Fake Streaming Iterator for Anthropic Messages
+
+This module provides a fake streaming iterator that converts non-streaming
+Anthropic Messages responses into proper streaming format.
+
+Used when WebSearch interception converts stream=True to stream=False but
+the LLM doesn't make a tool call, and we need to return a stream to the user.
+"""
+
+import json
+from typing import Any, Dict, List, cast
+
+from litellm.types.llms.anthropic_messages.anthropic_response import (
+ AnthropicMessagesResponse,
+)
+
+
+class FakeAnthropicMessagesStreamIterator:
+ """
+ Fake streaming iterator for Anthropic Messages responses.
+
+ Used when we need to convert a non-streaming response to a streaming format,
+ such as when WebSearch interception converts stream=True to stream=False but
+ the LLM doesn't make a tool call.
+
+ This creates a proper Anthropic-style streaming response with multiple events:
+ - message_start
+ - content_block_start (for each content block)
+ - content_block_delta (for text content, chunked)
+ - content_block_stop
+ - message_delta (for usage)
+ - message_stop
+ """
+
+ def __init__(self, response: AnthropicMessagesResponse):
+ self.response = response
+ self.chunks = self._create_streaming_chunks()
+ self.current_index = 0
+
+ def _create_streaming_chunks(self) -> List[bytes]:
+ """Convert the non-streaming response to streaming chunks"""
+ chunks = []
+
+ # Cast response to dict for easier access
+ response_dict = cast(Dict[str, Any], self.response)
+
+ # 1. message_start event
+ usage = response_dict.get("usage", {})
+ message_start = {
+ "type": "message_start",
+ "message": {
+ "id": response_dict.get("id"),
+ "type": "message",
+ "role": response_dict.get("role", "assistant"),
+ "model": response_dict.get("model"),
+ "content": [],
+ "stop_reason": None,
+ "stop_sequence": None,
+ "usage": {
+ "input_tokens": usage.get("input_tokens", 0) if usage else 0,
+ "output_tokens": 0
+ }
+ }
+ }
+ chunks.append(f"event: message_start\ndata: {json.dumps(message_start)}\n\n".encode())
+
+ # 2-4. For each content block, send start/delta/stop events
+ content_blocks = response_dict.get("content", [])
+ if content_blocks:
+ for index, block in enumerate(content_blocks):
+ # Cast block to dict for easier access
+ block_dict = cast(Dict[str, Any], block)
+ block_type = block_dict.get("type")
+
+ if block_type == "text":
+ # content_block_start
+ content_block_start = {
+ "type": "content_block_start",
+ "index": index,
+ "content_block": {
+ "type": "text",
+ "text": ""
+ }
+ }
+ chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
+
+ # content_block_delta (send full text as one delta for simplicity)
+ text = block_dict.get("text", "")
+ content_block_delta = {
+ "type": "content_block_delta",
+ "index": index,
+ "delta": {
+ "type": "text_delta",
+ "text": text
+ }
+ }
+ chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode())
+
+ # content_block_stop
+ content_block_stop = {
+ "type": "content_block_stop",
+ "index": index
+ }
+ chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
+
+ elif block_type == "thinking":
+ # content_block_start for thinking
+ content_block_start = {
+ "type": "content_block_start",
+ "index": index,
+ "content_block": {
+ "type": "thinking",
+ "thinking": "",
+ "signature": ""
+ }
+ }
+ chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
+
+ # content_block_delta for thinking text
+ thinking_text = block_dict.get("thinking", "")
+ if thinking_text:
+ content_block_delta = {
+ "type": "content_block_delta",
+ "index": index,
+ "delta": {
+ "type": "thinking_delta",
+ "thinking": thinking_text
+ }
+ }
+ chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode())
+
+ # content_block_delta for signature (if present)
+ signature = block_dict.get("signature", "")
+ if signature:
+ signature_delta = {
+ "type": "content_block_delta",
+ "index": index,
+ "delta": {
+ "type": "signature_delta",
+ "signature": signature
+ }
+ }
+ chunks.append(f"event: content_block_delta\ndata: {json.dumps(signature_delta)}\n\n".encode())
+
+ # content_block_stop
+ content_block_stop = {
+ "type": "content_block_stop",
+ "index": index
+ }
+ chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
+
+ elif block_type == "redacted_thinking":
+ # content_block_start for redacted_thinking
+ content_block_start = {
+ "type": "content_block_start",
+ "index": index,
+ "content_block": {
+ "type": "redacted_thinking"
+ }
+ }
+ chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
+
+ # content_block_stop (no delta for redacted thinking)
+ content_block_stop = {
+ "type": "content_block_stop",
+ "index": index
+ }
+ chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
+
+ elif block_type == "tool_use":
+ # content_block_start
+ content_block_start = {
+ "type": "content_block_start",
+ "index": index,
+ "content_block": {
+ "type": "tool_use",
+ "id": block_dict.get("id"),
+ "name": block_dict.get("name"),
+ "input": {}
+ }
+ }
+ chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
+
+ # content_block_delta (send input as JSON delta)
+ input_data = block_dict.get("input", {})
+ content_block_delta = {
+ "type": "content_block_delta",
+ "index": index,
+ "delta": {
+ "type": "input_json_delta",
+ "partial_json": json.dumps(input_data)
+ }
+ }
+ chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode())
+
+ # content_block_stop
+ content_block_stop = {
+ "type": "content_block_stop",
+ "index": index
+ }
+ chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
+
+ # 5. message_delta event (with final usage and stop_reason)
+ message_delta = {
+ "type": "message_delta",
+ "delta": {
+ "stop_reason": response_dict.get("stop_reason"),
+ "stop_sequence": response_dict.get("stop_sequence")
+ },
+ "usage": {
+ "output_tokens": usage.get("output_tokens", 0) if usage else 0
+ }
+ }
+ chunks.append(f"event: message_delta\ndata: {json.dumps(message_delta)}\n\n".encode())
+
+ # 6. message_stop event
+ message_stop = {
+ "type": "message_stop",
+ "usage": usage if usage else {}
+ }
+ chunks.append(f"event: message_stop\ndata: {json.dumps(message_stop)}\n\n".encode())
+
+ return chunks
+
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ if self.current_index >= len(self.chunks):
+ raise StopAsyncIteration
+
+ chunk = self.chunks[self.current_index]
+ self.current_index += 1
+ return chunk
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ if self.current_index >= len(self.chunks):
+ raise StopIteration
+
+ chunk = self.chunks[self.current_index]
+ self.current_index += 1
+ return chunk
diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/handler.py b/litellm/llms/anthropic/experimental_pass_through/messages/handler.py
index 908b46c11e2..7e5a4f22a7f 100644
--- a/litellm/llms/anthropic/experimental_pass_through/messages/handler.py
+++ b/litellm/llms/anthropic/experimental_pass_through/messages/handler.py
@@ -33,6 +33,70 @@ base_llm_http_handler = BaseLLMHTTPHandler()
#################################################
+async def _execute_pre_request_hooks(
+ model: str,
+ messages: List[Dict],
+ tools: Optional[List[Dict]],
+ stream: Optional[bool],
+ custom_llm_provider: Optional[str],
+ **kwargs,
+) -> Dict:
+ """
+ Execute pre-request hooks from CustomLogger callbacks.
+
+ Allows CustomLoggers to modify request parameters before the API call.
+ Used for WebSearch tool conversion, stream modification, etc.
+
+ Args:
+ model: Model name
+ messages: List of messages
+ tools: Optional tools list
+ stream: Optional stream flag
+ custom_llm_provider: Provider name (if not set, will be extracted from model)
+ **kwargs: Additional request parameters
+
+ Returns:
+ Dict containing all (potentially modified) request parameters including tools, stream
+ """
+ # If custom_llm_provider not provided, extract from model
+ if not custom_llm_provider:
+ try:
+ _, custom_llm_provider, _, _ = litellm.get_llm_provider(model=model)
+ except Exception:
+ # If extraction fails, continue without provider
+ pass
+
+ # Build complete request kwargs dict
+ request_kwargs = {
+ "tools": tools,
+ "stream": stream,
+ "litellm_params": {
+ "custom_llm_provider": custom_llm_provider,
+ },
+ **kwargs,
+ }
+
+ if not litellm.callbacks:
+ return request_kwargs
+
+ from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
+
+ for callback in litellm.callbacks:
+ if not isinstance(callback, _CustomLogger):
+ continue
+
+ # Call the pre-request hook
+ modified_kwargs = await callback.async_pre_request_hook(
+ model, messages, request_kwargs
+ )
+
+ # If hook returned modified kwargs, use them
+ if modified_kwargs is not None:
+ request_kwargs = modified_kwargs
+
+ return request_kwargs
+
+
@client
async def anthropic_messages(
max_tokens: int,
@@ -57,7 +121,24 @@ async def anthropic_messages(
"""
Async: Make llm api request in Anthropic /messages API spec
"""
- local_vars = locals()
+ # Execute pre-request hooks to allow CustomLoggers to modify request
+ request_kwargs = await _execute_pre_request_hooks(
+ model=model,
+ messages=messages,
+ tools=tools,
+ stream=stream,
+ custom_llm_provider=custom_llm_provider,
+ **kwargs,
+ )
+
+ # Extract modified parameters
+ tools = request_kwargs.pop("tools", tools)
+ stream = request_kwargs.pop("stream", stream)
+ # Remove litellm_params from kwargs (only needed for hooks)
+ request_kwargs.pop("litellm_params", None)
+ # Merge back any other modifications
+ kwargs.update(request_kwargs)
+
loop = asyncio.get_event_loop()
kwargs["is_async"] = True
@@ -145,6 +226,10 @@ def anthropic_messages_handler(
# Use provided client or create a new one
litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj") # type: ignore
+ # Store original model name before get_llm_provider strips the provider prefix
+ # This is needed by agentic hooks (e.g., websearch_interception) to make follow-up requests
+ original_model = model
+
litellm_params = GenericLiteLLMParams(
**kwargs,
api_key=api_key,
@@ -162,6 +247,19 @@ def anthropic_messages_handler(
api_base=litellm_params.api_base,
api_key=litellm_params.api_key,
)
+
+ # Store agentic loop params in logging object for agentic hooks
+ # This provides original request context needed for follow-up calls
+ if litellm_logging_obj is not None:
+ litellm_logging_obj.model_call_details["agentic_loop_params"] = {
+ "model": original_model,
+ "custom_llm_provider": custom_llm_provider,
+ }
+
+ # Check if stream was converted for WebSearch interception
+ # This is set in the async wrapper above when stream=True is converted to stream=False
+ if kwargs.get("_websearch_interception_converted_stream", False):
+ litellm_logging_obj.model_call_details["websearch_interception_converted_stream"] = True
if litellm_params.mock_response and isinstance(litellm_params.mock_response, str):
diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py b/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
index 790e7901960..f67e4c8382c 100644
--- a/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
+++ b/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
@@ -2,7 +2,8 @@ from typing import Any, AsyncIterator, Dict, List, Optional, Tuple
import httpx
-from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj, verbose_logger
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.litellm_core_utils.litellm_logging import verbose_logger
from litellm.llms.base_llm.anthropic_messages.transformation import (
BaseAnthropicMessagesConfig,
)
@@ -13,9 +14,10 @@ from litellm.types.llms.anthropic import (
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
)
+from litellm.types.llms.anthropic_tool_search import get_tool_search_beta_header
from litellm.types.router import GenericLiteLLMParams
-from ...common_utils import AnthropicError
+from ...common_utils import AnthropicError, AnthropicModelInfo
DEFAULT_ANTHROPIC_API_BASE = "https://api.anthropic.com"
DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
@@ -75,9 +77,9 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
if "content-type" not in headers:
headers["content-type"] = "application/json"
- headers = self._update_headers_with_optional_anthropic_beta(
+ headers = self._update_headers_with_anthropic_beta(
headers=headers,
- context_management=optional_params.get("context_management"),
+ optional_params=optional_params,
)
return headers, api_base
@@ -153,16 +155,44 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
)
@staticmethod
- def _update_headers_with_optional_anthropic_beta(
- headers: dict, context_management: Optional[Dict]
+ def _update_headers_with_anthropic_beta(
+ headers: dict,
+ optional_params: dict,
+ custom_llm_provider: str = "anthropic",
) -> dict:
- if context_management is None:
- return headers
-
+ """
+ Auto-inject anthropic-beta headers based on features used.
+
+ Handles:
+ - context_management: adds 'context-management-2025-06-27'
+ - tool_search: adds provider-specific tool search header
+
+ Args:
+ headers: Request headers dict
+ optional_params: Optional parameters including tools, context_management
+ custom_llm_provider: Provider name for looking up correct tool search header
+ """
+ beta_values: set = set()
+
+ # Get existing beta headers if any
existing_beta = headers.get("anthropic-beta")
- beta_value = ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value
- if existing_beta is None:
- headers["anthropic-beta"] = beta_value
- elif beta_value not in [beta.strip() for beta in existing_beta.split(",")]:
- headers["anthropic-beta"] = f"{existing_beta}, {beta_value}"
+ if existing_beta:
+ beta_values.update(b.strip() for b in existing_beta.split(","))
+
+ # Check for context management
+ if optional_params.get("context_management") is not None:
+ beta_values.add(ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value)
+
+ # Check for tool search tools
+ tools = optional_params.get("tools")
+ if tools:
+ anthropic_model_info = AnthropicModelInfo()
+ if anthropic_model_info.is_tool_search_used(tools):
+ # Use provider-specific tool search header
+ tool_search_header = get_tool_search_beta_header(custom_llm_provider)
+ beta_values.add(tool_search_header)
+
+ if beta_values:
+ headers["anthropic-beta"] = ",".join(sorted(beta_values))
+
return headers
diff --git a/litellm/llms/azure/azure.py b/litellm/llms/azure/azure.py
index ec4553fac4f..3ef0186ba0e 100644
--- a/litellm/llms/azure/azure.py
+++ b/litellm/llms/azure/azure.py
@@ -664,8 +664,29 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
**data, timeout=timeout
)
headers = dict(raw_response.headers)
- response = raw_response.parse()
+
+ # Convert json.JSONDecodeError to AzureOpenAIError for two critical reasons:
+ #
+ # 1. ROUTER BEHAVIOR: The router relies on exception.status_code to determine cooldown logic:
+ # - JSONDecodeError has no status_code → router skips cooldown evaluation
+ # - AzureOpenAIError has status_code → router properly evaluates for cooldown
+ #
+ # 2. CONNECTION CLEANUP: When response.parse() throws JSONDecodeError, the response
+ # body may not be fully consumed, preventing httpx from properly returning the
+ # connection to the pool. By catching the exception and accessing raw_response.status_code,
+ # we trigger httpx's internal cleanup logic. Without this:
+ # - parse() fails → JSONDecodeError bubbles up → httpx never knows response was acknowledged → connection leak
+ # This completely eliminates "Unclosed connection" warnings during high load.
+ try:
+ response = raw_response.parse()
+ except json.JSONDecodeError as json_error:
+ raise AzureOpenAIError(
+ status_code=raw_response.status_code or 500,
+ message=f"Failed to parse raw Azure embedding response: {str(json_error)}"
+ ) from json_error
+
stringified_response = response.model_dump()
+
## LOGGING
logging_obj.post_call(
input=input,
diff --git a/litellm/llms/azure/exception_mapping.py b/litellm/llms/azure/exception_mapping.py
index 70c2609c6b4..193f3d99955 100644
--- a/litellm/llms/azure/exception_mapping.py
+++ b/litellm/llms/azure/exception_mapping.py
@@ -7,6 +7,7 @@ class AzureOpenAIExceptionMapping:
"""
Class for creating Azure OpenAI specific exceptions
"""
+
@staticmethod
def create_content_policy_violation_error(
message: str,
@@ -16,18 +17,20 @@ class AzureOpenAIExceptionMapping:
) -> ContentPolicyViolationError:
"""
Create a content policy violation error
- """
+ """
raise ContentPolicyViolationError(
- message=f"litellm.ContentPolicyViolationError: AzureException - {message}",
+ message=f"AzureException - {message}",
llm_provider="azure",
model=model,
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
provider_specific_fields={
- "innererror": AzureOpenAIExceptionMapping._get_innererror_from_exception(original_exception)
+ "innererror": AzureOpenAIExceptionMapping._get_innererror_from_exception(
+ original_exception
+ )
},
)
-
+
@staticmethod
def _get_innererror_from_exception(original_exception: Exception) -> Optional[dict]:
"""
@@ -39,4 +42,3 @@ class AzureOpenAIExceptionMapping:
if isinstance(body_dict, dict):
innererror = body_dict.get("innererror")
return innererror
-
\ No newline at end of file
diff --git a/litellm/llms/azure_ai/anthropic/messages_transformation.py b/litellm/llms/azure_ai/anthropic/messages_transformation.py
index 55818cc07d6..0d00c907031 100644
--- a/litellm/llms/azure_ai/anthropic/messages_transformation.py
+++ b/litellm/llms/azure_ai/anthropic/messages_transformation.py
@@ -62,10 +62,10 @@ class AzureAnthropicMessagesConfig(AnthropicMessagesConfig):
if "content-type" not in headers:
headers["content-type"] = "application/json"
- # Update headers with optional anthropic beta features
- headers = self._update_headers_with_optional_anthropic_beta(
+ # Update headers with anthropic beta features (context management, tool search, etc.)
+ headers = self._update_headers_with_anthropic_beta(
headers=headers,
- context_management=optional_params.get("context_management"),
+ optional_params=optional_params,
)
return headers, api_base
diff --git a/litellm/llms/azure_ai/image_edit/flux2_transformation.py b/litellm/llms/azure_ai/image_edit/flux2_transformation.py
index caa39056675..87bae59ba0f 100644
--- a/litellm/llms/azure_ai/image_edit/flux2_transformation.py
+++ b/litellm/llms/azure_ai/image_edit/flux2_transformation.py
@@ -87,7 +87,7 @@ class AzureFoundryFlux2ImageEditConfig(OpenAIImageEditConfig):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
@@ -99,6 +99,9 @@ class AzureFoundryFlux2ImageEditConfig(OpenAIImageEditConfig):
FLUX 2 uses the same endpoint for generation and editing,
with the image passed as base64 in the JSON body.
"""
+ if prompt is None:
+ raise ValueError("FLUX 2 image edit requires a prompt.")
+
image_b64 = self._convert_image_to_base64(image)
# Build request body with required params
diff --git a/litellm/llms/base_llm/files/transformation.py b/litellm/llms/base_llm/files/transformation.py
index 7b0a1868f19..58df15f0c46 100644
--- a/litellm/llms/base_llm/files/transformation.py
+++ b/litellm/llms/base_llm/files/transformation.py
@@ -227,6 +227,7 @@ class BaseFileEndpoints(ABC):
self,
file_id: str,
litellm_parent_otel_span: Optional[Span],
+ llm_router: Optional[Router] = None,
) -> OpenAIFileObject:
pass
diff --git a/litellm/llms/base_llm/image_edit/transformation.py b/litellm/llms/base_llm/image_edit/transformation.py
index d522675296f..cc723480371 100644
--- a/litellm/llms/base_llm/image_edit/transformation.py
+++ b/litellm/llms/base_llm/image_edit/transformation.py
@@ -92,7 +92,7 @@ class BaseImageEditConfig(ABC):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
diff --git a/litellm/llms/bedrock/base_aws_llm.py b/litellm/llms/bedrock/base_aws_llm.py
index 18e9deb53b0..bfb25416cf4 100644
--- a/litellm/llms/bedrock/base_aws_llm.py
+++ b/litellm/llms/bedrock/base_aws_llm.py
@@ -87,7 +87,7 @@ class BaseAWSLLM:
"""
import litellm
from litellm.secret_managers.main import str_to_bool
-
+
# Check environment variable first (highest priority)
ssl_verify = os.getenv("SSL_VERIFY", litellm.ssl_verify)
@@ -1232,15 +1232,20 @@ class BaseAWSLLM:
else:
headers = {"Content-Type": "application/json"}
+ aws_signature_headers = self._filter_headers_for_aws_signature(headers)
request = AWSRequest(
method="POST",
url=api_base,
data=json.dumps(request_data),
- headers=headers,
+ headers=aws_signature_headers,
)
sigv4.add_auth(request)
request_headers_dict = dict(request.headers)
+ # Add back original headers after signing. Only headers in SignedHeaders
+ # are integrity-protected; forwarded headers (x-forwarded-*) must remain unsigned.
+ for header_name, header_value in headers.items():
+ request_headers_dict[header_name] = header_value
if (
headers is not None and "Authorization" in headers
): # prevent sigv4 from overwriting the auth header
diff --git a/litellm/llms/bedrock/chat/invoke_handler.py b/litellm/llms/bedrock/chat/invoke_handler.py
index 49292545208..c9677cf9edd 100644
--- a/litellm/llms/bedrock/chat/invoke_handler.py
+++ b/litellm/llms/bedrock/chat/invoke_handler.py
@@ -729,8 +729,6 @@ class BedrockLLM(BaseAWSLLM):
client: Optional[Union[AsyncHTTPHandler, HTTPHandler]] = None,
) -> Union[ModelResponse, CustomStreamWrapper]:
try:
- from botocore.auth import SigV4Auth
- from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
except ImportError:
raise ImportError("Missing boto3 to call bedrock. Run 'pip install boto3'.")
@@ -808,8 +806,6 @@ class BedrockLLM(BaseAWSLLM):
endpoint_url = f"{endpoint_url}/model/{modelId}/invoke"
proxy_endpoint_url = f"{proxy_endpoint_url}/model/{modelId}/invoke"
- sigv4 = SigV4Auth(credentials, "bedrock", aws_region_name)
-
prompt, chat_history = self.convert_messages_to_prompt(
model, messages, provider, custom_prompt_dict
)
@@ -970,15 +966,14 @@ class BedrockLLM(BaseAWSLLM):
headers = {"Content-Type": "application/json"}
if extra_headers is not None:
headers = {"Content-Type": "application/json", **extra_headers}
- request = AWSRequest(
- method="POST", url=endpoint_url, data=data, headers=headers
+ prepped = self.get_request_headers(
+ credentials=credentials,
+ aws_region_name=aws_region_name,
+ extra_headers=extra_headers,
+ endpoint_url=endpoint_url,
+ data=data,
+ headers=headers,
)
- sigv4.add_auth(request)
- if (
- extra_headers is not None and "Authorization" in extra_headers
- ): # prevent sigv4 from overwriting the auth header
- request.headers["Authorization"] = extra_headers["Authorization"]
- prepped = request.prepare()
## LOGGING
logging_obj.pre_call(
@@ -1533,6 +1528,7 @@ class AWSEventStreamDecoder:
)
],
id=self.response_id,
+ model=self.model,
usage=usage,
provider_specific_fields=model_response_provider_specific_fields,
)
diff --git a/litellm/llms/bedrock/common_utils.py b/litellm/llms/bedrock/common_utils.py
index f4b5de8f7c0..bdcc8ab8c24 100644
--- a/litellm/llms/bedrock/common_utils.py
+++ b/litellm/llms/bedrock/common_utils.py
@@ -425,6 +425,15 @@ def strip_bedrock_routing_prefix(model: str) -> str:
return model
+def strip_bedrock_throughput_suffix(model: str) -> str:
+ """ Strip throughput tier suffixes from Bedrock model names. """
+ import re
+
+ # Pattern matches model:version:throughput where throughput is like 51k, 18k, etc.
+ # Keep the model:version part, strip the :throughput suffix
+ return re.sub(r"(:\d+):\d+k$", r"\1", model)
+
+
def get_bedrock_base_model(model: str) -> str:
"""
Get the base model from the given model name.
@@ -432,9 +441,11 @@ def get_bedrock_base_model(model: str) -> str:
Handle model names like:
- "us.meta.llama3-2-11b-instruct-v1:0" -> "meta.llama3-2-11b-instruct-v1"
- "bedrock/converse/model" -> "model"
+ - "anthropic.claude-3-5-sonnet-20241022-v2:0:51k" -> "anthropic.claude-3-5-sonnet-20241022-v2:0"
"""
model = strip_bedrock_routing_prefix(model)
model = extract_model_name_from_bedrock_arn(model)
+ model = strip_bedrock_throughput_suffix(model)
potential_region = model.split(".", 1)[0]
alt_potential_region = model.split("/", 1)[0]
diff --git a/litellm/llms/bedrock/count_tokens/bedrock_token_counter.py b/litellm/llms/bedrock/count_tokens/bedrock_token_counter.py
index b680bd046ef..54f8a8dbd65 100644
--- a/litellm/llms/bedrock/count_tokens/bedrock_token_counter.py
+++ b/litellm/llms/bedrock/count_tokens/bedrock_token_counter.py
@@ -6,7 +6,7 @@ from typing import Any, Dict, List, Optional
from litellm._logging import verbose_logger
from litellm.llms.base_llm.base_utils import BaseTokenCounter
-from litellm.llms.bedrock.common_utils import get_bedrock_base_model
+from litellm.llms.bedrock.common_utils import BedrockError, get_bedrock_base_model
from litellm.llms.bedrock.count_tokens.handler import BedrockCountTokensHandler
from litellm.types.utils import LlmProviders, TokenCountResponse
@@ -79,9 +79,31 @@ class BedrockTokenCounter(BaseTokenCounter):
tokenizer_type="bedrock_api",
original_response=result,
)
+ except BedrockError as e:
+ verbose_logger.warning(
+ f"Bedrock CountTokens API error: status={e.status_code}, message={e.message}"
+ )
+ return TokenCountResponse(
+ total_tokens=0,
+ request_model=request_model,
+ model_used=model_to_use,
+ tokenizer_type="bedrock_api",
+ error=True,
+ error_message=e.message,
+ status_code=e.status_code,
+ )
except Exception as e:
verbose_logger.warning(
- f"Error calling Bedrock CountTokens API: {e}, falling back to default tokenizer"
+ f"Error calling Bedrock CountTokens API: {e}"
+ )
+ return TokenCountResponse(
+ total_tokens=0,
+ request_model=request_model,
+ model_used=model_to_use,
+ tokenizer_type="bedrock_api",
+ error=True,
+ error_message=str(e),
+ status_code=500,
)
return None
diff --git a/litellm/llms/bedrock/count_tokens/handler.py b/litellm/llms/bedrock/count_tokens/handler.py
index e8366165b65..9d2be6cca89 100644
--- a/litellm/llms/bedrock/count_tokens/handler.py
+++ b/litellm/llms/bedrock/count_tokens/handler.py
@@ -6,6 +6,8 @@ Simplified handler leveraging existing LiteLLM Bedrock infrastructure.
from typing import Any, Dict
+import httpx
+
import litellm
from litellm._logging import verbose_logger
from litellm.llms.bedrock.common_utils import BedrockError
@@ -98,7 +100,7 @@ class BedrockCountTokensHandler(BedrockCountTokensConfig):
verbose_logger.error(f"AWS Bedrock error: {error_text}")
raise BedrockError(
status_code=response.status_code,
- message=f"AWS Bedrock error: {error_text}",
+ message=error_text,
)
bedrock_response = response.json()
@@ -117,6 +119,13 @@ class BedrockCountTokensHandler(BedrockCountTokensConfig):
except BedrockError:
# Re-raise Bedrock exceptions as-is
raise
+ except httpx.HTTPStatusError as e:
+ # HTTP errors - preserve the actual status code
+ verbose_logger.error(f"HTTP error in CountTokens handler: {str(e)}")
+ raise BedrockError(
+ status_code=e.response.status_code,
+ message=e.response.text,
+ )
except Exception as e:
verbose_logger.error(f"Error in CountTokens handler: {str(e)}")
raise BedrockError(
diff --git a/litellm/llms/bedrock/image_edit/handler.py b/litellm/llms/bedrock/image_edit/handler.py
index b4b6c8d7622..0f1dcff6294 100644
--- a/litellm/llms/bedrock/image_edit/handler.py
+++ b/litellm/llms/bedrock/image_edit/handler.py
@@ -261,7 +261,7 @@ class BedrockImageEdit(BaseAWSLLM):
"""
config_class = self.get_config_class(model=model)
config_instance = config_class()
- request_body = config_instance.transform_image_edit_request(
+ request_body, _ = config_instance.transform_image_edit_request(
model=model,
prompt=prompt,
image=image[0] if image else None,
diff --git a/litellm/llms/bedrock/image_edit/stability_transformation.py b/litellm/llms/bedrock/image_edit/stability_transformation.py
index bcaf0923f69..e8b77812988 100644
--- a/litellm/llms/bedrock/image_edit/stability_transformation.py
+++ b/litellm/llms/bedrock/image_edit/stability_transformation.py
@@ -21,18 +21,18 @@ Supported models:
API Reference: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html
"""
-import json
import base64
+import json
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple
import httpx
from litellm.llms.base_llm.image_edit.transformation import BaseImageEditConfig
from litellm.types.images.main import ImageEditOptionalRequestParams
-from litellm.types.router import GenericLiteLLMParams
from litellm.types.llms.stability import (
OPENAI_SIZE_TO_STABILITY_ASPECT_RATIO,
)
+from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import FileTypes, ImageObject, ImageResponse
from litellm.utils import get_model_info
@@ -153,7 +153,7 @@ class BedrockStabilityImageEditConfig(BaseImageEditConfig):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
@@ -164,6 +164,9 @@ class BedrockStabilityImageEditConfig(BaseImageEditConfig):
Returns the request body dict that will be JSON-encoded by the handler.
"""
+ if prompt is None:
+ raise ValueError("Bedrock Stability image edit requires a prompt.")
+
# Build Bedrock Stability request
data: Dict[str, Any] = {
"prompt": prompt,
diff --git a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
index 81225159a7c..fa5002fcad8 100644
--- a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
+++ b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
@@ -129,6 +129,37 @@ class AmazonAnthropicClaudeMessagesConfig(
if isinstance(cache_control, dict) and "ttl" in cache_control:
cache_control.pop("ttl", None)
+ def _get_tool_search_beta_header_for_bedrock(
+ self,
+ model: str,
+ tool_search_used: bool,
+ programmatic_tool_calling_used: bool,
+ input_examples_used: bool,
+ beta_set: set,
+ ) -> None:
+ """
+ Adjust tool search beta header for Bedrock.
+
+ Bedrock requires a different beta header for tool search on Opus 4 models
+ when tool search is used without programmatic tool calling or input examples.
+
+ Note: On Amazon Bedrock, server-side tool search is only supported on Claude Opus 4
+ with the `tool-search-tool-2025-10-19` beta header.
+
+ Ref: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool
+
+ Args:
+ model: The model name
+ tool_search_used: Whether tool search is used
+ programmatic_tool_calling_used: Whether programmatic tool calling is used
+ input_examples_used: Whether input examples are used
+ beta_set: The set of beta headers to modify in-place
+ """
+ if tool_search_used and not (programmatic_tool_calling_used or input_examples_used):
+ beta_set.discard(ANTHROPIC_TOOL_SEARCH_BETA_HEADER)
+ if "opus-4" in model.lower() or "opus_4" in model.lower():
+ beta_set.add("tool-search-tool-2025-10-19")
+
def transform_anthropic_messages_request(
self,
model: str,
@@ -189,13 +220,13 @@ class AmazonAnthropicClaudeMessagesConfig(
)
beta_set.update(auto_betas)
- if (
- tool_search_used
- and not (programmatic_tool_calling_used or input_examples_used)
- ):
- beta_set.discard(ANTHROPIC_TOOL_SEARCH_BETA_HEADER)
- if "opus-4" in model.lower() or "opus_4" in model.lower():
- beta_set.add("tool-search-tool-2025-10-19")
+ self._get_tool_search_beta_header_for_bedrock(
+ model=model,
+ tool_search_used=tool_search_used,
+ programmatic_tool_calling_used=programmatic_tool_calling_used,
+ input_examples_used=input_examples_used,
+ beta_set=beta_set,
+ )
if beta_set:
anthropic_messages_request["anthropic_beta"] = list(beta_set)
diff --git a/litellm/llms/bedrock/passthrough/transformation.py b/litellm/llms/bedrock/passthrough/transformation.py
index 71c418757bf..5efd3ba1d9f 100644
--- a/litellm/llms/bedrock/passthrough/transformation.py
+++ b/litellm/llms/bedrock/passthrough/transformation.py
@@ -236,6 +236,7 @@ class BedrockPassthroughConfig(
if len(all_translated_chunks) > 0:
model_response = stream_chunk_builder(
chunks=all_translated_chunks,
+ logging_obj=litellm_logging_obj,
)
return model_response
return None
diff --git a/litellm/llms/custom_httpx/aiohttp_transport.py b/litellm/llms/custom_httpx/aiohttp_transport.py
index f845bf7cb90..a7b83d8c802 100644
--- a/litellm/llms/custom_httpx/aiohttp_transport.py
+++ b/litellm/llms/custom_httpx/aiohttp_transport.py
@@ -245,7 +245,6 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
allow_redirects=False,
auto_decompress=False,
timeout=ClientTimeout(
- total=timeout.get("read"),
sock_connect=timeout.get("connect"),
sock_read=timeout.get("read"),
connect=timeout.get("pool"),
diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
index 1da6e61252f..ab1e735fca7 100644
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -1929,6 +1929,7 @@ class BaseLLMHTTPHandler:
# used for logging + cost tracking
logging_obj.model_call_details["httpx_response"] = response
+ initial_response: Union[AsyncIterator, AnthropicMessagesResponse]
if stream:
completion_stream = anthropic_messages_provider_config.get_async_streaming_response_iterator(
model=model,
@@ -1936,14 +1937,29 @@ class BaseLLMHTTPHandler:
request_body=request_body,
litellm_logging_obj=logging_obj,
)
- return completion_stream
+ initial_response = completion_stream
else:
- return anthropic_messages_provider_config.transform_anthropic_messages_response(
+ initial_response = anthropic_messages_provider_config.transform_anthropic_messages_response(
model=model,
raw_response=response,
logging_obj=logging_obj,
)
+ # Call agentic completion hooks
+ final_response = await self._call_agentic_completion_hooks(
+ response=initial_response,
+ model=model,
+ messages=messages,
+ anthropic_messages_provider_config=anthropic_messages_provider_config,
+ anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
+ logging_obj=logging_obj,
+ stream=stream or False,
+ custom_llm_provider=custom_llm_provider,
+ kwargs=kwargs,
+ )
+
+ return final_response if final_response is not None else initial_response
+
def anthropic_messages_handler(
self,
model: str,
@@ -4334,6 +4350,111 @@ class BaseLLMHTTPHandler:
return stream, data
return stream, data
+ async def _call_agentic_completion_hooks(
+ self,
+ response: Any,
+ model: str,
+ messages: List[Dict],
+ anthropic_messages_provider_config: "BaseAnthropicMessagesConfig",
+ anthropic_messages_optional_request_params: Dict,
+ logging_obj: "LiteLLMLoggingObj",
+ stream: bool,
+ custom_llm_provider: str,
+ kwargs: Dict,
+ ) -> Optional[Any]:
+ """
+ Call agentic completion hooks for all custom loggers.
+
+ 1. Call async_should_run_agentic_completion to check if agentic loop is needed
+ 2. If yes, call async_run_agentic_completion to execute the loop
+
+ Returns the response from agentic loop, or None if no hook runs.
+ """
+ from litellm._logging import verbose_logger
+ from litellm.integrations.custom_logger import CustomLogger
+
+ callbacks = litellm.callbacks + (
+ logging_obj.dynamic_success_callbacks or []
+ )
+ tools = anthropic_messages_optional_request_params.get("tools", [])
+
+ for callback in callbacks:
+ try:
+ if isinstance(callback, CustomLogger):
+ # First: Check if agentic loop should run
+ should_run, tool_calls = (
+ await callback.async_should_run_agentic_loop(
+ response=response,
+ model=model,
+ messages=messages,
+ tools=tools,
+ stream=stream,
+ custom_llm_provider=custom_llm_provider,
+ kwargs=kwargs,
+ )
+ )
+
+ if should_run:
+ # Second: Execute agentic loop
+ # Add custom_llm_provider to kwargs so the agentic loop can reconstruct the full model name
+ kwargs_with_provider = kwargs.copy() if kwargs else {}
+ kwargs_with_provider["custom_llm_provider"] = custom_llm_provider
+ agentic_response = await callback.async_run_agentic_loop(
+ tools=tool_calls,
+ model=model,
+ messages=messages,
+ response=response,
+ anthropic_messages_provider_config=anthropic_messages_provider_config,
+ anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
+ logging_obj=logging_obj,
+ stream=stream,
+ kwargs=kwargs_with_provider,
+ )
+ # First hook that runs agentic loop wins
+ return agentic_response
+
+ except Exception as e:
+ verbose_logger.exception(
+ f"LiteLLM.AgenticHookError: Exception in agentic completion hooks: {str(e)}"
+ )
+
+ # Check if we need to convert response to fake stream
+ # This happens when:
+ # 1. Stream was originally True but converted to False for WebSearch interception
+ # 2. No agentic loop ran (LLM didn't use the tool)
+ # 3. We have a non-streaming response that needs to be converted to streaming
+ websearch_converted_stream = (
+ logging_obj.model_call_details.get("websearch_interception_converted_stream", False)
+ if logging_obj is not None
+ else False
+ )
+
+ if websearch_converted_stream:
+ from typing import cast
+
+ from litellm._logging import verbose_logger
+ from litellm.llms.anthropic.experimental_pass_through.messages.fake_stream_iterator import (
+ FakeAnthropicMessagesStreamIterator,
+ )
+ from litellm.types.llms.anthropic_messages.anthropic_response import (
+ AnthropicMessagesResponse,
+ )
+
+ verbose_logger.debug(
+ "WebSearchInterception: No tool call made, converting non-streaming response to fake stream"
+ )
+
+ # Convert the non-streaming response to a fake stream
+ # The response should be an AnthropicMessagesResponse (dict)
+ if isinstance(response, dict):
+ # Create a fake streaming iterator
+ fake_stream = FakeAnthropicMessagesStreamIterator(
+ response=cast(AnthropicMessagesResponse, response)
+ )
+ return fake_stream
+
+ return None
+
def _handle_error(
self,
e: Exception,
@@ -4453,7 +4574,7 @@ class BaseLLMHTTPHandler:
self,
model: str,
image: Any,
- prompt: str,
+ prompt: Optional[str],
image_edit_provider_config: BaseImageEditConfig,
image_edit_optional_request_params: Dict,
custom_llm_provider: str,
@@ -4572,7 +4693,7 @@ class BaseLLMHTTPHandler:
self,
model: str,
image: FileTypes,
- prompt: str,
+ prompt: Optional[str],
image_edit_provider_config: BaseImageEditConfig,
image_edit_optional_request_params: Dict,
custom_llm_provider: str,
diff --git a/litellm/llms/custom_llm.py b/litellm/llms/custom_llm.py
index d235df30f25..a820ac7f345 100644
--- a/litellm/llms/custom_llm.py
+++ b/litellm/llms/custom_llm.py
@@ -201,7 +201,7 @@ class CustomLLM(BaseLLM):
self,
model: str,
image: Any,
- prompt: str,
+ prompt: Optional[str],
model_response: ImageResponse,
api_key: Optional[str],
api_base: Optional[str],
@@ -216,7 +216,7 @@ class CustomLLM(BaseLLM):
self,
model: str,
image: Any,
- prompt: str,
+ prompt: Optional[str],
model_response: ImageResponse,
api_key: Optional[str],
api_base: Optional[str],
diff --git a/litellm/llms/gemini/image_edit/transformation.py b/litellm/llms/gemini/image_edit/transformation.py
index 78a7ff9546f..0015155b47f 100644
--- a/litellm/llms/gemini/image_edit/transformation.py
+++ b/litellm/llms/gemini/image_edit/transformation.py
@@ -80,7 +80,7 @@ class GeminiImageEditConfig(BaseImageEditConfig):
def transform_image_edit_request( # type: ignore[override]
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict[str, Any],
litellm_params: GenericLiteLLMParams,
@@ -90,6 +90,9 @@ class GeminiImageEditConfig(BaseImageEditConfig):
if not inline_parts:
raise ValueError("Gemini image edit requires at least one image.")
+ if prompt is None:
+ raise ValueError("Gemini image edit requires a prompt.")
+
contents = [
{
"parts": inline_parts + [{"text": prompt}],
diff --git a/litellm/llms/ollama/chat/transformation.py b/litellm/llms/ollama/chat/transformation.py
index 9c8700daf83..8c98cc54050 100644
--- a/litellm/llms/ollama/chat/transformation.py
+++ b/litellm/llms/ollama/chat/transformation.py
@@ -190,46 +190,14 @@ class OllamaChatConfig(BaseConfig):
else:
optional_params["think"] = value in {"low", "medium", "high"}
### FUNCTION CALLING LOGIC ###
+ # Ollama 0.4+ supports native tool calling - pass tools directly
+ # and let Ollama handle model capability detection
+ # Fixes: https://github.com/BerriAI/litellm/issues/18922
if param == "tools":
- ## CHECK IF MODEL SUPPORTS TOOL CALLING ##
- try:
- model_info = litellm.get_model_info(
- model=model, custom_llm_provider="ollama"
- )
- if model_info.get("supports_function_calling") is True:
- optional_params["tools"] = value
- else:
- raise Exception
- except Exception:
- optional_params["format"] = "json"
- litellm.add_function_to_prompt = (
- True # so that main.py adds the function call to the prompt
- )
- optional_params["functions_unsupported_model"] = value
-
- if len(optional_params["functions_unsupported_model"]) == 1:
- optional_params["function_name"] = optional_params[
- "functions_unsupported_model"
- ][0]["function"]["name"]
+ optional_params["tools"] = value
if param == "functions":
- ## CHECK IF MODEL SUPPORTS TOOL CALLING ##
- try:
- model_info = litellm.get_model_info(
- model=model, custom_llm_provider="ollama"
- )
- if model_info.get("supports_function_calling") is True:
- optional_params["tools"] = value
- else:
- raise Exception
- except Exception:
- optional_params["format"] = "json"
- litellm.add_function_to_prompt = (
- True # so that main.py adds the function call to the prompt
- )
- optional_params["functions_unsupported_model"] = (
- non_default_params.get("functions")
- )
+ optional_params["tools"] = value
non_default_params.pop("tool_choice", None) # causes ollama requests to hang
non_default_params.pop("functions", None) # causes ollama requests to hang
return optional_params
@@ -431,6 +399,10 @@ class OllamaChatConfig(BaseConfig):
_message = litellm.Message(**response_json_message)
model_response.choices[0].message = _message # type: ignore
+ # Set finish_reason to "tool_calls" when tool_calls are present
+ # Fixes: https://github.com/BerriAI/litellm/issues/18922
+ if _message.tool_calls:
+ model_response.choices[0].finish_reason = "tool_calls"
model_response.created = int(time.time())
model_response.model = "ollama_chat/" + model
prompt_tokens = response_json.get("prompt_eval_count", litellm.token_counter(messages=messages)) # type: ignore
@@ -563,6 +535,10 @@ class OllamaChatCompletionResponseIterator(BaseModelResponseIterator):
if chunk["done"] is True:
finish_reason = chunk.get("done_reason", "stop")
+ # Override finish_reason when tool_calls are present
+ # Fixes: https://github.com/BerriAI/litellm/issues/18922
+ if tool_calls is not None:
+ finish_reason = "tool_calls"
choices = [
StreamingChoices(
delta=delta,
diff --git a/litellm/llms/openai/chat/guardrail_translation/handler.py b/litellm/llms/openai/chat/guardrail_translation/handler.py
index e2cf9f46104..d0ed3f165cc 100644
--- a/litellm/llms/openai/chat/guardrail_translation/handler.py
+++ b/litellm/llms/openai/chat/guardrail_translation/handler.py
@@ -362,7 +362,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if has_stream_ended:
# convert to model response
model_response = cast(
- ModelResponse, stream_chunk_builder(chunks=responses_so_far)
+ ModelResponse, stream_chunk_builder(chunks=responses_so_far, logging_obj=litellm_logging_obj)
)
# run process_output_response
await self.process_output_response(
diff --git a/litellm/llms/openai/containers/transformation.py b/litellm/llms/openai/containers/transformation.py
index 46718816f37..e67bfbe0c62 100644
--- a/litellm/llms/openai/containers/transformation.py
+++ b/litellm/llms/openai/containers/transformation.py
@@ -83,8 +83,13 @@ class OpenAIContainerConfig(BaseContainerConfig):
) -> str:
"""Get the complete URL for OpenAI container API.
"""
- if api_base is None:
- api_base = "https://api.openai.com/v1"
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("OPENAI_BASE_URL")
+ or get_secret_str("OPENAI_API_BASE")
+ or "https://api.openai.com/v1"
+ )
return f"{api_base.rstrip('/')}/containers"
diff --git a/litellm/llms/openai/image_edit/dalle2_transformation.py b/litellm/llms/openai/image_edit/dalle2_transformation.py
index 37e92be17a8..13531546d2e 100644
--- a/litellm/llms/openai/image_edit/dalle2_transformation.py
+++ b/litellm/llms/openai/image_edit/dalle2_transformation.py
@@ -1,5 +1,5 @@
from io import BufferedReader
-from typing import TYPE_CHECKING, Any, Dict, List, Tuple, cast
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, cast
from httpx._types import RequestFiles
@@ -30,7 +30,7 @@ class DallE2ImageEditConfig(OpenAIImageEditConfig):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
@@ -41,6 +41,9 @@ class DallE2ImageEditConfig(OpenAIImageEditConfig):
DALL-E-2 only accepts a single image with field name "image" (not "image[]").
"""
+ if prompt is None:
+ raise ValueError("DALL-E-2 image edit requires a prompt.")
+
request = ImageEditRequestParams(
model=model,
image=image,
diff --git a/litellm/llms/openai/image_edit/transformation.py b/litellm/llms/openai/image_edit/transformation.py
index 1b90d96fa92..9edad9ee2c9 100644
--- a/litellm/llms/openai/image_edit/transformation.py
+++ b/litellm/llms/openai/image_edit/transformation.py
@@ -79,7 +79,7 @@ class OpenAIImageEditConfig(BaseImageEditConfig):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
@@ -91,6 +91,9 @@ class OpenAIImageEditConfig(BaseImageEditConfig):
Handles multipart/form-data for images. Uses "image[]" field name
to support multiple images (e.g., for gpt-image-1).
"""
+ if prompt is None:
+ raise ValueError("OpenAI image edit requires a prompt.")
+
request = ImageEditRequestParams(
model=model,
image=image,
diff --git a/litellm/llms/openai/realtime/handler.py b/litellm/llms/openai/realtime/handler.py
index 3ae4d2bc9f7..6ab43ab31e4 100644
--- a/litellm/llms/openai/realtime/handler.py
+++ b/litellm/llms/openai/realtime/handler.py
@@ -57,6 +57,19 @@ class OpenAIRealtime(OpenAIChatCompletion):
try:
ssl_context = get_shared_realtime_ssl_context()
+ # Log a masked request preview consistent with other endpoints.
+ logging_obj.pre_call(
+ input=None,
+ api_key=api_key,
+ additional_args={
+ "api_base": url,
+ "headers": {
+ "Authorization": f"Bearer {api_key}",
+ "OpenAI-Beta": "realtime=v1",
+ },
+ "complete_input_dict": {"query_params": query_params},
+ },
+ )
async with websockets.connect( # type: ignore
url,
additional_headers={
diff --git a/litellm/llms/openrouter/image_generation/__init__.py b/litellm/llms/openrouter/image_generation/__init__.py
new file mode 100644
index 00000000000..f2d06439d40
--- /dev/null
+++ b/litellm/llms/openrouter/image_generation/__init__.py
@@ -0,0 +1,13 @@
+from litellm.llms.base_llm.image_generation.transformation import (
+ BaseImageGenerationConfig,
+)
+
+from .transformation import OpenRouterImageGenerationConfig
+
+__all__ = [
+ "OpenRouterImageGenerationConfig",
+]
+
+
+def get_openrouter_image_generation_config(model: str) -> BaseImageGenerationConfig:
+ return OpenRouterImageGenerationConfig()
\ No newline at end of file
diff --git a/litellm/llms/openrouter/image_generation/transformation.py b/litellm/llms/openrouter/image_generation/transformation.py
new file mode 100644
index 00000000000..92084b533af
--- /dev/null
+++ b/litellm/llms/openrouter/image_generation/transformation.py
@@ -0,0 +1,414 @@
+"""
+OpenRouter Image Generation Support
+
+OpenRouter provides image generation through chat completion endpoints.
+Models like google/gemini-2.5-flash-image return images in the message content.
+
+Response format:
+{
+ "choices": [{
+ "message": {
+ "content": "Here is a beautiful sunset for you! ",
+ "role": "assistant",
+ "images": [{
+ "image_url": {"url": "data:image/png;base64,..."},
+ "index": 0,
+ "type": "image_url"
+ }]
+ }
+ }],
+ "usage": {
+ "completion_tokens": 1299,
+ "prompt_tokens": 6,
+ "total_tokens": 1305,
+ "completion_tokens_details": {"image_tokens": 1290},
+ "cost": 0.0387243
+ }
+}
+"""
+
+from typing import TYPE_CHECKING, Any, List, Optional, Union
+
+import httpx
+
+import litellm
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.image_generation.transformation import (
+ BaseImageGenerationConfig,
+)
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams, AllMessageValues
+from litellm.types.utils import ImageObject, ImageResponse, ImageUsage, ImageUsageInputTokensDetails
+from litellm.llms.openrouter.common_utils import OpenRouterException
+
+
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+else:
+ LiteLLMLoggingObj = Any
+
+
+class OpenRouterImageGenerationConfig(BaseImageGenerationConfig):
+ """
+ Configuration for OpenRouter image generation via chat completions.
+
+ OpenRouter uses chat completion endpoints for image generation,
+ so we need to transform image generation requests to chat format
+ and extract images from chat responses.
+ """
+
+ def get_supported_openai_params(
+ self, model: str
+ ) -> List[OpenAIImageGenerationOptionalParams]:
+ """
+ Get supported OpenAI parameters for OpenRouter image generation.
+
+ Since OpenRouter uses chat completions for image generation,
+ we support standard image generation params.
+ """
+ return [
+ "size",
+ "quality",
+ "n",
+ ]
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ """
+ Map image generation params to OpenRouter chat completion format.
+
+ Maps OpenAI parameters to OpenRouter's image_config format:
+ - size -> image_config.aspect_ratio
+ - quality -> image_config.image_size
+ """
+ supported_params = self.get_supported_openai_params(model)
+
+ for key, value in non_default_params.items():
+ if key in supported_params:
+ if key == "size":
+ # Map OpenAI size to OpenRouter aspect_ratio
+ aspect_ratio = self._map_size_to_aspect_ratio(value)
+ if "image_config" not in optional_params:
+ optional_params["image_config"] = {}
+ optional_params["image_config"]["aspect_ratio"] = aspect_ratio
+ elif key == "quality":
+ # Map OpenAI quality to OpenRouter image_size
+ image_size = self._map_quality_to_image_size(value)
+ if image_size:
+ if "image_config" not in optional_params:
+ optional_params["image_config"] = {}
+ optional_params["image_config"]["image_size"] = image_size
+ else:
+ # Pass through other supported params (like n)
+ optional_params[key] = value
+ elif not drop_params:
+ # If not supported and drop_params is False, pass through
+ optional_params[key] = value
+
+ return optional_params
+
+ def _map_size_to_aspect_ratio(self, size: str) -> str:
+ """
+ Map OpenAI size format to OpenRouter aspect_ratio format.
+
+ OpenAI sizes:
+ - 1024x1024 (square)
+ - 1536x1024 (landscape)
+ - 1024x1536 (portrait)
+ - 1792x1024 (wide landscape, dall-e-3)
+ - 1024x1792 (tall portrait, dall-e-3)
+ - 256x256, 512x512 (dall-e-2)
+ - auto (default)
+
+ OpenRouter aspect_ratios:
+ - 1:1 → 1024×1024 (default)
+ - 2:3 → 832×1248
+ - 3:2 → 1248×832
+ - 3:4 → 864×1184
+ - 4:3 → 1184×864
+ - 4:5 → 896×1152
+ - 5:4 → 1152×896
+ - 9:16 → 768×1344
+ - 16:9 → 1344×768
+ - 21:9 → 1536×672
+ """
+ size_to_aspect_ratio = {
+ # Square formats
+ "256x256": "1:1",
+ "512x512": "1:1",
+ "1024x1024": "1:1",
+ # Landscape formats
+ "1536x1024": "3:2", # 1.5:1 ratio, closest to 3:2
+ "1792x1024": "16:9", # 1.75:1 ratio, closest to 16:9
+ # Portrait formats
+ "1024x1536": "2:3", # 0.67:1 ratio, closest to 2:3
+ "1024x1792": "9:16", # 0.57:1 ratio, closest to 9:16
+ # Default
+ "auto": "1:1",
+ }
+ return size_to_aspect_ratio.get(size, "1:1")
+
+ def _map_quality_to_image_size(self, quality: str) -> Optional[str]:
+ """
+ Map OpenAI quality to OpenRouter image_size format.
+
+ OpenAI quality values:
+ - auto (default) - automatically select best quality
+ - high, medium, low - for GPT image models
+ - hd, standard - for dall-e-3
+
+ OpenRouter image_size values (Gemini only):
+ - 1K → Standard resolution (default)
+ - 2K → Higher resolution
+ - 4K → Highest resolution
+ """
+ quality_to_image_size = {
+ # OpenAI quality mappings
+ "low": "1K",
+ "standard": "1K",
+ "medium": "2K",
+ "high": "4K",
+ "hd": "4K",
+ # Auto defaults to standard
+ "auto": "1K",
+ }
+ return quality_to_image_size.get(quality)
+
+ def _set_usage_and_cost(
+ self,
+ model_response: ImageResponse,
+ response_json: dict,
+ model: str,
+ ) -> None:
+ """
+ Extract and set usage and cost information from OpenRouter response.
+
+ Args:
+ model_response: ImageResponse object to populate
+ response_json: Parsed JSON response from OpenRouter
+ model: The model name
+ """
+ usage_data = response_json.get("usage", {})
+ if usage_data:
+ prompt_tokens = usage_data.get("prompt_tokens", 0)
+ total_tokens = usage_data.get("total_tokens", 0)
+
+ completion_tokens_details = usage_data.get("completion_tokens_details", {})
+ image_tokens = completion_tokens_details.get("image_tokens", 0)
+
+ model_response.usage = ImageUsage(
+ input_tokens=prompt_tokens,
+ input_tokens_details=ImageUsageInputTokensDetails(
+ image_tokens=0, # Input doesn't contain images for generation
+ text_tokens=prompt_tokens,
+ ),
+ output_tokens=image_tokens,
+ total_tokens=total_tokens,
+ )
+
+ cost = usage_data.get("cost")
+ if cost is not None:
+ if not hasattr(model_response, "_hidden_params"):
+ model_response._hidden_params = {}
+ if "additional_headers" not in model_response._hidden_params:
+ model_response._hidden_params["additional_headers"] = {}
+ model_response._hidden_params["additional_headers"][
+ "llm_provider-x-litellm-response-cost"
+ ] = float(cost)
+
+ cost_details = usage_data.get("cost_details", {})
+ if cost_details:
+ if "response_cost_details" not in model_response._hidden_params:
+ model_response._hidden_params["response_cost_details"] = {}
+ model_response._hidden_params["response_cost_details"].update(cost_details)
+
+ model_response._hidden_params["model"] = response_json.get("model", model)
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ """
+ Get the complete URL for OpenRouter image generation.
+
+ OpenRouter uses chat completions endpoint for image generation.
+ Default: https://openrouter.ai/api/v1/chat/completions
+ """
+ if api_base:
+ if not api_base.endswith("/chat/completions"):
+ api_base = api_base.rstrip("/")
+ return f"{api_base}/chat/completions"
+ return api_base
+
+ return "https://openrouter.ai/api/v1/chat/completions"
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ api_key = (
+ api_key
+ or litellm.api_key
+ or get_secret_str("OPENROUTER_API_KEY")
+ )
+ headers.update(
+ {
+ "Authorization": f"Bearer {api_key}",
+ }
+ )
+ return headers
+
+ def transform_image_generation_request(
+ self,
+ model: str,
+ prompt: str,
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ """
+ Transform image generation request to OpenRouter chat completion format.
+
+ Args:
+ model: The model name
+ prompt: The image generation prompt
+ optional_params: Optional parameters (including image_config)
+ litellm_params: LiteLLM parameters
+ headers: Request headers
+
+ Returns:
+ dict: Request body in chat completion format with image_config
+ """
+ request_body = {
+ "model": model,
+ "messages": [
+ {
+ "role": "user",
+ "content": prompt
+ }
+ ]
+ }
+
+ # These will be passed through to OpenRouter
+ for key, value in optional_params.items():
+ if key not in ["model", "messages", "modalities"]:
+ request_body[key] = value
+
+ return request_body
+
+ def transform_image_generation_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: ImageResponse,
+ logging_obj: LiteLLMLoggingObj,
+ request_data: dict,
+ optional_params: dict,
+ litellm_params: dict,
+ encoding: Any,
+ api_key: Optional[str] = None,
+ json_mode: Optional[bool] = None,
+ ) -> ImageResponse:
+ """
+ Transform OpenRouter chat completion response to ImageResponse format.
+
+ Extracts images from the message content and maps usage/cost information.
+
+ Args:
+ model: The model name
+ raw_response: Raw HTTP response from OpenRouter
+ model_response: ImageResponse object to populate
+ logging_obj: Logging object
+ request_data: Original request data
+ optional_params: Optional parameters
+ litellm_params: LiteLLM parameters
+ encoding: Encoding
+ api_key: API key
+ json_mode: JSON mode flag
+
+ Returns:
+ ImageResponse: Populated image response
+ """
+ try:
+ response_json = raw_response.json()
+ except Exception as e:
+ raise OpenRouterException(
+ message=f"Error parsing OpenRouter response: {str(e)}",
+ status_code=raw_response.status_code,
+ headers=raw_response.headers,
+ )
+
+ if not model_response.data:
+ model_response.data = []
+
+ try:
+ choices = response_json.get("choices", [])
+
+ for choice in choices:
+ message = choice.get("message", {})
+ images = message.get("images", [])
+
+ for image_data in images:
+ image_url_obj = image_data.get("image_url", {})
+ image_url = image_url_obj.get("url")
+
+ if image_url:
+ if image_url.startswith("data:"):
+ # Extract base64 data
+ # Format: data:image/png;base64,
+ parts = image_url.split(",", 1)
+ b64_data = parts[1] if len(parts) > 1 else None
+
+ model_response.data.append(
+ ImageObject(
+ b64_json=b64_data,
+ url=None,
+ revised_prompt=None,
+ )
+ )
+ else:
+ model_response.data.append(
+ ImageObject(
+ b64_json=None,
+ url=image_url,
+ revised_prompt=None,
+ )
+ )
+
+ # Extract and set usage and cost information
+ self._set_usage_and_cost(model_response, response_json, model)
+
+ return model_response
+
+ except Exception as e:
+ raise OpenRouterException(
+ message=f"Error transforming OpenRouter image generation response: {str(e)}",
+ status_code=500,
+ headers={},
+ )
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ """Get the appropriate error class for OpenRouter errors."""
+ return OpenRouterException(
+ message=error_message,
+ status_code=status_code,
+ headers=headers,
+ )
diff --git a/litellm/llms/recraft/image_edit/transformation.py b/litellm/llms/recraft/image_edit/transformation.py
index 94449257694..9bf46704ed1 100644
--- a/litellm/llms/recraft/image_edit/transformation.py
+++ b/litellm/llms/recraft/image_edit/transformation.py
@@ -101,7 +101,7 @@ class RecraftImageEditConfig(BaseImageEditConfig):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
@@ -114,6 +114,9 @@ class RecraftImageEditConfig(BaseImageEditConfig):
https://www.recraft.ai/docs#image-to-image
"""
+ if prompt is None:
+ raise ValueError("Recraft image edit requires a prompt.")
+
request_body: RecraftImageEditRequestParams = RecraftImageEditRequestParams(
model=model,
prompt=prompt,
@@ -124,7 +127,7 @@ class RecraftImageEditConfig(BaseImageEditConfig):
#########################################################
# Reuse OpenAI logic: Separate images as `files` and send other parameters as `data`
#########################################################
- files_list = self._get_image_files_for_request(image=image)
+ files_list = self._get_image_files_for_request(image=image) if image is not None else []
data_without_images = {k: v for k, v in request_dict.items() if k != "image"}
return data_without_images, files_list
@@ -132,7 +135,7 @@ class RecraftImageEditConfig(BaseImageEditConfig):
def _get_image_files_for_request(
self,
- image: FileTypes,
+ image: Optional[FileTypes],
) -> List[Tuple[str, Any]]:
files_list: List[Tuple[str, Any]] = []
diff --git a/litellm/llms/stability/image_edit/transformations.py b/litellm/llms/stability/image_edit/transformations.py
index 173fae2d6fd..013e3f27a02 100644
--- a/litellm/llms/stability/image_edit/transformations.py
+++ b/litellm/llms/stability/image_edit/transformations.py
@@ -14,11 +14,11 @@ from httpx._types import RequestFiles
from litellm.llms.base_llm.image_edit.transformation import BaseImageEditConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.images.main import ImageEditOptionalRequestParams
-from litellm.types.router import GenericLiteLLMParams
from litellm.types.llms.stability import (
OPENAI_SIZE_TO_STABILITY_ASPECT_RATIO,
STABILITY_EDIT_ENDPOINTS,
)
+from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import FileTypes, ImageObject, ImageResponse
from litellm.utils import get_model_info
@@ -170,7 +170,7 @@ class StabilityImageEditConfig(BaseImageEditConfig):
def transform_image_edit_request(
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict,
litellm_params: GenericLiteLLMParams,
@@ -186,9 +186,12 @@ class StabilityImageEditConfig(BaseImageEditConfig):
# Populate multipart form-data as separate text fields (data) and files.
# Stability expects prompt/output_format/etc. as normal form fields, not file parts.
data: Dict[str, Any] = {
- "prompt": prompt,
"output_format": "png", # Default to PNG
}
+
+ # Add prompt only if provided (some Stability endpoints don't require it)
+ if prompt is not None:
+ data["prompt"] = prompt
# Handle image parameter - could be a single file or list
image_file = image[0] if isinstance(image, list) else image # type: ignore
files: Dict[str, Any] = {"image": image_file}
diff --git a/litellm/llms/vertex_ai/common_utils.py b/litellm/llms/vertex_ai/common_utils.py
index 2aa6a00c72b..5aa7662f175 100644
--- a/litellm/llms/vertex_ai/common_utils.py
+++ b/litellm/llms/vertex_ai/common_utils.py
@@ -655,16 +655,21 @@ def convert_anyof_null_to_nullable(schema, depth=0):
def add_object_type(schema):
+ # Gemini requires all function parameters to be type OBJECT
+ # Handle case where schema has no properties and no type (e.g. tools with no arguments)
+ if "type" not in schema and "anyOf" not in schema and "oneOf" not in schema and "allOf" not in schema:
+ schema["type"] = "object"
+
properties = schema.get("properties", None)
if properties is not None:
if "required" in schema and schema["required"] is None:
schema.pop("required", None)
# Gemini doesn't accept empty properties for object types
- # If properties is empty, remove it and the type field
+ # If properties is empty, remove it but keep type as object
if not properties:
schema.pop("properties", None)
- schema.pop("type", None)
schema.pop("required", None)
+ schema["type"] = "object"
else:
schema["type"] = "object"
for name, value in properties.items():
@@ -771,6 +776,16 @@ def get_vertex_location_from_url(url: str) -> Optional[str]:
return match.group(1) if match else None
+def get_vertex_model_id_from_url(url: str) -> Optional[str]:
+ """
+ Get the vertex model id from the url
+
+ `https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:streamGenerateContent`
+ """
+ match = re.search(r"/models/([^/:]+)", url)
+ return match.group(1) if match else None
+
+
def replace_project_and_location_in_route(
requested_route: str, vertex_project: str, vertex_location: str
) -> str:
@@ -820,6 +835,15 @@ def construct_target_url(
if "cachedContent" in requested_route:
vertex_version = "v1beta1"
+ # Check if the requested route starts with a version
+ # e.g. /v1beta1/publishers/google/models/gemini-3-pro-preview:streamGenerateContent
+ if requested_route.startswith("/v1/"):
+ vertex_version = "v1"
+ requested_route = requested_route.replace("/v1/", "/", 1)
+ elif requested_route.startswith("/v1beta1/"):
+ vertex_version = "v1beta1"
+ requested_route = requested_route.replace("/v1beta1/", "/", 1)
+
base_requested_route = "{}/projects/{}/locations/{}".format(
vertex_version, vertex_project, vertex_location
)
diff --git a/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py b/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
index cff1bebceb9..289963e917a 100644
--- a/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
+++ b/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
@@ -304,7 +304,7 @@ class ContextCachingEndpoints(VertexBase):
## CHECK IF CACHED ALREADY
generated_cache_key = local_cache_obj.get_cache_key(
- messages=cached_messages, tools=tools
+ messages=cached_messages, tools=tools, model=model
)
google_cache_name = self.check_cache(
cache_key=generated_cache_key,
@@ -433,7 +433,7 @@ class ContextCachingEndpoints(VertexBase):
## CHECK IF CACHED ALREADY
generated_cache_key = local_cache_obj.get_cache_key(
- messages=cached_messages, tools=tools
+ messages=cached_messages, tools=tools, model=model
)
google_cache_name = await self.async_check_cache(
cache_key=generated_cache_key,
diff --git a/litellm/llms/vertex_ai/gemini/transformation.py b/litellm/llms/vertex_ai/gemini/transformation.py
index 22042f7d641..8f1338db92e 100644
--- a/litellm/llms/vertex_ai/gemini/transformation.py
+++ b/litellm/llms/vertex_ai/gemini/transformation.py
@@ -68,6 +68,8 @@ def _convert_detail_to_media_resolution_enum(
) -> Optional[Dict[str, str]]:
if detail == "low":
return {"level": "MEDIA_RESOLUTION_LOW"}
+ elif detail == "medium":
+ return {"level": "MEDIA_RESOLUTION_MEDIUM"}
elif detail == "high":
return {"level": "MEDIA_RESOLUTION_HIGH"}
return None
diff --git a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
index 7a37812bcff..f65a19ac46f 100644
--- a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
+++ b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
@@ -1535,9 +1535,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
f"usageMetadata not found in completion_response. Got={completion_response}"
)
cached_tokens: Optional[int] = None
- audio_tokens: Optional[int] = None
- image_tokens: Optional[int] = None
- text_tokens: Optional[int] = None
+ # Separate variables for prompt tokens by modality
+ prompt_audio_tokens: Optional[int] = None
+ prompt_image_tokens: Optional[int] = None
+ prompt_text_tokens: Optional[int] = None
prompt_tokens_details: Optional[PromptTokensDetailsWrapper] = None
reasoning_tokens: Optional[int] = None
response_tokens: Optional[int] = None
@@ -1580,10 +1581,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
if response_tokens_details is None:
response_tokens_details = CompletionTokensDetailsWrapper()
if response_tokens_details.text_tokens is None:
- image_tokens = response_tokens_details.image_tokens or 0
- audio_tokens_candidate = response_tokens_details.audio_tokens or 0
+ completion_image_tokens = response_tokens_details.image_tokens or 0
+ completion_audio_tokens = response_tokens_details.audio_tokens or 0
calculated_text_tokens = (
- candidates_token_count - image_tokens - audio_tokens_candidate
+ candidates_token_count - completion_image_tokens - completion_audio_tokens
)
response_tokens_details.text_tokens = calculated_text_tokens
#########################################################
@@ -1592,11 +1593,11 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
if "promptTokensDetails" in usage_metadata:
for detail in usage_metadata["promptTokensDetails"]:
if detail["modality"] == "AUDIO":
- audio_tokens = detail.get("tokenCount", 0)
+ prompt_audio_tokens = detail.get("tokenCount", 0)
elif detail["modality"] == "TEXT":
- text_tokens = detail.get("tokenCount", 0)
+ prompt_text_tokens = detail.get("tokenCount", 0)
elif detail["modality"] == "IMAGE":
- image_tokens = detail.get("tokenCount", 0)
+ prompt_image_tokens = detail.get("tokenCount", 0)
## Parse cacheTokensDetails (breakdown of cached tokens by modality)
## When explicit caching is used, Gemini provides this field to show which modalities were cached
@@ -1616,12 +1617,12 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
## Calculate non-cached tokens by subtracting cached from total (per modality)
## This is necessary because promptTokensDetails includes both cached and non-cached tokens
## See: https://github.com/BerriAI/litellm/issues/18750
- if cached_text_tokens is not None and text_tokens is not None:
- text_tokens = text_tokens - cached_text_tokens
- if cached_audio_tokens is not None and audio_tokens is not None:
- audio_tokens = audio_tokens - cached_audio_tokens
- if cached_image_tokens is not None and image_tokens is not None:
- image_tokens = image_tokens - cached_image_tokens
+ if cached_text_tokens is not None and prompt_text_tokens is not None:
+ prompt_text_tokens = prompt_text_tokens - cached_text_tokens
+ if cached_audio_tokens is not None and prompt_audio_tokens is not None:
+ prompt_audio_tokens = prompt_audio_tokens - cached_audio_tokens
+ if cached_image_tokens is not None and prompt_image_tokens is not None:
+ prompt_image_tokens = prompt_image_tokens - cached_image_tokens
if "thoughtsTokenCount" in usage_metadata:
reasoning_tokens = usage_metadata["thoughtsTokenCount"]
@@ -1632,9 +1633,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
prompt_tokens_details = PromptTokensDetailsWrapper(
cached_tokens=cached_tokens,
- audio_tokens=audio_tokens,
- text_tokens=text_tokens,
- image_tokens=image_tokens,
+ audio_tokens=prompt_audio_tokens,
+ text_tokens=prompt_text_tokens,
+ image_tokens=prompt_image_tokens,
)
completion_tokens = response_tokens or completion_response["usageMetadata"].get(
diff --git a/litellm/llms/vertex_ai/image_edit/vertex_gemini_transformation.py b/litellm/llms/vertex_ai/image_edit/vertex_gemini_transformation.py
index 174d05cf7cf..154d5669eb8 100644
--- a/litellm/llms/vertex_ai/image_edit/vertex_gemini_transformation.py
+++ b/litellm/llms/vertex_ai/image_edit/vertex_gemini_transformation.py
@@ -151,7 +151,7 @@ class VertexAIGeminiImageEditConfig(BaseImageEditConfig, VertexLLM):
def transform_image_edit_request( # type: ignore[override]
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict[str, Any],
litellm_params: GenericLiteLLMParams,
@@ -161,6 +161,9 @@ class VertexAIGeminiImageEditConfig(BaseImageEditConfig, VertexLLM):
if not inline_parts:
raise ValueError("Vertex AI Gemini image edit requires at least one image.")
+ if prompt is None:
+ raise ValueError("Vertex AI Gemini image edit requires a prompt.")
+
# Correct format for Vertex AI Gemini image editing
contents = {
"role": "USER",
diff --git a/litellm/llms/vertex_ai/image_edit/vertex_imagen_transformation.py b/litellm/llms/vertex_ai/image_edit/vertex_imagen_transformation.py
index b61af6ffd3a..337a4bd4dd6 100644
--- a/litellm/llms/vertex_ai/image_edit/vertex_imagen_transformation.py
+++ b/litellm/llms/vertex_ai/image_edit/vertex_imagen_transformation.py
@@ -143,17 +143,22 @@ class VertexAIImagenImageEditConfig(BaseImageEditConfig, VertexLLM):
def transform_image_edit_request( # type: ignore[override]
self,
model: str,
- prompt: str,
+ prompt: Optional[str],
image: FileTypes,
image_edit_optional_request_params: Dict[str, Any],
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Tuple[Dict[str, Any], Optional[RequestFiles]]:
# Prepare reference images in the correct Imagen format
+ if image is None:
+ raise ValueError("Vertex AI Imagen image edit requires at least one reference image.")
reference_images = self._prepare_reference_images(image, image_edit_optional_request_params)
if not reference_images:
raise ValueError("Vertex AI Imagen image edit requires at least one reference image.")
+ if prompt is None:
+ raise ValueError("Vertex AI Imagen image edit requires a prompt.")
+
# Correct Imagen instances format
instances = [
{
diff --git a/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py b/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py
index c22072af2f3..0bedef3276b 100644
--- a/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py
+++ b/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py
@@ -1,11 +1,16 @@
from typing import Any, Dict, List, Optional, Tuple
+from litellm.llms.anthropic.common_utils import AnthropicModelInfo
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
+from litellm.types.llms.anthropic import (
+ ANTHROPIC_BETA_HEADER_VALUES,
+ ANTHROPIC_HOSTED_TOOLS,
+)
+from litellm.types.llms.anthropic_tool_search import get_tool_search_beta_header
from litellm.types.llms.vertex_ai import VertexPartnerProvider
from litellm.types.router import GenericLiteLLMParams
-from litellm.types.llms.anthropic import ANTHROPIC_BETA_HEADER_VALUES, ANTHROPIC_HOSTED_TOOLS
from ....vertex_llm_base import VertexBase
@@ -51,13 +56,28 @@ class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, Vert
headers["content-type"] = "application/json"
- # Add web search beta header for Vertex AI only if not already set
- if "anthropic-beta" not in headers:
- tools = optional_params.get("tools", [])
- for tool in tools:
- if isinstance(tool, dict) and tool.get("type", "").startswith(ANTHROPIC_HOSTED_TOOLS.WEB_SEARCH.value):
- headers["anthropic-beta"] = ANTHROPIC_BETA_HEADER_VALUES.WEB_SEARCH_2025_03_05.value
- break
+ # Add beta headers for Vertex AI
+ tools = optional_params.get("tools", [])
+ beta_values: set[str] = set()
+
+ # Get existing beta headers if any
+ existing_beta = headers.get("anthropic-beta")
+ if existing_beta:
+ beta_values.update(b.strip() for b in existing_beta.split(","))
+
+ # Check for web search tool
+ for tool in tools:
+ if isinstance(tool, dict) and tool.get("type", "").startswith(ANTHROPIC_HOSTED_TOOLS.WEB_SEARCH.value):
+ beta_values.add(ANTHROPIC_BETA_HEADER_VALUES.WEB_SEARCH_2025_03_05.value)
+ break
+
+ # Check for tool search tools - Vertex AI uses different beta header
+ anthropic_model_info = AnthropicModelInfo()
+ if anthropic_model_info.is_tool_search_used(tools):
+ beta_values.add(get_tool_search_beta_header("vertex_ai"))
+
+ if beta_values:
+ headers["anthropic-beta"] = ",".join(beta_values)
return headers, api_base
diff --git a/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py b/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py
index 24425f08b56..1df07f405e6 100644
--- a/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py
+++ b/litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py
@@ -69,6 +69,9 @@ class VertexAIAnthropicConfig(AnthropicConfig):
data.pop("model", None) # vertex anthropic doesn't accept 'model' parameter
+ # VertexAI doesn't support output_format parameter, remove it if present
+ data.pop("output_format", None)
+
tools = optional_params.get("tools")
tool_search_used = self.is_tool_search_used(tools)
auto_betas = self.get_anthropic_beta_list(
@@ -89,6 +92,37 @@ class VertexAIAnthropicConfig(AnthropicConfig):
return data
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ """
+ Override parent method to ensure VertexAI always uses tool-based structured outputs.
+ VertexAI doesn't support the output_format parameter, so we force all models
+ to use the tool-based approach for structured outputs.
+ """
+ # Temporarily override model name to force tool-based approach
+ # This ensures Claude Sonnet 4.5 uses tools instead of output_format
+ original_model = model
+ if "response_format" in non_default_params:
+ model = "claude-3-sonnet-20240229" # Use a model that will use tool-based approach
+
+ # Call parent method with potentially modified model name
+ optional_params = super().map_openai_params(
+ non_default_params=non_default_params,
+ optional_params=optional_params,
+ model=model,
+ drop_params=drop_params,
+ )
+
+ # Restore original model name for any other processing
+ model = original_model
+
+ return optional_params
+
def transform_response(
self,
model: str,
diff --git a/litellm/llms/watsonx/audio_transcription/transformation.py b/litellm/llms/watsonx/audio_transcription/transformation.py
index c7e6a77b96f..5944705258e 100644
--- a/litellm/llms/watsonx/audio_transcription/transformation.py
+++ b/litellm/llms/watsonx/audio_transcription/transformation.py
@@ -22,7 +22,7 @@ from ...base_llm.audio_transcription.transformation import (
from ...openai.transcriptions.whisper_transformation import (
OpenAIWhisperAudioTranscriptionConfig,
)
-from ..common_utils import IBMWatsonXMixin, _get_api_params
+from ..common_utils import IBMWatsonXMixin
class IBMWatsonXAudioTranscriptionConfig(
@@ -48,7 +48,7 @@ class IBMWatsonXAudioTranscriptionConfig(
) -> Dict:
"""
Validate environment for audio transcription.
-
+
Removes Content-Type header so httpx can set multipart/form-data automatically.
"""
result = IBMWatsonXMixin.validate_environment(
@@ -88,31 +88,37 @@ class IBMWatsonXAudioTranscriptionConfig(
) -> AudioTranscriptionRequestData:
"""
Transform the audio transcription request for WatsonX.
-
+
WatsonX expects multipart/form-data with:
- file: the audio file
- model: the model name (without watsonx/ prefix)
- project_id: the project ID (as form field, not query param)
+ - space_id: the space ID (as form field, not query param)
- other optional params
"""
# Use common utility to process the audio file
processed_audio = process_audio_file(audio_file)
-
- # Get API params to extract project_id
- api_params = _get_api_params(params=optional_params.copy())
-
+ project_id = optional_params.get("project_id") or optional_params.get(
+ "watsonx_project"
+ )
+ space_id = optional_params.get("space_id")
+ # api_params = _get_api_params(params=optional_params, model=model)
+
# Initialize form data with required fields
- form_data: WatsonXAudioTranscriptionRequestBody = {
- "model": model,
- "project_id": api_params.get("project_id", ""),
- }
-
+ form_data: WatsonXAudioTranscriptionRequestBody = {"model": model}
+
+ # Only add project_id or space_id if they were explicitly provided by the user
+ if project_id:
+ form_data["project_id"] = project_id
+ elif space_id:
+ form_data["space_id"] = space_id
+
# Add supported OpenAI params to form data
supported_params = self.get_supported_openai_params(model)
for key, value in optional_params.items():
if key in supported_params and value is not None:
form_data[key] = value # type: ignore
-
+
# Prepare files dict with the audio file
files = {
"file": (
@@ -121,10 +127,10 @@ class IBMWatsonXAudioTranscriptionConfig(
processed_audio.content_type,
)
}
-
+
# Convert TypedDict to regular dict for AudioTranscriptionRequestData
form_data_dict: Dict[str, Any] = dict(form_data)
-
+
return AudioTranscriptionRequestData(data=form_data_dict, files=files)
def get_complete_url(
@@ -140,8 +146,8 @@ class IBMWatsonXAudioTranscriptionConfig(
Construct the complete URL for WatsonX audio transcription.
URL format: {api_base}/ml/v1/audio/transcriptions?version={version}
-
- Note: project_id is sent as form data, not as a query parameter
+
+ Note: project_id or space_id is sent as form data, not as a query parameter
"""
# Get base URL
url = self._get_base_url(api_base=api_base)
@@ -151,9 +157,10 @@ class IBMWatsonXAudioTranscriptionConfig(
url = f"{url}/ml/v1/audio/transcriptions"
# Add version parameter (only version in query string, not project_id)
- api_version = optional_params.get(
- "api_version", None
- ) or litellm.WATSONX_DEFAULT_API_VERSION
+ api_version = (
+ optional_params.get("api_version", None)
+ or litellm.WATSONX_DEFAULT_API_VERSION
+ )
url = f"{url}?version={api_version}"
return url
@@ -164,7 +171,7 @@ class IBMWatsonXAudioTranscriptionConfig(
) -> TranscriptionResponse:
"""
Transform the audio transcription response from WatsonX.
-
+
WatsonX may include a 'model' field in the response, which needs to be
removed before creating the TranscriptionResponse object.
"""
@@ -179,26 +186,30 @@ class IBMWatsonXAudioTranscriptionConfig(
# TranscriptionResponse only accepts 'text' and 'usage' in __init__()
text = raw_response_json.get("text")
usage = raw_response_json.get("usage")
-
+
# Create response with only valid fields
response_kwargs = {}
if text is not None:
response_kwargs["text"] = text
if usage is not None:
response_kwargs["usage"] = usage
-
+
if not response_kwargs:
raise ValueError(
"Invalid response format. Received response does not match the expected format. Got: ",
raw_response_json,
)
-
+
response = TranscriptionResponse(**response_kwargs)
-
+
# Add other fields using dictionary-style assignment (like duration, task, etc.)
# Skip fields that TranscriptionResponse doesn't accept in __init__()
for key, value in raw_response_json.items():
- if key not in ["text", "usage", "model"]: # text/usage already set, model should be excluded
+ if key not in [
+ "text",
+ "usage",
+ "model",
+ ]: # text/usage already set, model should be excluded
response[key] = value
-
+
return response
diff --git a/litellm/llms/watsonx/chat/handler.py b/litellm/llms/watsonx/chat/handler.py
index bc0effe4a1a..40ccc45497b 100644
--- a/litellm/llms/watsonx/chat/handler.py
+++ b/litellm/llms/watsonx/chat/handler.py
@@ -40,7 +40,7 @@ class WatsonXChatHandler(OpenAILikeChatHandler):
streaming_decoder: Optional[CustomStreamingDecoder] = None,
fake_stream: bool = False,
):
- api_params = _get_api_params(params=optional_params)
+ api_params = _get_api_params(params=optional_params, model=model)
## UPDATE HEADERS
headers = watsonx_chat_transformation.validate_environment(
diff --git a/litellm/llms/watsonx/chat/transformation.py b/litellm/llms/watsonx/chat/transformation.py
index 0bb96673ef6..157493a4ce8 100644
--- a/litellm/llms/watsonx/chat/transformation.py
+++ b/litellm/llms/watsonx/chat/transformation.py
@@ -10,7 +10,6 @@ from litellm import verbose_logger
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.watsonx import (
WatsonXAIEndpoint,
- WatsonXAPIParams,
WatsonXModelPattern,
)
@@ -115,18 +114,6 @@ class IBMWatsonXChatConfig(IBMWatsonXMixin, OpenAIGPTConfig):
)
return url
- def _prepare_payload(self, model: str, api_params: WatsonXAPIParams) -> dict:
- """
- Prepare payload for deployment models.
- Deployment models cannot have 'model_id' or 'model' in the request body.
- """
- payload: dict = {}
- payload["model_id"] = None if model.startswith("deployment/") else model
- payload["project_id"] = (
- None if model.startswith("deployment/") else api_params["project_id"]
- )
- return payload
-
@staticmethod
def _apply_prompt_template_core(
model: str, messages: List[Dict[str, str]], hf_template_fn
diff --git a/litellm/llms/watsonx/common_utils.py b/litellm/llms/watsonx/common_utils.py
index 0207020534c..774f6dc1f3d 100644
--- a/litellm/llms/watsonx/common_utils.py
+++ b/litellm/llms/watsonx/common_utils.py
@@ -80,9 +80,7 @@ def _generate_watsonx_token(api_key: Optional[str], token: Optional[str]) -> str
return token
-def _get_api_params(
- params: dict,
-) -> WatsonXAPIParams:
+def _get_api_params(params: dict, model: Optional[str] = None) -> WatsonXAPIParams:
"""
Find watsonx.ai credentials in the params or environment variables and return the headers for authentication.
"""
@@ -118,10 +116,15 @@ def _get_api_params(
or get_secret_str("SPACE_ID")
)
- if project_id is None:
+ if (
+ project_id is None
+ and space_id is None
+ and model is not None
+ and not model.startswith("deployment/")
+ ):
raise WatsonXAIError(
status_code=401,
- message="Error: Watsonx project_id not set. Set WX_PROJECT_ID in environment variables or pass in as a parameter.",
+ message="Error: Watsonx project_id and space_id not set. Set WX_PROJECT_ID or WX_SPACE_ID in environment variables or pass in as a parameter.",
)
return WatsonXAPIParams(
@@ -146,7 +149,9 @@ async def _aconvert_watsonx_messages_core(
model_prompt_dict = custom_prompt_dict[model]
return ptf.custom_prompt(
messages=messages,
- role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
+ role_dict=model_prompt_dict.get(
+ "role_dict", model_prompt_dict.get("roles")
+ ),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
@@ -180,7 +185,9 @@ def _convert_watsonx_messages_core(
model_prompt_dict = custom_prompt_dict[model]
return ptf.custom_prompt(
messages=messages,
- role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
+ role_dict=model_prompt_dict.get(
+ "role_dict", model_prompt_dict.get("roles")
+ ),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
@@ -200,7 +207,10 @@ def _convert_watsonx_messages_core(
async def aconvert_watsonx_messages_to_prompt(
- model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
+ model: str,
+ messages: List[AllMessageValues],
+ provider: str,
+ custom_prompt_dict: Dict,
) -> str:
"""Async version of convert_watsonx_messages_to_prompt"""
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
@@ -215,7 +225,10 @@ async def aconvert_watsonx_messages_to_prompt(
def convert_watsonx_messages_to_prompt(
- model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
+ model: str,
+ messages: List[AllMessageValues],
+ provider: str,
+ custom_prompt_dict: Dict,
) -> str:
"""Sync version of convert_watsonx_messages_to_prompt"""
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
@@ -254,7 +267,8 @@ class IBMWatsonXMixin:
)
zen_api_key = cast(
Optional[str],
- optional_params.pop("zen_api_key", None) or get_secret_str("WATSONX_ZENAPIKEY"),
+ optional_params.pop("zen_api_key", None)
+ or get_secret_str("WATSONX_ZENAPIKEY"),
)
if token:
headers["Authorization"] = f"Bearer {token}"
@@ -360,5 +374,8 @@ class IBMWatsonXMixin:
{}
) # Deployment models do not support 'space_id' or 'project_id' in their payload
payload["model_id"] = model
- payload["project_id"] = api_params["project_id"]
+ if api_params["project_id"] is not None:
+ payload["project_id"] = api_params["project_id"]
+ else:
+ payload["space_id"] = api_params["space_id"]
return payload
diff --git a/litellm/llms/watsonx/completion/transformation.py b/litellm/llms/watsonx/completion/transformation.py
index 3c1229ecd2b..7180e12162a 100644
--- a/litellm/llms/watsonx/completion/transformation.py
+++ b/litellm/llms/watsonx/completion/transformation.py
@@ -228,13 +228,17 @@ class IBMWatsonXAIConfig(IBMWatsonXMixin, BaseConfig):
"us-south",
]
- def _build_request_payload(self, model: str, prompt: str, optional_params: Dict) -> Dict:
+ def _build_request_payload(
+ self, model: str, prompt: str, optional_params: Dict
+ ) -> Dict:
"""Shared logic to build request payload"""
extra_body_params = optional_params.pop("extra_body", {})
optional_params.update(extra_body_params)
- watsonx_api_params = _get_api_params(params=optional_params)
- watsonx_auth_payload = self._prepare_payload(model=model, api_params=watsonx_api_params)
-
+ watsonx_api_params = _get_api_params(params=optional_params, model=model)
+ watsonx_auth_payload = self._prepare_payload(
+ model=model, api_params=watsonx_api_params
+ )
+
return {
"input": prompt,
"moderations": optional_params.pop("moderations", {}),
@@ -242,21 +246,43 @@ class IBMWatsonXAIConfig(IBMWatsonXMixin, BaseConfig):
**watsonx_auth_payload,
}
- async def atransform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
+ async def atransform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: Dict,
+ litellm_params: Dict,
+ headers: Dict,
+ ) -> Dict:
"""Async version of transform_request"""
from litellm.llms.watsonx.common_utils import (
aconvert_watsonx_messages_to_prompt,
)
-
+
provider = model.split("/")[0]
- prompt = await aconvert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
- return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
-
- def transform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
+ prompt = await aconvert_watsonx_messages_to_prompt(
+ model=model, messages=messages, provider=provider, custom_prompt_dict={}
+ )
+ return self._build_request_payload(
+ model=model, prompt=prompt, optional_params=optional_params
+ )
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: Dict,
+ litellm_params: Dict,
+ headers: Dict,
+ ) -> Dict:
"""Sync version of transform_request"""
provider = model.split("/")[0]
- prompt = convert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
- return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
+ prompt = convert_watsonx_messages_to_prompt(
+ model=model, messages=messages, provider=provider, custom_prompt_dict={}
+ )
+ return self._build_request_payload(
+ model=model, prompt=prompt, optional_params=optional_params
+ )
def transform_response(
self,
diff --git a/litellm/llms/watsonx/embed/transformation.py b/litellm/llms/watsonx/embed/transformation.py
index 21f508da015..930212e3ef3 100644
--- a/litellm/llms/watsonx/embed/transformation.py
+++ b/litellm/llms/watsonx/embed/transformation.py
@@ -37,7 +37,7 @@ class IBMWatsonXEmbeddingConfig(IBMWatsonXMixin, BaseEmbeddingConfig):
optional_params: dict,
headers: dict,
) -> dict:
- watsonx_api_params = _get_api_params(params=optional_params)
+ watsonx_api_params = _get_api_params(params=optional_params, model=model)
watsonx_auth_payload = self._prepare_payload(
model=model,
api_params=watsonx_api_params,
diff --git a/litellm/main.py b/litellm/main.py
index a1992752bdb..969cf55a3d6 100644
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -28,6 +28,7 @@ from typing import (
Callable,
Coroutine,
Dict,
+ Iterable,
List,
Literal,
Mapping,
@@ -1094,23 +1095,68 @@ def completion( # type: ignore # noqa: PLR0915
# validate tool_choice
tool_choice = validate_chat_completion_tool_choice(tool_choice=tool_choice)
+ ######### unpacking kwargs #####################
+ args = locals()
+
skip_mcp_handler = kwargs.pop("_skip_mcp_handler", False)
if not skip_mcp_handler and tools:
from litellm.responses.mcp.chat_completions_handler import (
- handle_chat_completion_with_mcp,
+ acompletion_with_mcp,
)
+ from litellm.responses.mcp.litellm_proxy_mcp_handler import (
+ LiteLLM_Proxy_MCP_Handler,
+ )
+ from litellm.types.llms.openai import ToolParam
- mcp_handler_context = locals().copy()
- completion_callable = globals().get("acompletion")
- mcp_result = run_async_function(
- handle_chat_completion_with_mcp,
- mcp_handler_context,
- completion_callable,
- )
- if mcp_result is not None:
- return mcp_result
- ######### unpacking kwargs #####################
- args = locals()
+ # Check if MCP tools are present (following responses pattern)
+ # Cast tools to Optional[Iterable[ToolParam]] for type checking
+ tools_for_mcp = cast(Optional[Iterable[ToolParam]], tools)
+ if LiteLLM_Proxy_MCP_Handler._should_use_litellm_mcp_gateway(tools=tools_for_mcp):
+ # Return coroutine - acompletion will await it
+ # completion() can return a coroutine when MCP tools are present, which acompletion() awaits
+ return acompletion_with_mcp( # type: ignore[return-value]
+ model=model,
+ messages=messages,
+ functions=functions,
+ function_call=function_call,
+ timeout=timeout,
+ temperature=temperature,
+ top_p=top_p,
+ n=n,
+ stream=stream,
+ stream_options=stream_options,
+ stop=stop,
+ max_tokens=max_tokens,
+ max_completion_tokens=max_completion_tokens,
+ modalities=modalities,
+ prediction=prediction,
+ audio=audio,
+ presence_penalty=presence_penalty,
+ frequency_penalty=frequency_penalty,
+ logit_bias=logit_bias,
+ user=user,
+ response_format=response_format,
+ seed=seed,
+ tools=tools,
+ tool_choice=tool_choice,
+ parallel_tool_calls=parallel_tool_calls,
+ logprobs=logprobs,
+ top_logprobs=top_logprobs,
+ deployment_id=deployment_id,
+ reasoning_effort=reasoning_effort,
+ verbosity=verbosity,
+ safety_identifier=safety_identifier,
+ service_tier=service_tier,
+ base_url=base_url,
+ api_version=api_version,
+ api_key=api_key,
+ model_list=model_list,
+ extra_headers=extra_headers,
+ thinking=thinking,
+ web_search_options=web_search_options,
+ shared_session=shared_session,
+ **kwargs,
+ )
api_base = kwargs.get("api_base", None)
mock_response: Optional[MOCK_RESPONSE_TYPE] = kwargs.get("mock_response", None)
mock_tool_calls = kwargs.get("mock_tool_calls", None)
@@ -4227,6 +4273,71 @@ async def acompletion_with_retries(*args, **kwargs):
return await retryer(original_function, *args, **kwargs)
+def responses_with_retries(*args, **kwargs):
+ """
+ Executes a litellm.responses() with retries
+ """
+ try:
+ import tenacity
+ except Exception as e:
+ raise Exception(
+ f"tenacity import failed please run `pip install tenacity`. Error{e}"
+ )
+
+ from litellm.responses.main import responses
+
+ num_retries = kwargs.pop("num_retries", 3)
+ # reset retries in .responses()
+ kwargs["max_retries"] = 0
+ kwargs["num_retries"] = 0
+ retry_strategy: Literal["exponential_backoff_retry", "constant_retry"] = kwargs.pop(
+ "retry_strategy", "constant_retry"
+ ) # type: ignore
+ original_function = kwargs.pop("original_function", responses)
+ if retry_strategy == "exponential_backoff_retry":
+ retryer = tenacity.Retrying(
+ wait=tenacity.wait_exponential(multiplier=1, max=10),
+ stop=tenacity.stop_after_attempt(num_retries),
+ reraise=True,
+ )
+ else:
+ retryer = tenacity.Retrying(
+ stop=tenacity.stop_after_attempt(num_retries), reraise=True
+ )
+ return retryer(original_function, *args, **kwargs)
+
+
+async def aresponses_with_retries(*args, **kwargs):
+ """
+ Executes a litellm.aresponses() with retries
+ """
+ try:
+ import tenacity
+ except Exception as e:
+ raise Exception(
+ f"tenacity import failed please run `pip install tenacity`. Error{e}"
+ )
+
+ from litellm.responses.main import aresponses
+
+ num_retries = kwargs.pop("num_retries", 3)
+ kwargs["max_retries"] = 0
+ kwargs["num_retries"] = 0
+ retry_strategy = kwargs.pop("retry_strategy", "constant_retry")
+ original_function = kwargs.pop("original_function", aresponses)
+ if retry_strategy == "exponential_backoff_retry":
+ retryer = tenacity.AsyncRetrying(
+ wait=tenacity.wait_exponential(multiplier=1, max=10),
+ stop=tenacity.stop_after_attempt(num_retries),
+ reraise=True,
+ )
+ else:
+ retryer = tenacity.AsyncRetrying(
+ stop=tenacity.stop_after_attempt(num_retries), reraise=True
+ )
+ return await retryer(original_function, *args, **kwargs)
+
+
### EMBEDDING ENDPOINTS ####################
@client
async def aembedding(*args, **kwargs) -> EmbeddingResponse:
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index ed218d50af7..470d598a25f 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -1329,6 +1329,24 @@
"supports_tool_choice": true,
"supports_vision": true
},
+ "azure_ai/claude-opus-4-5": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
"azure_ai/claude-opus-4-1": {
"input_cost_per_token": 1.5e-05,
"litellm_provider": "azure_ai",
@@ -1626,7 +1644,7 @@
"cache_read_input_token_cost": 1.4e-07,
"input_cost_per_token": 1.38e-06,
"litellm_provider": "azure",
- "max_input_tokens": 272000,
+ "max_input_tokens": 128000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -1926,7 +1944,7 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "azure",
- "max_input_tokens": 272000,
+ "max_input_tokens": 128000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -3074,9 +3092,9 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "azure",
- "max_input_tokens": 272000,
- "max_output_tokens": 128000,
- "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1e-05,
"source": "https://azure.microsoft.com/en-us/blog/gpt-5-in-azure-ai-foundry-the-future-of-ai-apps-and-agents-starts-here/",
@@ -3359,7 +3377,7 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "azure",
- "max_input_tokens": 272000,
+ "max_input_tokens": 128000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -3422,7 +3440,7 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "azure",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
@@ -3482,7 +3500,7 @@
"cache_read_input_token_cost": 1.75e-07,
"input_cost_per_token": 1.75e-06,
"litellm_provider": "azure",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -3517,7 +3535,7 @@
"input_cost_per_token": 1.75e-06,
"input_cost_per_token_priority": 3.5e-06,
"litellm_provider": "azure",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -3616,10 +3634,41 @@
"supports_tool_choice": true,
"supports_vision": true
},
+ "azure/gpt-5.2-codex": {
+ "cache_read_input_token_cost": 1.75e-07,
+ "input_cost_per_token": 1.75e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.4e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
"azure/gpt-5.2-pro": {
"input_cost_per_token": 2.1e-05,
"litellm_provider": "azure",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
@@ -3650,7 +3699,7 @@
"azure/gpt-5.2-pro-2025-12-11": {
"input_cost_per_token": 2.1e-05,
"litellm_provider": "azure",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
@@ -4666,7 +4715,7 @@
"cache_read_input_token_cost": 1.4e-07,
"input_cost_per_token": 1.38e-06,
"litellm_provider": "azure",
- "max_input_tokens": 272000,
+ "max_input_tokens": 128000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -5409,28 +5458,28 @@
"supports_web_search": true
},
"azure_ai/grok-3": {
- "input_cost_per_token": 3.3e-06,
+ "input_cost_per_token": 3e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
- "output_cost_per_token": 1.65e-05,
- "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/grok/",
"supports_function_calling": true,
"supports_response_schema": false,
"supports_tool_choice": true,
"supports_web_search": true
},
"azure_ai/grok-3-mini": {
- "input_cost_per_token": 2.75e-07,
+ "input_cost_per_token": 2.5e-07,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
- "output_cost_per_token": 1.38e-06,
- "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
+ "output_cost_per_token": 1.27e-06,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/grok/",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": false,
@@ -5438,22 +5487,22 @@
"supports_web_search": true
},
"azure_ai/grok-4": {
- "input_cost_per_token": 5.5e-06,
+ "input_cost_per_token": 3e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
- "output_cost_per_token": 2.75e-05,
- "source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/grok/",
"supports_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_web_search": true
},
"azure_ai/grok-4-fast-non-reasoning": {
- "input_cost_per_token": 4.3e-07,
- "output_cost_per_token": 1.73e-06,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 5e-07,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
@@ -5465,28 +5514,28 @@
"supports_web_search": true
},
"azure_ai/grok-4-fast-reasoning": {
- "input_cost_per_token": 4.3e-07,
- "output_cost_per_token": 1.73e-06,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 5e-07,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
- "source": "https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/announcing-the-grok-4-fast-models-from-xai-now-available-in-azure-ai-foundry/4456701",
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/grok/",
"supports_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_web_search": true
},
"azure_ai/grok-code-fast-1": {
- "input_cost_per_token": 3.5e-06,
+ "input_cost_per_token": 2e-07,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
- "output_cost_per_token": 1.75e-05,
- "source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/grok/",
"supports_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true,
@@ -6598,6 +6647,20 @@
"supports_tool_choice": true
},
"cerebras/zai-glm-4.6": {
+ "deprecation_date": "2026-01-20",
+ "input_cost_per_token": 2.25e-06,
+ "litellm_provider": "cerebras",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.75e-06,
+ "source": "https://www.cerebras.ai/pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "cerebras/zai-glm-4.7": {
"input_cost_per_token": 2.25e-06,
"litellm_provider": "cerebras",
"max_input_tokens": 128000,
@@ -7795,14 +7858,14 @@
"supports_vision": true
},
"deepseek-chat": {
- "cache_read_input_token_cost": 6e-08,
- "input_cost_per_token": 6e-07,
+ "cache_read_input_token_cost": 2.8e-08,
+ "input_cost_per_token": 2.8e-07,
"litellm_provider": "deepseek",
"max_input_tokens": 131072,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
- "output_cost_per_token": 1.7e-06,
+ "output_cost_per_token": 4.2e-07,
"source": "https://api-docs.deepseek.com/quick_start/pricing",
"supported_endpoints": [
"/v1/chat/completions"
@@ -7816,14 +7879,14 @@
"supports_tool_choice": true
},
"deepseek-reasoner": {
- "cache_read_input_token_cost": 6e-08,
- "input_cost_per_token": 6e-07,
+ "cache_read_input_token_cost": 2.8e-08,
+ "input_cost_per_token": 2.8e-07,
"litellm_provider": "deepseek",
"max_input_tokens": 131072,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
- "output_cost_per_token": 1.7e-06,
+ "output_cost_per_token": 4.2e-07,
"source": "https://api-docs.deepseek.com/quick_start/pricing",
"supported_endpoints": [
"/v1/chat/completions"
@@ -8592,7 +8655,7 @@
"input_cost_per_token": 1.24999e-06,
"input_dbu_cost_per_token": 1.7857e-05,
"litellm_provider": "databricks",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"metadata": {
@@ -8607,7 +8670,7 @@
"input_cost_per_token": 1.24999e-06,
"input_dbu_cost_per_token": 1.7857e-05,
"litellm_provider": "databricks",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"metadata": {
@@ -8622,7 +8685,7 @@
"input_cost_per_token": 2.4997000000000006e-07,
"input_dbu_cost_per_token": 3.571e-06,
"litellm_provider": "databricks",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"metadata": {
@@ -8637,7 +8700,7 @@
"input_cost_per_token": 4.998e-08,
"input_dbu_cost_per_token": 7.14e-07,
"litellm_provider": "databricks",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"metadata": {
@@ -10027,15 +10090,15 @@
},
"deepseek/deepseek-chat": {
"cache_creation_input_token_cost": 0.0,
- "cache_read_input_token_cost": 7e-08,
- "input_cost_per_token": 2.7e-07,
- "input_cost_per_token_cache_hit": 7e-08,
+ "cache_read_input_token_cost": 2.8e-08,
+ "input_cost_per_token": 2.8e-07,
+ "input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "deepseek",
- "max_input_tokens": 65536,
+ "max_input_tokens": 128000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
- "output_cost_per_token": 1.1e-06,
+ "output_cost_per_token": 4.2e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
@@ -10071,14 +10134,15 @@
"supports_tool_choice": true
},
"deepseek/deepseek-reasoner": {
- "input_cost_per_token": 5.5e-07,
- "input_cost_per_token_cache_hit": 1.4e-07,
+ "cache_read_input_token_cost": 2.8e-08,
+ "input_cost_per_token": 2.8e-07,
+ "input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "deepseek",
- "max_input_tokens": 65536,
+ "max_input_tokens": 128000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
- "output_cost_per_token": 2.19e-06,
+ "output_cost_per_token": 4.2e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
@@ -12978,6 +13042,7 @@
},
"gemini-2.5-pro": {
"cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"cache_creation_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -13160,7 +13225,8 @@
"supports_web_search": true
},
"gemini-2.5-pro-exp-03-25": {
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
"litellm_provider": "vertex_ai-language-models",
@@ -13204,7 +13270,8 @@
},
"gemini-2.5-pro-preview-03-25": {
"deprecation_date": "2025-12-02",
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 1.25e-06,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -13250,7 +13317,8 @@
},
"gemini-2.5-pro-preview-05-06": {
"deprecation_date": "2025-12-02",
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 1.25e-06,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -13298,7 +13366,8 @@
"supports_web_search": true
},
"gemini-2.5-pro-preview-06-05": {
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 1.25e-06,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -13343,7 +13412,8 @@
"supports_web_search": true
},
"gemini-2.5-pro-preview-tts": {
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 7e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -14832,7 +14902,8 @@
"tpm": 250000
},
"gemini/gemini-2.5-pro": {
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
"litellm_provider": "gemini",
@@ -15094,7 +15165,8 @@
},
"gemini/gemini-2.5-pro-preview-03-25": {
"deprecation_date": "2025-12-02",
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 7e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -15135,7 +15207,8 @@
},
"gemini/gemini-2.5-pro-preview-05-06": {
"deprecation_date": "2025-12-02",
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 7e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -15176,7 +15249,8 @@
"tpm": 10000000
},
"gemini/gemini-2.5-pro-preview-06-05": {
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 7e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -15217,7 +15291,8 @@
"tpm": 10000000
},
"gemini/gemini-2.5-pro-preview-tts": {
- "cache_read_input_token_cost": 3.125e-07,
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_audio_token": 7e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
@@ -17596,7 +17671,7 @@
"input_cost_per_token": 1.75e-06,
"input_cost_per_token_priority": 3.5e-06,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -17633,7 +17708,7 @@
"input_cost_per_token": 1.75e-06,
"input_cost_per_token_priority": 3.5e-06,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -17701,7 +17776,7 @@
"gpt-5.2-pro": {
"input_cost_per_token": 2.1e-05,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
@@ -17732,7 +17807,7 @@
"gpt-5.2-pro-2025-12-11": {
"input_cost_per_token": 2.1e-05,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
@@ -17764,7 +17839,7 @@
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 128000,
"max_output_tokens": 272000,
"max_tokens": 272000,
"mode": "responses",
@@ -17797,7 +17872,7 @@
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 128000,
"max_output_tokens": 272000,
"max_tokens": 272000,
"mode": "responses",
@@ -17869,9 +17944,9 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "openai",
- "max_input_tokens": 272000,
- "max_output_tokens": 128000,
- "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supported_endpoints": [
@@ -17996,7 +18071,7 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "openai",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
@@ -18055,6 +18130,39 @@
"supports_tool_choice": true,
"supports_vision": true
},
+ "gpt-5.2-codex": {
+ "cache_read_input_token_cost": 1.75e-07,
+ "cache_read_input_token_cost_priority": 3.5e-07,
+ "input_cost_per_token": 1.75e-06,
+ "input_cost_per_token_priority": 3.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 400000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "responses",
+ "output_cost_per_token": 1.4e-05,
+ "output_cost_per_token_priority": 2.8e-05,
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": false,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
"gpt-5-mini": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_flex": 1.25e-08,
@@ -23188,9 +23296,9 @@
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "openrouter",
- "max_input_tokens": 272000,
- "max_output_tokens": 128000,
- "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supported_modalities": [
@@ -23222,6 +23330,25 @@
"supports_reasoning": true,
"supports_tool_choice": true
},
+ "openrouter/openai/gpt-5.2-codex": {
+ "cache_read_input_token_cost": 1.75e-07,
+ "input_cost_per_token": 1.75e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 400000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.4e-05,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
"openrouter/openai/gpt-5": {
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
@@ -23284,7 +23411,7 @@
"cache_read_input_token_cost": 1.75e-07,
"input_cost_per_token": 1.75e-06,
"litellm_provider": "openrouter",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -23314,7 +23441,7 @@
"input_cost_per_image": 0,
"input_cost_per_token": 2.1e-05,
"litellm_provider": "openrouter",
- "max_input_tokens": 400000,
+ "max_input_tokens": 272000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
@@ -24434,7 +24561,7 @@
},
"replicate/openai/gpt-5": {
"input_cost_per_token": 1.25e-06,
- "output_cost_per_token": 1.0e-05,
+ "output_cost_per_token": 1e-05,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24445,7 +24572,7 @@
"supports_response_schema": true
},
"replicateopenai/gpt-oss-20b": {
- "input_cost_per_token": 9.0e-08,
+ "input_cost_per_token": 9e-08,
"output_cost_per_token": 3.6e-07,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24453,8 +24580,8 @@
"supports_system_messages": true
},
"replicate/anthropic/claude-4.5-haiku": {
- "input_cost_per_token": 1.0e-06,
- "output_cost_per_token": 5.0e-06,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 5e-06,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24466,7 +24593,7 @@
"supports_prompt_caching": true
},
"replicate/ibm-granite/granite-3.3-8b-instruct": {
- "input_cost_per_token": 3.0e-08,
+ "input_cost_per_token": 3e-08,
"output_cost_per_token": 2.5e-07,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24475,7 +24602,7 @@
},
"replicate/openai/gpt-4o": {
"input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1.0e-05,
+ "output_cost_per_token": 1e-05,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24488,9 +24615,9 @@
"supports_audio_output": true
},
"replicate/openai/o4-mini": {
- "input_cost_per_token": 1.0e-06,
- "output_cost_per_token": 4.0e-06,
- "output_cost_per_reasoning_token": 4.0e-06,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 4e-06,
+ "output_cost_per_reasoning_token": 4e-06,
"litellm_provider": "replicate",
"mode": "chat",
"supports_reasoning": true,
@@ -24507,8 +24634,8 @@
},
"replicate/openai/o1": {
"input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6.0e-05,
- "output_cost_per_reasoning_token": 6.0e-05,
+ "output_cost_per_token": 6e-05,
+ "output_cost_per_reasoning_token": 6e-05,
"litellm_provider": "replicate",
"mode": "chat",
"supports_reasoning": true,
@@ -24516,7 +24643,7 @@
},
"replicate/openai/gpt-4o-mini": {
"input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6.0e-07,
+ "output_cost_per_token": 6e-07,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24535,7 +24662,7 @@
"supports_system_messages": true
},
"replicate/anthropic/claude-4-sonnet": {
- "input_cost_per_token": 3.0e-06,
+ "input_cost_per_token": 3e-06,
"output_cost_per_token": 1.5e-05,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24559,7 +24686,7 @@
"supports_system_messages": true
},
"replicate/anthropic/claude-3.7-sonnet": {
- "input_cost_per_token": 3.0e-06,
+ "input_cost_per_token": 3e-06,
"output_cost_per_token": 1.5e-05,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24572,8 +24699,8 @@
"supports_prompt_caching": true
},
"replicate/anthropic/claude-3.5-haiku": {
- "input_cost_per_token": 1.0e-06,
- "output_cost_per_token": 5.0e-06,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 5e-06,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24598,7 +24725,7 @@
"supports_prompt_caching": true
},
"replicate/google/gemini-3-pro": {
- "input_cost_per_token": 2.0e-06,
+ "input_cost_per_token": 2e-06,
"output_cost_per_token": 1.2e-05,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24610,7 +24737,7 @@
"supports_response_schema": true
},
"replicate/anthropic/claude-4.5-sonnet": {
- "input_cost_per_token": 3.0e-06,
+ "input_cost_per_token": 3e-06,
"output_cost_per_token": 1.5e-05,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24623,8 +24750,8 @@
"supports_prompt_caching": true
},
"replicate/openai/gpt-4.1": {
- "input_cost_per_token": 2.0e-06,
- "output_cost_per_token": 8.0e-06,
+ "input_cost_per_token": 2e-06,
+ "output_cost_per_token": 8e-06,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24635,15 +24762,15 @@
"supports_response_schema": true
},
"replicate/openai/gpt-4.1-nano": {
- "input_cost_per_token": 1.0e-07,
- "output_cost_per_token": 4.0e-07,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 4e-07,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
"supports_system_messages": true
},
"replicate/openai/gpt-4.1-mini": {
- "input_cost_per_token": 4.0e-07,
+ "input_cost_per_token": 4e-07,
"output_cost_per_token": 1.6e-06,
"litellm_provider": "replicate",
"mode": "chat",
@@ -24655,8 +24782,8 @@
"supports_response_schema": true
},
"replicate/openai/gpt-5-nano": {
- "input_cost_per_token": 5.0e-08,
- "output_cost_per_token": 4.0e-07,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 4e-07,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24664,7 +24791,7 @@
},
"replicate/openai/gpt-5-mini": {
"input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.0e-06,
+ "output_cost_per_token": 2e-06,
"litellm_provider": "replicate",
"mode": "chat",
"supports_function_calling": true,
@@ -24716,8 +24843,8 @@
},
"replicate/deepseek-ai/deepseek-r1": {
"input_cost_per_token": 3.75e-06,
- "output_cost_per_token": 1.0e-05,
- "output_cost_per_reasoning_token": 1.0e-05,
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_reasoning_token": 1e-05,
"litellm_provider": "replicate",
"mode": "chat",
"max_input_tokens": 65536,
@@ -25430,13 +25557,13 @@
"litellm_provider": "bedrock",
"max_input_tokens": 77,
"mode": "image_edit",
- "output_cost_per_image": 0.4
+ "output_cost_per_image": 0.40
},
"stability.stable-creative-upscale-v1:0": {
"litellm_provider": "bedrock",
"max_input_tokens": 77,
"mode": "image_edit",
- "output_cost_per_image": 0.6
+ "output_cost_per_image": 0.60
},
"stability.stable-fast-upscale-v1:0": {
"litellm_provider": "bedrock",
@@ -28686,13 +28813,13 @@
"supports_web_search": true
},
"vertex_ai/zai-org/glm-4.7-maas": {
- "input_cost_per_token": 3e-07,
+ "input_cost_per_token": 6e-07,
"litellm_provider": "vertex_ai-zai_models",
"max_input_tokens": 200000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
- "output_cost_per_token": 1.2e-06,
+ "output_cost_per_token": 2.2e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
"supports_function_calling": true,
"supports_reasoning": true,
@@ -32534,8 +32661,8 @@
"novita/deepseek/deepseek-v3.2": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.69e-03,
- "output_cost_per_token": 4e-03,
+ "input_cost_per_token": 2.69e-07,
+ "output_cost_per_token": 4e-07,
"max_input_tokens": 163840,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -32544,15 +32671,15 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 1.345e-03,
- "input_cost_per_token_cache_hit": 1.345e-03,
+ "cache_read_input_token_cost": 1.345e-07,
+ "input_cost_per_token_cache_hit": 1.345e-07,
"supports_reasoning": true
},
"novita/minimax/minimax-m2.1": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3e-03,
- "output_cost_per_token": 1.2e-02,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.2e-06,
"max_input_tokens": 204800,
"max_output_tokens": 131072,
"max_tokens": 131072,
@@ -32561,15 +32688,14 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 3e-04,
- "input_cost_per_token_cache_hit": 3e-04,
- "supports_reasoning": true
+ "cache_read_input_token_cost": 3e-08,
+ "input_cost_per_token_cache_hit": 3e-08
},
"novita/zai-org/glm-4.7": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 6e-03,
- "output_cost_per_token": 2.2e-02,
+ "input_cost_per_token": 6e-07,
+ "output_cost_per_token": 2.2e-06,
"max_input_tokens": 204800,
"max_output_tokens": 131072,
"max_tokens": 131072,
@@ -32578,15 +32704,15 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 1.1e-03,
- "input_cost_per_token_cache_hit": 1.1e-03,
+ "cache_read_input_token_cost": 1.1e-07,
+ "input_cost_per_token_cache_hit": 1.1e-07,
"supports_reasoning": true
},
"novita/xiaomimimo/mimo-v2-flash": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1e-03,
- "output_cost_per_token": 3e-03,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 3e-07,
"max_input_tokens": 262144,
"max_output_tokens": 32000,
"max_tokens": 32000,
@@ -32595,15 +32721,15 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 2e-04,
- "input_cost_per_token_cache_hit": 2e-04,
+ "cache_read_input_token_cost": 2e-08,
+ "input_cost_per_token_cache_hit": 2e-08,
"supports_reasoning": true
},
"novita/zai-org/autoglm-phone-9b-multilingual": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.5e-04,
- "output_cost_per_token": 1.38e-03,
+ "input_cost_per_token": 3.5e-08,
+ "output_cost_per_token": 1.38e-07,
"max_input_tokens": 65536,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -32613,8 +32739,8 @@
"novita/moonshotai/kimi-k2-thinking": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.8e-03,
- "output_cost_per_token": 2e-02,
+ "input_cost_per_token": 6e-07,
+ "output_cost_per_token": 2.5e-06,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"max_tokens": 262144,
@@ -32628,8 +32754,8 @@
"novita/minimax/minimax-m2": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-03,
- "output_cost_per_token": 9.6e-03,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.2e-06,
"max_input_tokens": 204800,
"max_output_tokens": 131072,
"max_tokens": 131072,
@@ -32637,15 +32763,15 @@
"supports_parallel_function_calling": true,
"supports_tool_choice": true,
"supports_system_messages": true,
- "cache_read_input_token_cost": 2.4e-04,
- "input_cost_per_token_cache_hit": 2.4e-04,
+ "cache_read_input_token_cost": 3e-08,
+ "input_cost_per_token_cache_hit": 3e-08,
"supports_reasoning": true
},
"novita/paddlepaddle/paddleocr-vl": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.6e-04,
- "output_cost_per_token": 1.6e-04,
+ "input_cost_per_token": 2e-08,
+ "output_cost_per_token": 2e-08,
"max_input_tokens": 16384,
"max_output_tokens": 16384,
"max_tokens": 16384,
@@ -32655,8 +32781,8 @@
"novita/deepseek/deepseek-v3.2-exp": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.16e-03,
- "output_cost_per_token": 3.28e-03,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 4.1e-07,
"max_input_tokens": 163840,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -32670,8 +32796,8 @@
"novita/qwen/qwen3-vl-235b-a22b-thinking": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 7.84e-03,
- "output_cost_per_token": 3.16e-02,
+ "input_cost_per_token": 9.8e-07,
+ "output_cost_per_token": 3.95e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32682,8 +32808,8 @@
"novita/zai-org/glm-4.6v": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3e-03,
- "output_cost_per_token": 9e-03,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 9e-07,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32693,15 +32819,15 @@
"supports_vision": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 5.5e-04,
- "input_cost_per_token_cache_hit": 5.5e-04,
+ "cache_read_input_token_cost": 5.5e-08,
+ "input_cost_per_token_cache_hit": 5.5e-08,
"supports_reasoning": true
},
"novita/zai-org/glm-4.6": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.4e-03,
- "output_cost_per_token": 1.76e-02,
+ "input_cost_per_token": 5.5e-07,
+ "output_cost_per_token": 2.2e-06,
"max_input_tokens": 204800,
"max_output_tokens": 131072,
"max_tokens": 131072,
@@ -32710,15 +32836,31 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 8.8e-04,
- "input_cost_per_token_cache_hit": 8.8e-04,
+ "cache_read_input_token_cost": 1.1e-07,
+ "input_cost_per_token_cache_hit": 1.1e-07,
"supports_reasoning": true
},
+ "novita/kwaipilot/kat-coder-pro": {
+ "litellm_provider": "novita",
+ "mode": "chat",
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.2e-06,
+ "max_input_tokens": 256000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_system_messages": true,
+ "supports_response_schema": true,
+ "cache_read_input_token_cost": 6e-08,
+ "input_cost_per_token_cache_hit": 6e-08
+ },
"novita/qwen/qwen3-next-80b-a3b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.2e-03,
- "output_cost_per_token": 1.2e-02,
+ "input_cost_per_token": 1.5e-07,
+ "output_cost_per_token": 1.5e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32731,8 +32873,8 @@
"novita/qwen/qwen3-next-80b-a3b-thinking": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.2e-03,
- "output_cost_per_token": 1.2e-02,
+ "input_cost_per_token": 1.5e-07,
+ "output_cost_per_token": 1.5e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32746,8 +32888,8 @@
"novita/deepseek/deepseek-ocr": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-04,
- "output_cost_per_token": 2.4e-04,
+ "input_cost_per_token": 3e-08,
+ "output_cost_per_token": 3e-08,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -32759,8 +32901,8 @@
"novita/deepseek/deepseek-v3.1-terminus": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.16e-03,
- "output_cost_per_token": 8e-03,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 1e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32769,15 +32911,15 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 1.08e-03,
- "input_cost_per_token_cache_hit": 1.08e-03,
+ "cache_read_input_token_cost": 1.35e-07,
+ "input_cost_per_token_cache_hit": 1.35e-07,
"supports_reasoning": true
},
"novita/qwen/qwen3-vl-235b-a22b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-03,
- "output_cost_per_token": 1.2e-02,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.5e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32791,8 +32933,8 @@
"novita/qwen/qwen3-max": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.688e-02,
- "output_cost_per_token": 6.76e-02,
+ "input_cost_per_token": 2.11e-06,
+ "output_cost_per_token": 8.45e-06,
"max_input_tokens": 262144,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -32805,8 +32947,8 @@
"novita/skywork/r1v4-lite": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2e-03,
- "output_cost_per_token": 6e-03,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 6e-07,
"max_input_tokens": 262144,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -32818,8 +32960,8 @@
"novita/deepseek/deepseek-v3.1": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.16e-03,
- "output_cost_per_token": 8e-03,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 1e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32828,15 +32970,15 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 1.08e-03,
- "input_cost_per_token_cache_hit": 1.08e-03,
+ "cache_read_input_token_cost": 1.35e-07,
+ "input_cost_per_token_cache_hit": 1.35e-07,
"supports_reasoning": true
},
"novita/moonshotai/kimi-k2-0905": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.8e-03,
- "output_cost_per_token": 2e-02,
+ "input_cost_per_token": 6e-07,
+ "output_cost_per_token": 2.5e-06,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"max_tokens": 262144,
@@ -32849,8 +32991,8 @@
"novita/qwen/qwen3-coder-480b-a35b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-03,
- "output_cost_per_token": 1.04e-02,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.3e-06,
"max_input_tokens": 262144,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -32863,8 +33005,8 @@
"novita/qwen/qwen3-coder-30b-a3b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 7e-04,
- "output_cost_per_token": 2.7e-03,
+ "input_cost_per_token": 7e-08,
+ "output_cost_per_token": 2.7e-07,
"max_input_tokens": 160000,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32877,8 +33019,8 @@
"novita/openai/gpt-oss-120b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4e-04,
- "output_cost_per_token": 2e-03,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 2.5e-07,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32893,8 +33035,8 @@
"novita/moonshotai/kimi-k2-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.56e-03,
- "output_cost_per_token": 1.84e-02,
+ "input_cost_per_token": 5.7e-07,
+ "output_cost_per_token": 2.3e-06,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
@@ -32907,8 +33049,8 @@
"novita/deepseek/deepseek-v3-0324": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.16e-03,
- "output_cost_per_token": 8.96e-03,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 1.12e-06,
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 163840,
@@ -32917,14 +33059,14 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 1.08e-03,
- "input_cost_per_token_cache_hit": 1.08e-03
+ "cache_read_input_token_cost": 1.35e-07,
+ "input_cost_per_token_cache_hit": 1.35e-07
},
"novita/zai-org/glm-4.5": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.8e-03,
- "output_cost_per_token": 1.76e-02,
+ "input_cost_per_token": 6e-07,
+ "output_cost_per_token": 2.2e-06,
"max_input_tokens": 131072,
"max_output_tokens": 98304,
"max_tokens": 98304,
@@ -32932,15 +33074,15 @@
"supports_parallel_function_calling": true,
"supports_tool_choice": true,
"supports_system_messages": true,
- "cache_read_input_token_cost": 8.8e-04,
- "input_cost_per_token_cache_hit": 8.8e-04,
+ "cache_read_input_token_cost": 1.1e-07,
+ "input_cost_per_token_cache_hit": 1.1e-07,
"supports_reasoning": true
},
"novita/qwen/qwen3-235b-a22b-thinking-2507": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-03,
- "output_cost_per_token": 2.4e-02,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 3e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -32953,8 +33095,8 @@
"novita/meta-llama/llama-3.1-8b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2e-04,
- "output_cost_per_token": 5e-04,
+ "input_cost_per_token": 2e-08,
+ "output_cost_per_token": 5e-08,
"max_input_tokens": 16384,
"max_output_tokens": 16384,
"max_tokens": 16384,
@@ -32963,8 +33105,8 @@
"novita/google/gemma-3-12b-it": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4e-04,
- "output_cost_per_token": 8e-04,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 1e-07,
"max_input_tokens": 131072,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -32976,8 +33118,8 @@
"novita/zai-org/glm-4.5v": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.8e-03,
- "output_cost_per_token": 1.44e-02,
+ "input_cost_per_token": 6e-07,
+ "output_cost_per_token": 1.8e-06,
"max_input_tokens": 65536,
"max_output_tokens": 16384,
"max_tokens": 16384,
@@ -32987,15 +33129,15 @@
"supports_vision": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 8.8e-04,
- "input_cost_per_token_cache_hit": 8.8e-04,
+ "cache_read_input_token_cost": 1.1e-07,
+ "input_cost_per_token_cache_hit": 1.1e-07,
"supports_reasoning": true
},
"novita/openai/gpt-oss-20b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.2e-04,
- "output_cost_per_token": 1.2e-03,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 1.5e-07,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -33008,8 +33150,8 @@
"novita/qwen/qwen3-235b-a22b-instruct-2507": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 7.2e-04,
- "output_cost_per_token": 4.64e-03,
+ "input_cost_per_token": 9e-08,
+ "output_cost_per_token": 5.8e-07,
"max_input_tokens": 131072,
"max_output_tokens": 16384,
"max_tokens": 16384,
@@ -33022,8 +33164,8 @@
"novita/deepseek/deepseek-r1-distill-qwen-14b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.2e-03,
- "output_cost_per_token": 1.2e-03,
+ "input_cost_per_token": 1.5e-07,
+ "output_cost_per_token": 1.5e-07,
"max_input_tokens": 32768,
"max_output_tokens": 16384,
"max_tokens": 16384,
@@ -33035,8 +33177,8 @@
"novita/meta-llama/llama-3.3-70b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.08e-03,
- "output_cost_per_token": 3.2e-03,
+ "input_cost_per_token": 1.35e-07,
+ "output_cost_per_token": 4e-07,
"max_input_tokens": 131072,
"max_output_tokens": 120000,
"max_tokens": 120000,
@@ -33048,8 +33190,8 @@
"novita/qwen/qwen-2.5-72b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.04e-03,
- "output_cost_per_token": 3.2e-03,
+ "input_cost_per_token": 3.8e-07,
+ "output_cost_per_token": 4e-07,
"max_input_tokens": 32000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33062,8 +33204,8 @@
"novita/mistralai/mistral-nemo": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.2e-04,
- "output_cost_per_token": 1.36e-03,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 1.7e-07,
"max_input_tokens": 60288,
"max_output_tokens": 16000,
"max_tokens": 16000,
@@ -33074,8 +33216,8 @@
"novita/minimaxai/minimax-m1-80k": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.4e-03,
- "output_cost_per_token": 1.76e-02,
+ "input_cost_per_token": 5.5e-07,
+ "output_cost_per_token": 2.2e-06,
"max_input_tokens": 1000000,
"max_output_tokens": 40000,
"max_tokens": 40000,
@@ -33088,8 +33230,8 @@
"novita/deepseek/deepseek-r1-0528": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-03,
- "output_cost_per_token": 2e-02,
+ "input_cost_per_token": 7e-07,
+ "output_cost_per_token": 2.5e-06,
"max_input_tokens": 163840,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -33098,15 +33240,15 @@
"supports_tool_choice": true,
"supports_system_messages": true,
"supports_response_schema": true,
- "cache_read_input_token_cost": 2.8e-03,
- "input_cost_per_token_cache_hit": 2.8e-03,
+ "cache_read_input_token_cost": 3.5e-07,
+ "input_cost_per_token_cache_hit": 3.5e-07,
"supports_reasoning": true
},
"novita/deepseek/deepseek-r1-distill-qwen-32b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-03,
- "output_cost_per_token": 2.4e-03,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 3e-07,
"max_input_tokens": 64000,
"max_output_tokens": 32000,
"max_tokens": 32000,
@@ -33118,8 +33260,8 @@
"novita/meta-llama/llama-3-8b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.2e-04,
- "output_cost_per_token": 3.2e-04,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 4e-08,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33128,8 +33270,8 @@
"novita/microsoft/wizardlm-2-8x22b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.96e-03,
- "output_cost_per_token": 4.96e-03,
+ "input_cost_per_token": 6.2e-07,
+ "output_cost_per_token": 6.2e-07,
"max_input_tokens": 65535,
"max_output_tokens": 8000,
"max_tokens": 8000,
@@ -33138,8 +33280,8 @@
"novita/deepseek/deepseek-r1-0528-qwen3-8b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 4.8e-04,
- "output_cost_per_token": 7.2e-04,
+ "input_cost_per_token": 6e-08,
+ "output_cost_per_token": 9e-08,
"max_input_tokens": 128000,
"max_output_tokens": 32000,
"max_tokens": 32000,
@@ -33149,8 +33291,8 @@
"novita/deepseek/deepseek-r1-distill-llama-70b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 6.4e-03,
- "output_cost_per_token": 6.4e-03,
+ "input_cost_per_token": 8e-07,
+ "output_cost_per_token": 8e-07,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33162,8 +33304,8 @@
"novita/meta-llama/llama-3-70b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.1e-03,
- "output_cost_per_token": 7.4e-03,
+ "input_cost_per_token": 5.1e-07,
+ "output_cost_per_token": 7.4e-07,
"max_input_tokens": 8192,
"max_output_tokens": 8000,
"max_tokens": 8000,
@@ -33174,8 +33316,8 @@
"novita/qwen/qwen3-235b-a22b-fp8": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.6e-03,
- "output_cost_per_token": 6.4e-03,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 8e-07,
"max_input_tokens": 40960,
"max_output_tokens": 20000,
"max_tokens": 20000,
@@ -33185,8 +33327,8 @@
"novita/meta-llama/llama-4-maverick-17b-128e-instruct-fp8": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.6e-03,
- "output_cost_per_token": 7.2e-03,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 8.5e-07,
"max_input_tokens": 1048576,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33196,8 +33338,8 @@
"novita/meta-llama/llama-4-scout-17b-16e-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 8e-04,
- "output_cost_per_token": 4e-03,
+ "input_cost_per_token": 1.8e-07,
+ "output_cost_per_token": 5.9e-07,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
@@ -33207,8 +33349,8 @@
"novita/nousresearch/hermes-2-pro-llama-3-8b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.4e-03,
- "output_cost_per_token": 1.4e-03,
+ "input_cost_per_token": 1.4e-07,
+ "output_cost_per_token": 1.4e-07,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33219,8 +33361,8 @@
"novita/qwen/qwen2.5-vl-72b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 6.4e-03,
- "output_cost_per_token": 6.4e-03,
+ "input_cost_per_token": 8e-07,
+ "output_cost_per_token": 8e-07,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -33230,8 +33372,8 @@
"novita/sao10k/l3-70b-euryale-v2.1": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.48e-02,
- "output_cost_per_token": 1.48e-02,
+ "input_cost_per_token": 1.48e-06,
+ "output_cost_per_token": 1.48e-06,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33243,8 +33385,8 @@
"novita/baidu/ernie-4.5-21B-a3b-thinking": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-04,
- "output_cost_per_token": 2.24e-03,
+ "input_cost_per_token": 7e-08,
+ "output_cost_per_token": 2.8e-07,
"max_input_tokens": 131072,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -33254,8 +33396,8 @@
"novita/sao10k/l3-8b-lunaris": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5e-04,
- "output_cost_per_token": 5e-04,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 5e-08,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33266,33 +33408,18 @@
"novita/baichuan/baichuan-m2-32b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-04,
- "output_cost_per_token": 5.6e-04,
+ "input_cost_per_token": 7e-08,
+ "output_cost_per_token": 7e-08,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"max_tokens": 131072,
- "supports_tool_choice": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_reasoning": true
- },
- "novita/thudm/glm-4.1v-9b-thinking": {
- "litellm_provider": "novita",
- "mode": "chat",
- "input_cost_per_token": 2.8e-04,
- "output_cost_per_token": 1.104e-03,
- "max_input_tokens": 65536,
- "max_output_tokens": 8000,
- "max_tokens": 8000,
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_reasoning": true
+ "supports_system_messages": true
},
"novita/baidu/ernie-4.5-vl-424b-a47b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.36e-03,
- "output_cost_per_token": 1e-02,
+ "input_cost_per_token": 4.2e-07,
+ "output_cost_per_token": 1.25e-06,
"max_input_tokens": 123000,
"max_output_tokens": 16000,
"max_tokens": 16000,
@@ -33303,8 +33430,8 @@
"novita/baidu/ernie-4.5-300b-a47b-paddle": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.24e-03,
- "output_cost_per_token": 8.8e-03,
+ "input_cost_per_token": 2.8e-07,
+ "output_cost_per_token": 1.1e-06,
"max_input_tokens": 123000,
"max_output_tokens": 12000,
"max_tokens": 12000,
@@ -33315,8 +33442,8 @@
"novita/deepseek/deepseek-prover-v2-671b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-03,
- "output_cost_per_token": 2e-02,
+ "input_cost_per_token": 7e-07,
+ "output_cost_per_token": 2.5e-06,
"max_input_tokens": 160000,
"max_output_tokens": 160000,
"max_tokens": 160000,
@@ -33325,8 +33452,8 @@
"novita/qwen/qwen3-32b-fp8": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 8e-04,
- "output_cost_per_token": 3.6e-03,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 4.5e-07,
"max_input_tokens": 40960,
"max_output_tokens": 20000,
"max_tokens": 20000,
@@ -33336,8 +33463,8 @@
"novita/qwen/qwen3-30b-a3b-fp8": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 7.2e-04,
- "output_cost_per_token": 3.6e-03,
+ "input_cost_per_token": 9e-08,
+ "output_cost_per_token": 4.5e-07,
"max_input_tokens": 40960,
"max_output_tokens": 20000,
"max_tokens": 20000,
@@ -33347,8 +33474,8 @@
"novita/google/gemma-3-27b-it": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 9.52e-04,
- "output_cost_per_token": 1.6e-03,
+ "input_cost_per_token": 1.19e-07,
+ "output_cost_per_token": 2e-07,
"max_input_tokens": 98304,
"max_output_tokens": 16384,
"max_tokens": 16384,
@@ -33358,8 +33485,8 @@
"novita/deepseek/deepseek-v3-turbo": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.2e-03,
- "output_cost_per_token": 1.04e-02,
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_token": 1.3e-06,
"max_input_tokens": 64000,
"max_output_tokens": 16000,
"max_tokens": 16000,
@@ -33371,8 +33498,8 @@
"novita/deepseek/deepseek-r1-turbo": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-03,
- "output_cost_per_token": 2e-02,
+ "input_cost_per_token": 7e-07,
+ "output_cost_per_token": 2.5e-06,
"max_input_tokens": 64000,
"max_output_tokens": 16000,
"max_tokens": 16000,
@@ -33385,8 +33512,8 @@
"novita/Sao10K/L3-8B-Stheno-v3.2": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5e-04,
- "output_cost_per_token": 5e-04,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 5e-08,
"max_input_tokens": 8192,
"max_output_tokens": 32000,
"max_tokens": 32000,
@@ -33398,8 +33525,8 @@
"novita/gryphe/mythomax-l2-13b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 7.2e-04,
- "output_cost_per_token": 7.2e-04,
+ "input_cost_per_token": 9e-08,
+ "output_cost_per_token": 9e-08,
"max_input_tokens": 4096,
"max_output_tokens": 3200,
"max_tokens": 3200,
@@ -33408,8 +33535,8 @@
"novita/baidu/ernie-4.5-vl-28b-a3b-thinking": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 3.9e-03,
- "output_cost_per_token": 3.9e-03,
+ "input_cost_per_token": 3.9e-07,
+ "output_cost_per_token": 3.9e-07,
"max_input_tokens": 131072,
"max_output_tokens": 65536,
"max_tokens": 65536,
@@ -33424,8 +33551,8 @@
"novita/qwen/qwen3-vl-8b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 6.4e-04,
- "output_cost_per_token": 4e-03,
+ "input_cost_per_token": 8e-08,
+ "output_cost_per_token": 5e-07,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -33439,8 +33566,8 @@
"novita/zai-org/glm-4.5-air": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.04e-03,
- "output_cost_per_token": 6.8e-03,
+ "input_cost_per_token": 1.3e-07,
+ "output_cost_per_token": 8.5e-07,
"max_input_tokens": 131072,
"max_output_tokens": 98304,
"max_tokens": 98304,
@@ -33453,8 +33580,8 @@
"novita/qwen/qwen3-vl-30b-a3b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.6e-03,
- "output_cost_per_token": 5.6e-03,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 7e-07,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -33468,8 +33595,8 @@
"novita/qwen/qwen3-vl-30b-a3b-thinking": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.6e-03,
- "output_cost_per_token": 8e-03,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 1e-06,
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
@@ -33480,11 +33607,45 @@
"supports_system_messages": true,
"supports_response_schema": true
},
+ "novita/qwen/qwen3-omni-30b-a3b-thinking": {
+ "litellm_provider": "novita",
+ "mode": "chat",
+ "input_cost_per_token": 2.5e-07,
+ "output_cost_per_token": 9.7e-07,
+ "max_input_tokens": 65536,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_system_messages": true,
+ "supports_response_schema": true,
+ "supports_reasoning": true,
+ "supports_audio_input": true
+ },
+ "novita/qwen/qwen3-omni-30b-a3b-instruct": {
+ "litellm_provider": "novita",
+ "mode": "chat",
+ "input_cost_per_token": 2.5e-07,
+ "output_cost_per_token": 9.7e-07,
+ "max_input_tokens": 65536,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_system_messages": true,
+ "supports_response_schema": true,
+ "supports_audio_input": true,
+ "supports_audio_output": true
+ },
"novita/qwen/qwen-mt-plus": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2e-03,
- "output_cost_per_token": 6e-03,
+ "input_cost_per_token": 2.5e-07,
+ "output_cost_per_token": 7.5e-07,
"max_input_tokens": 16384,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33493,8 +33654,8 @@
"novita/baidu/ernie-4.5-vl-28b-a3b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.12e-03,
- "output_cost_per_token": 4.48e-03,
+ "input_cost_per_token": 1.4e-07,
+ "output_cost_per_token": 5.6e-07,
"max_input_tokens": 30000,
"max_output_tokens": 8000,
"max_tokens": 8000,
@@ -33508,8 +33669,8 @@
"novita/baidu/ernie-4.5-21B-a3b": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-04,
- "output_cost_per_token": 2.24e-03,
+ "input_cost_per_token": 7e-08,
+ "output_cost_per_token": 2.8e-07,
"max_input_tokens": 120000,
"max_output_tokens": 8000,
"max_tokens": 8000,
@@ -33521,8 +33682,8 @@
"novita/qwen/qwen3-8b-fp8": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.8e-04,
- "output_cost_per_token": 1.104e-03,
+ "input_cost_per_token": 3.5e-08,
+ "output_cost_per_token": 1.38e-07,
"max_input_tokens": 128000,
"max_output_tokens": 20000,
"max_tokens": 20000,
@@ -33532,8 +33693,8 @@
"novita/qwen/qwen3-4b-fp8": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-04,
- "output_cost_per_token": 2.4e-04,
+ "input_cost_per_token": 3e-08,
+ "output_cost_per_token": 3e-08,
"max_input_tokens": 128000,
"max_output_tokens": 20000,
"max_tokens": 20000,
@@ -33543,8 +33704,8 @@
"novita/qwen/qwen2.5-7b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 5.6e-04,
- "output_cost_per_token": 5.6e-04,
+ "input_cost_per_token": 7e-08,
+ "output_cost_per_token": 7e-08,
"max_input_tokens": 32000,
"max_output_tokens": 32000,
"max_tokens": 32000,
@@ -33557,8 +33718,8 @@
"novita/meta-llama/llama-3.2-3b-instruct": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 2.4e-04,
- "output_cost_per_token": 4e-04,
+ "input_cost_per_token": 3e-08,
+ "output_cost_per_token": 5e-08,
"max_input_tokens": 32768,
"max_output_tokens": 32000,
"max_tokens": 32000,
@@ -33570,8 +33731,8 @@
"novita/sao10k/l31-70b-euryale-v2.2": {
"litellm_provider": "novita",
"mode": "chat",
- "input_cost_per_token": 1.48e-02,
- "output_cost_per_token": 1.48e-02,
+ "input_cost_per_token": 1.48e-06,
+ "output_cost_per_token": 1.48e-06,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"max_tokens": 8192,
@@ -33583,7 +33744,7 @@
"novita/qwen/qwen3-embedding-0.6b": {
"litellm_provider": "novita",
"mode": "embedding",
- "input_cost_per_token": 5.6e-04,
+ "input_cost_per_token": 7e-08,
"output_cost_per_token": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
@@ -33592,7 +33753,7 @@
"novita/qwen/qwen3-embedding-8b": {
"litellm_provider": "novita",
"mode": "embedding",
- "input_cost_per_token": 5.6e-04,
+ "input_cost_per_token": 7e-08,
"output_cost_per_token": 0,
"max_input_tokens": 32768,
"max_output_tokens": 4096,
@@ -33601,8 +33762,8 @@
"novita/baai/bge-m3": {
"litellm_provider": "novita",
"mode": "embedding",
- "input_cost_per_token": 1e-04,
- "output_cost_per_token": 1e-04,
+ "input_cost_per_token": 1e-08,
+ "output_cost_per_token": 1e-08,
"max_input_tokens": 8192,
"max_output_tokens": 96000,
"max_tokens": 96000
@@ -33610,8 +33771,8 @@
"novita/qwen/qwen3-reranker-8b": {
"litellm_provider": "novita",
"mode": "rerank",
- "input_cost_per_token": 4e-04,
- "output_cost_per_token": 4e-04,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 5e-08,
"max_input_tokens": 32768,
"max_output_tokens": 4096,
"max_tokens": 4096
@@ -33619,8 +33780,8 @@
"novita/baai/bge-reranker-v2-m3": {
"litellm_provider": "novita",
"mode": "rerank",
- "input_cost_per_token": 1e-04,
- "output_cost_per_token": 1e-04,
+ "input_cost_per_token": 1e-08,
+ "output_cost_per_token": 1e-08,
"max_input_tokens": 8000,
"max_output_tokens": 8000,
"max_tokens": 8000
diff --git a/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py b/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
index b43f4217177..49d6ac7d898 100644
--- a/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
+++ b/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
@@ -516,7 +516,7 @@ class MCPRequestHandler:
Check if the tool is allowed for the given user/key based on permissions
"""
if len(allowed_mcp_servers) == 0:
- return True
+ return False
elif server_name in allowed_mcp_servers:
return True
return False
diff --git a/litellm/proxy/_experimental/mcp_server/rest_endpoints.py b/litellm/proxy/_experimental/mcp_server/rest_endpoints.py
index 642cb0cec2d..48f7a8b0b7b 100644
--- a/litellm/proxy/_experimental/mcp_server/rest_endpoints.py
+++ b/litellm/proxy/_experimental/mcp_server/rest_endpoints.py
@@ -1,9 +1,13 @@
import importlib
+from datetime import datetime
from typing import Dict, List, Optional, Union
-from fastapi import APIRouter, Depends, Query, Request
+from fastapi import APIRouter, Depends, HTTPException, Query, Request
from litellm._logging import verbose_logger
+from litellm.proxy._experimental.mcp_server.ui_session_utils import (
+ build_effective_auth_contexts,
+)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.types.mcp import MCPAuth
@@ -22,13 +26,14 @@ router = APIRouter(
)
if MCP_AVAILABLE:
- from litellm.experimental_mcp_client.client import MCPTool
+ from mcp.types import Tool as MCPTool
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
from litellm.proxy._experimental.mcp_server.server import (
ListMCPToolsRestAPIResponseObject,
- call_mcp_tool,
+ MCPServer,
+ execute_mcp_tool,
filter_tools_by_allowed_tools,
)
@@ -134,11 +139,30 @@ if MCP_AVAILABLE:
MCPRequestHandler._get_mcp_server_auth_headers_from_headers(headers)
)
+ auth_contexts = await build_effective_auth_contexts(user_api_key_dict)
+
+ allowed_server_ids_set = set()
+ for auth_context in auth_contexts:
+ servers = await global_mcp_server_manager.get_allowed_mcp_servers(
+ user_api_key_auth=auth_context
+ )
+ allowed_server_ids_set.update(servers)
+
+ allowed_server_ids = list(allowed_server_ids_set)
+
list_tools_result = []
error_message = None
# If server_id is specified, only query that specific server
if server_id:
+ if server_id not in allowed_server_ids_set:
+ raise HTTPException(
+ status_code=403,
+ detail={
+ "error": "access_denied",
+ "message": f"The key is not allowed to access server {server_id}",
+ },
+ )
server = global_mcp_server_manager.get_mcp_server_by_id(server_id)
if server is None:
return {
@@ -165,9 +189,24 @@ if MCP_AVAILABLE:
"message": f"Failed to get tools from server {server.name}: {str(e)}",
}
else:
- # Query all servers
+ if not allowed_server_ids:
+ raise HTTPException(
+ status_code=403,
+ detail={
+ "error": "access_denied",
+ "message": "The key is not allowed to access any MCP servers.",
+ },
+ )
+
+ # Query all servers the user has access to
errors = []
- for server in global_mcp_server_manager.get_registry().values():
+ for allowed_server_id in allowed_server_ids:
+ server = global_mcp_server_manager.get_mcp_server_by_id(
+ allowed_server_id
+ )
+ if server is None:
+ continue
+
server_auth_header = _get_server_auth_header(
server, mcp_server_auth_headers, mcp_auth_header
)
@@ -225,6 +264,30 @@ if MCP_AVAILABLE:
try:
data = await request.json()
+
+ # Validate required parameters early
+ server_id = data.get("server_id")
+ if not server_id:
+ raise HTTPException(
+ status_code=400,
+ detail={
+ "error": "missing_parameter",
+ "message": "server_id is required in request body",
+ },
+ )
+
+ tool_name = data.get("name")
+ if not tool_name:
+ raise HTTPException(
+ status_code=400,
+ detail={
+ "error": "missing_parameter",
+ "message": "name is required in request body",
+ },
+ )
+
+ tool_arguments = data.get("arguments")
+
data = await add_litellm_data_to_request(
data=data,
request=request,
@@ -252,13 +315,55 @@ if MCP_AVAILABLE:
if mcp_server_auth_headers:
data["mcp_server_auth_headers"] = mcp_server_auth_headers
data["raw_headers"] = raw_headers_from_request
-
+
# Extract user_api_key_auth from metadata and add to top level
# call_mcp_tool expects user_api_key_auth as a top-level parameter
if "metadata" in data and "user_api_key_auth" in data["metadata"]:
data["user_api_key_auth"] = data["metadata"]["user_api_key_auth"]
-
- result = await call_mcp_tool(**data)
+
+ # Get all auth contexts
+ auth_contexts = await build_effective_auth_contexts(user_api_key_dict)
+
+ # Collect allowed server IDs from all contexts
+ allowed_server_ids_set = set()
+ for auth_context in auth_contexts:
+ servers = await global_mcp_server_manager.get_allowed_mcp_servers(
+ user_api_key_auth=auth_context
+ )
+ allowed_server_ids_set.update(servers)
+
+ # Check if the specified server_id is allowed
+ if server_id not in allowed_server_ids_set:
+ raise HTTPException(
+ status_code=403,
+ detail={
+ "error": "access_denied",
+ "message": f"The key is not allowed to access server {server_id}",
+ },
+ )
+
+ # Build allowed_mcp_servers list (only include allowed servers)
+ allowed_mcp_servers: List[MCPServer] = []
+ for allowed_server_id in allowed_server_ids_set:
+ server = global_mcp_server_manager.get_mcp_server_by_id(
+ allowed_server_id
+ )
+ if server is not None:
+ allowed_mcp_servers.append(server)
+
+ # Call execute_mcp_tool directly (permission checks already done)
+ result = await execute_mcp_tool(
+ name=tool_name,
+ arguments=tool_arguments,
+ allowed_mcp_servers=allowed_mcp_servers,
+ start_time=datetime.now(),
+ user_api_key_auth=data.get("user_api_key_auth"),
+ mcp_auth_header=data.get("mcp_auth_header"),
+ mcp_server_auth_headers=data.get("mcp_server_auth_headers"),
+ oauth2_headers=data.get("oauth2_headers"),
+ raw_headers=data.get("raw_headers"),
+ litellm_logging_obj=data.get("litellm_logging_obj"),
+ )
return result
except BlockedPiiEntityError as e:
verbose_logger.error(f"BlockedPiiEntityError in MCP tool call: {str(e)}")
@@ -301,7 +406,6 @@ if MCP_AVAILABLE:
# /health/tools/list -> List tools from MCP server
# For these routes users will dynamically pass the MCP connection params, they don't need to be on the MCP registry
########################################################
- from litellm.proxy._experimental.mcp_server.server import MCPServer
from litellm.proxy.management_endpoints.mcp_management_endpoints import (
NewMCPServerRequest,
)
diff --git a/litellm/proxy/_experimental/mcp_server/server.py b/litellm/proxy/_experimental/mcp_server/server.py
index 2adfe97c611..f22040a7dd9 100644
--- a/litellm/proxy/_experimental/mcp_server/server.py
+++ b/litellm/proxy/_experimental/mcp_server/server.py
@@ -1200,47 +1200,38 @@ if MCP_AVAILABLE:
return managed_resource_templates
- @client
- async def call_mcp_tool(
+ async def execute_mcp_tool(
name: str,
- arguments: Optional[Dict[str, Any]] = None,
+ arguments: Dict[str, Any],
+ allowed_mcp_servers: List[MCPServer],
+ start_time: datetime,
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
mcp_auth_header: Optional[str] = None,
- mcp_servers: Optional[List[str]] = None,
mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
oauth2_headers: Optional[Dict[str, str]] = None,
raw_headers: Optional[Dict[str, str]] = None,
**kwargs: Any,
) -> CallToolResult:
"""
- Call a specific tool with the provided arguments (handles prefixed tool names)
+ Execute MCP tool.
+
+ This function assumes permission checks have already been performed.
+
+ Args:
+ name: Tool name (may include server prefix)
+ arguments: Tool arguments
+ allowed_mcp_servers: Pre-validated list of servers the user can access
+ start_time: Start time for logging
+ user_api_key_auth: Optional user API key auth for logging
+ mcp_auth_header: Optional MCP auth header
+ mcp_server_auth_headers: Optional server-specific auth headers
+ oauth2_headers: Optional OAuth2 headers
+ raw_headers: Optional raw HTTP headers
+ **kwargs: Additional arguments (e.g., litellm_logging_obj)
+
+ Returns:
+ CallToolResult: Tool execution result
"""
- start_time = datetime.now()
- if arguments is None:
- raise HTTPException(
- status_code=400, detail="Request arguments are required"
- )
-
- ## CHECK IF USER IS ALLOWED TO CALL THIS TOOL
- allowed_mcp_server_ids = (
- await global_mcp_server_manager.get_allowed_mcp_servers(
- user_api_key_auth=user_api_key_auth,
- )
- )
-
- allowed_mcp_servers: List[MCPServer] = []
- for allowed_mcp_server_id in allowed_mcp_server_ids:
- allowed_server = global_mcp_server_manager.get_mcp_server_by_id(
- allowed_mcp_server_id
- )
- if allowed_server is not None:
- allowed_mcp_servers.append(allowed_server)
-
- allowed_mcp_servers = await _get_allowed_mcp_servers_from_mcp_server_names(
- mcp_servers=mcp_servers,
- allowed_mcp_servers=allowed_mcp_servers,
- )
-
# Track resolved MCP server for both permission checks and dispatch
mcp_server: Optional[MCPServer] = None
@@ -1359,6 +1350,66 @@ if MCP_AVAILABLE:
)
return response
+ @client
+ async def call_mcp_tool(
+ name: str,
+ arguments: Optional[Dict[str, Any]] = None,
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ mcp_auth_header: Optional[str] = None,
+ mcp_servers: Optional[List[str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
+ **kwargs: Any,
+ ) -> CallToolResult:
+ """
+ Call a specific tool with the provided arguments (handles prefixed tool names).
+ """
+ start_time = datetime.now()
+ if arguments is None:
+ raise HTTPException(
+ status_code=400, detail="Request arguments are required"
+ )
+
+ ## CHECK IF USER IS ALLOWED TO CALL THIS TOOL
+ allowed_mcp_server_ids = (
+ await global_mcp_server_manager.get_allowed_mcp_servers(
+ user_api_key_auth=user_api_key_auth,
+ )
+ )
+
+ allowed_mcp_servers: List[MCPServer] = []
+ for allowed_mcp_server_id in allowed_mcp_server_ids:
+ allowed_server = global_mcp_server_manager.get_mcp_server_by_id(
+ allowed_mcp_server_id
+ )
+ if allowed_server is not None:
+ allowed_mcp_servers.append(allowed_server)
+
+ allowed_mcp_servers = await _get_allowed_mcp_servers_from_mcp_server_names(
+ mcp_servers=mcp_servers,
+ allowed_mcp_servers=allowed_mcp_servers,
+ )
+ if not allowed_mcp_servers:
+ raise HTTPException(
+ status_code=403,
+ detail="User not allowed to call this tool.",
+ )
+
+ # Delegate to execute_mcp_tool for execution
+ return await execute_mcp_tool(
+ name=name,
+ arguments=arguments,
+ allowed_mcp_servers=allowed_mcp_servers,
+ start_time=start_time,
+ user_api_key_auth=user_api_key_auth,
+ mcp_auth_header=mcp_auth_header,
+ mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
+ **kwargs,
+ )
+
async def mcp_get_prompt(
name: str,
arguments: Optional[Dict[str, Any]] = None,
diff --git a/litellm/proxy/_experimental/out/404.html b/litellm/proxy/_experimental/out/404.html
new file mode 100644
index 00000000000..c6035eb40ca
--- /dev/null
+++ b/litellm/proxy/_experimental/out/404.html
@@ -0,0 +1 @@
+404: This page could not be found.LiteLLM Dashboard404
This page could not be found.