scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-09 22:31:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 87

Merge branch 'BerriAI:main' into kowyo/fix-ollama-think

Browse source
This commit is contained in:
Kowyo 2025-10-11 11:36:39 +08:00 • committed by GitHub
commit 753812e1cc
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
762 changed files with 44044 additions and 24254 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

0
MCP_SSL_CHANGES_SUMMARY.md Normal file
View file

2
docker-compose.yml
View file

@ -8,7 +8,7 @@ services:
#########################################
## Uncomment these lines to start proxy with a config.yaml file ##
# volumes:
# - ./config.yaml:/app/config.yaml <<- this is missing in the docker-compose file currently
# - ./config.yaml:/app/config.yaml
# command:
# - "--config=/app/config.yaml"
##############################################

721
docs/my-website/docs/mcp.md
View file

@ -2,7 +2,7 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# /mcp - Model Context Protocol
# MCP Overview
LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint for all MCP tools and control MCP access by Key, Team.
@ -23,6 +23,43 @@ LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint fo
## Adding your MCP
### Prerequisites
To store MCP servers in the database, you need to enable database storage:
**Environment Variable:**
```bash
export STORE_MODEL_IN_DB=True
```
**OR in config.yaml:**
```yaml
general_settings:
store_model_in_db: true
```
#### Fine-grained Database Storage Control
By default, when `store_model_in_db` is `true`, all object types (models, MCPs, guardrails, vector stores, etc.) are stored in the database. If you want to store only specific object types, use the `supported_db_objects` setting.
**Example: Store only MCP servers in the database**
```yaml title="config.yaml" showLineNumbers
general_settings:
store_model_in_db: true
supported_db_objects: ["mcp"] # Only store MCP servers in DB
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
```
**See all available object types:** [Config Settings - supported_db_objects](./proxy/config_settings.md#general_settings---reference)
If `supported_db_objects` is not set, all object types are loaded from the database (default behavior).
<Tabs>
<TabItem value="ui" label="LiteLLM UI">
@ -209,8 +246,203 @@ litellm_settings:
</TabItem>
</Tabs>
## MCP Tool Filtering
## Converting OpenAPI Specs to MCP Servers
LiteLLM can automatically convert OpenAPI specifications into MCP servers, allowing you to expose any REST API as MCP tools. This is useful when you have existing APIs with OpenAPI/Swagger documentation and want to make them available as MCP tools.
### Benefits
- **Rapid Integration**: Convert existing APIs to MCP tools without writing custom MCP server code
- **Automatic Tool Generation**: LiteLLM automatically generates MCP tools from your OpenAPI spec
- **Unified Interface**: Use the same MCP interface for both native MCP servers and OpenAPI-based APIs
- **Easy Testing**: Test and iterate on API integrations quickly
### Configuration
Add your OpenAPI-based MCP server to your `config.yaml`:
```yaml title="config.yaml - OpenAPI to MCP" showLineNumbers
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
mcp_servers:
# OpenAPI Spec Example - Petstore API
petstore_mcp:
url: "https://petstore.swagger.io/v2"
spec_path: "/path/to/openapi.json"
auth_type: "none"
# OpenAPI Spec with API Key Authentication
my_api_mcp:
url: "http://0.0.0.0:8090"
spec_path: "/path/to/openapi.json"
auth_type: "api_key"
auth_value: "your-api-key-here"
# OpenAPI Spec with Bearer Token
secured_api_mcp:
url: "https://api.example.com"
spec_path: "/path/to/openapi.json"
auth_type: "bearer_token"
auth_value: "your-bearer-token"
```
### Configuration Parameters
| Parameter | Required | Description |
|-----------|----------|-------------|
| `url` | Yes | The base URL of your API endpoint |
| `spec_path` | Yes | Path or URL to your OpenAPI specification file (JSON or YAML) |
| `auth_type` | No | Authentication type: `none`, `api_key`, `bearer_token`, `basic`, `authorization` |
| `auth_value` | No | Authentication value (required if `auth_type` is set) |
| `description` | No | Optional description for the MCP server |
| `allowed_tools` | No | List of specific tools to allow (see [MCP Tool Filtering](#mcp-tool-filtering)) |
| `disallowed_tools` | No | List of specific tools to block (see [MCP Tool Filtering](#mcp-tool-filtering)) |
### Usage Example
Once configured, you can use the OpenAPI-based MCP server just like any other MCP server:
<Tabs>
<TabItem value="fastmcp" label="Python FastMCP">
```python title="Using OpenAPI-based MCP Server" showLineNumbers
from fastmcp import Client
import asyncio
# Standard MCP configuration
config = {
"mcpServers": {
"petstore": {
"url": "http://localhost:4000/petstore_mcp/mcp",
"headers": {
"x-litellm-api-key": "Bearer sk-1234"
}
}
}
}
# Create a client that connects to the server
client = Client(config)
async def main():
async with client:
# List available tools generated from OpenAPI spec
tools = await client.list_tools()
print(f"Available tools: {[tool.name for tool in tools]}")
# Example: Get a pet by ID (from Petstore API)
response = await client.call_tool(
name="getpetbyid",
arguments={"petId": "1"}
)
print(f"Response:\n{response}\n")
# Example: Find pets by status
response = await client.call_tool(
name="findpetsbystatus",
arguments={"status": "available"}
)
print(f"Response:\n{response}\n")
if __name__ == "__main__":
asyncio.run(main())
```
</TabItem>
<TabItem value="cursor" label="Cursor IDE">
```json title="Cursor MCP Configuration for OpenAPI Server" showLineNumbers
{
"mcpServers": {
"Petstore": {
"url": "http://localhost:4000/petstore_mcp/mcp",
"headers": {
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
}
}
}
}
```
</TabItem>
<TabItem value="openai" label="OpenAI Responses API">
```bash title="Using OpenAPI MCP Server with OpenAI" showLineNumbers
curl --location 'https://api.openai.com/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $OPENAI_API_KEY" \
--data '{
"model": "gpt-4o",
"tools": [
{
"type": "mcp",
"server_label": "petstore",
"server_url": "http://localhost:4000/petstore_mcp/mcp",
"require_approval": "never",
"headers": {
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY"
}
}
],
"input": "Find all available pets in the petstore",
"tool_choice": "required"
}'
```
</TabItem>
</Tabs>
### How It Works
1. **Spec Loading**: LiteLLM loads your OpenAPI specification from the provided `spec_path`
2. **Tool Generation**: Each API endpoint in the spec becomes an MCP tool
3. **Parameter Mapping**: OpenAPI parameters are automatically mapped to MCP tool parameters
4. **Request Handling**: When a tool is called, LiteLLM converts the MCP request to the appropriate HTTP request
5. **Response Translation**: API responses are converted back to MCP format
### OpenAPI Spec Requirements
Your OpenAPI specification should follow standard OpenAPI/Swagger conventions:
- **Supported versions**: OpenAPI 3.0.x, OpenAPI 3.1.x, Swagger 2.0
- **Required fields**: `paths`, `info` sections should be properly defined
- **Operation IDs**: Each operation should have a unique `operationId` (this becomes the tool name)
- **Parameters**: Request parameters should be properly documented with types and descriptions
### Example OpenAPI Spec Structure
```yaml title="sample-openapi.yaml" showLineNumbers
openapi: 3.0.0
info:
title: My API
version: 1.0.0
paths:
/pets/{petId}:
get:
operationId: getPetById
summary: Get a pet by ID
parameters:
- name: petId
in: path
required: true
schema:
type: integer
responses:
'200':
description: Successful response
content:
application/json:
schema:
type: object
```
## Allow/Disallow MCP Tools
Control which tools are available from your MCP servers. You can either allow only specific tools or block dangerous ones.
<Tabs>
@ -269,210 +501,118 @@ mcp_servers:
- If you specify both `allowed_tools` and `disallowed_tools`, the allowed list takes priority
- Tool names are case-sensitive
## Using your MCP
---
### Use on LiteLLM UI
## Allow/Disallow MCP Tool Parameters
Follow this walkthrough to use your MCP on LiteLLM UI
Control which parameters are allowed for specific MCP tools using the `allowed_params` configuration. This provides fine-grained control over tool usage by restricting the parameters that can be passed to each tool.
<iframe width="840" height="500" src="https://www.loom.com/embed/57e0763267254bc79dbe6658d0b8758c" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
### Configuration
### Use with Responses API
`allowed_params` is a dictionary that maps tool names to lists of allowed parameter names. When configured, only the specified parameters will be accepted for that tool - any other parameters will be rejected with a 403 error.
Replace `http://localhost:4000` with your LiteLLM Proxy base URL.
Demo Video Using Responses API with LiteLLM Proxy: [Demo video here](https://www.loom.com/share/34587e618c5c47c0b0d67b4e4d02718f?sid=2caf3d45-ead4-4490-bcc1-8d6dd6041c02)
<Tabs>
<TabItem value="curl" label="cURL">
```bash title="cURL Example" showLineNumbers
curl --location 'http://localhost:4000/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer sk-1234" \
--data '{
"model": "gpt-5",
"input": [
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
"tools": [
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy",
"require_approval": "never"
}
],
"stream": true,
"tool_choice": "required"
}'
```yaml title="config.yaml with allowed_params" showLineNumbers
mcp_servers:
deepwiki_mcp:
url: https://mcp.deepwiki.com/mcp
transport: "http"
auth_type: "none"
allowed_params:
# Tool name: list of allowed parameters
read_wiki_contents: ["status"]
my_api_mcp:
url: "https://my-api-server.com"
auth_type: "api_key"
auth_value: "my-key"
allowed_params:
# Using unprefixed tool name
getpetbyid: ["status"]
# Using prefixed tool name (both formats work)
my_api_mcp-findpetsbystatus: ["status", "limit"]
# Another tool with multiple allowed params
create_issue: ["title", "body", "labels"]
```
</TabItem>
<TabItem value="python" label="Python SDK">
### How It Works
```python title="Python SDK Example" showLineNumbers
"""
Use LiteLLM Proxy MCP Gateway to call MCP tools.
1. **Tool-specific filtering**: Each tool can have its own list of allowed parameters
2. **Flexible naming**: Tool names can be specified with or without the server prefix (e.g., both `"getpetbyid"` and `"my_api_mcp-getpetbyid"` work)
3. **Whitelist approach**: Only parameters in the allowed list are permitted
4. **Unlisted tools**: If `allowed_params` is not set, all parameters are allowed
5. **Error handling**: Requests with disallowed parameters receive a 403 error with details about which parameters are allowed
When using LiteLLM Proxy, you can use the same MCP tools across all your LLM providers.
"""
import openai
### Example Request Behavior
client = openai.OpenAI(
api_key="sk-1234", # paste your litellm proxy api key here
base_url="http://localhost:4000" # paste your litellm proxy base url here
)
print("Making API request to Responses API with MCP tools")
With the configuration above, here's how requests would be handled:
response = client.responses.create(
model="gpt-5",
input=[
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
tools=[
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy",
"require_approval": "never"
}
],
stream=True,
tool_choice="required"
)
for chunk in response:
print("response chunk: ", chunk)
```
</TabItem>
</Tabs>
#### Specifying MCP Tools
You can specify which MCP tools are available by using the `allowed_tools` parameter. This allows you to restrict access to specific tools within an MCP server.
To get the list of allowed tools when using LiteLLM MCP Gateway, you can naigate to the LiteLLM UI on MCP Servers > MCP Tools > Click the Tool > Copy Tool Name.
<Tabs>
<TabItem value="curl" label="cURL">
```bash title="cURL Example with allowed_tools" showLineNumbers
curl --location 'http://localhost:4000/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer sk-1234" \
--data '{
"model": "gpt-5",
"input": [
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
"tools": [
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy/mcp",
"require_approval": "never",
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
}
],
"stream": true,
"tool_choice": "required"
}'
```
</TabItem>
<TabItem value="python" label="Python SDK">
```python title="Python SDK Example with allowed_tools" showLineNumbers
import openai
client = openai.OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000"
)
response = client.responses.create(
model="gpt-5",
input=[
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
tools=[
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy/mcp",
"require_approval": "never",
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
}
],
stream=True,
tool_choice="required"
)
print(response)
```
</TabItem>
</Tabs>
### Use with Cursor IDE
Use tools directly from Cursor IDE with LiteLLM MCP:
**Setup Instructions:**
1. **Open Cursor Settings**: Use `⇧+⌘+J` (Mac) or `Ctrl+Shift+J` (Windows/Linux)
2. **Navigate to MCP Tools**: Go to the "MCP Tools" tab and click "New MCP Server"
3. **Add Configuration**: Copy and paste the JSON configuration below, then save with `Cmd+S` or `Ctrl+S`
```json title="Basic Cursor MCP Configuration" showLineNumbers
**✅ Allowed Request:**
```json
{
"mcpServers": {
"LiteLLM": {
"url": "litellm_proxy",
"headers": {
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
}
}
"tool": "read_wiki_contents",
"arguments": {
"status": "active"
}
}
```
#### How it works when server_url="litellm_proxy"
**❌ Rejected Request:**
```json
{
"tool": "read_wiki_contents",
"arguments": {
"status": "active",
"limit": 10 // This parameter is not allowed
}
}
```
When server_url="litellm_proxy", LiteLLM bridges non-MCP providers to your MCP tools.
**Error Response:**
```json
{
"error": "Parameters ['limit'] are not allowed for tool read_wiki_contents. Allowed parameters: ['status']. Contact proxy admin to allow these parameters."
}
```
- Tool Discovery: LiteLLM fetches MCP tools and converts them to OpenAI-compatible definitions
- LLM Call: Tools are sent to the LLM with your input; LLM selects which tools to call
- Tool Execution: LiteLLM automatically parses arguments, routes calls to MCP servers, executes tools, and retrieves results
- Response Integration: Tool results are sent back to LLM for final response generation
- Output: Complete response combining LLM reasoning with tool execution results
### Use Cases
This enables MCP tool usage with any LiteLLM-supported provider, regardless of native MCP support.
- **Security**: Prevent users from accessing sensitive parameters or dangerous operations
- **Cost control**: Restrict expensive parameters (e.g., limiting result counts)
- **Compliance**: Enforce parameter usage policies for regulatory requirements
- **Staged rollouts**: Gradually enable parameters as tools are tested
- **Multi-tenant isolation**: Different parameter access for different user groups
#### Auto-execution for require_approval: "never"
### Combining with Tool Filtering
Setting require_approval: "never" triggers automatic tool execution, returning the final response in a single API call without additional user interaction.
`allowed_params` works alongside `allowed_tools` and `disallowed_tools` for complete control:
```yaml title="Combined filtering example" showLineNumbers
mcp_servers:
github_mcp:
url: "https://api.githubcopilot.com/mcp"
auth_type: oauth2
authorization_url: https://github.com/login/oauth/authorize
token_url: https://github.com/login/oauth/access_token
client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
scopes: ["public_repo", "user:email"]
# Only allow specific tools
allowed_tools: ["create_issue", "list_issues", "search_issues"]
# Block dangerous operations
disallowed_tools: ["delete_repo"]
# Restrict parameters per tool
allowed_params:
create_issue: ["title", "body", "labels"]
list_issues: ["state", "sort", "perPage"]
search_issues: ["query", "sort", "order", "perPage"]
```
This configuration ensures that:
1. Only the three listed tools are available
2. The `delete_repo` tool is explicitly blocked
3. Each tool can only use its specified parameters
---
## MCP Server Access Control
@ -1452,221 +1592,6 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
}'
```
## MCP Cost Tracking
LiteLLM provides two ways to track costs for MCP tool calls:
| Method | When to Use | What It Does |
|--------|-------------|--------------|
| **Config-based Cost Tracking** | Simple cost tracking with fixed costs per tool/server | Automatically tracks costs based on configuration |
| **Custom Post-MCP Hook** | Dynamic cost tracking with custom logic | Allows custom cost calculations and response modifications |
### Config-based Cost Tracking
Configure fixed costs for MCP servers directly in your config.yaml:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
mcp_servers:
zapier_server:
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
mcp_info:
mcp_server_cost_info:
# Default cost for all tools in this server
default_cost_per_query: 0.01
# Custom cost for specific tools
tool_name_to_cost_per_query:
send_email: 0.05
create_document: 0.03
expensive_api_server:
url: "https://api.expensive-service.com/mcp"
mcp_info:
mcp_server_cost_info:
default_cost_per_query: 1.50
```
### Custom Post-MCP Hook
Use this when you need dynamic cost calculation or want to modify the MCP response before it's returned to the user.
#### 1. Create a custom MCP hook file
```python title="custom_mcp_hook.py" showLineNumbers
from typing import Optional
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.mcp import MCPPostCallResponseObject
class CustomMCPCostTracker(CustomLogger):
"""
Custom handler for MCP cost tracking and response modification
"""
async def async_post_mcp_tool_call_hook(
self,
kwargs,
response_obj: MCPPostCallResponseObject,
start_time,
end_time
) -> Optional[MCPPostCallResponseObject]:
"""
Called after each MCP tool call.
Modify costs and response before returning to user.
"""
# Extract tool information from kwargs
tool_name = kwargs.get("name", "")
server_name = kwargs.get("server_name", "")
# Calculate custom cost based on your logic
custom_cost = 42.00
# Set the response cost
response_obj.hidden_params.response_cost = custom_cost
return response_obj
# Create instance for LiteLLM to use
custom_mcp_cost_tracker = CustomMCPCostTracker()
```
#### 2. Configure in config.yaml
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
# Add your custom MCP hook
callbacks:
- custom_mcp_hook.custom_mcp_cost_tracker
mcp_servers:
zapier_server:
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
```
#### 3. Start the proxy
```shell
$ litellm --config /path/to/config.yaml
```
When MCP tools are called, your custom hook will:
1. Calculate costs based on your custom logic
2. Modify the response if needed
3. Track costs in LiteLLM's logging system
## MCP Guardrails
LiteLLM supports applying guardrails to MCP tool calls to ensure security and compliance. You can configure guardrails to run before or during MCP calls to validate inputs and block or mask sensitive information.
### Supported MCP Guardrail Modes
MCP guardrails support the following modes:
- `pre_mcp_call`: Run **before** MCP call, on **input**. Use this mode when you want to apply validation/masking/blocking for MCP requests
- `during_mcp_call`: Run **during** MCP call execution. Use this mode for real-time monitoring and intervention
### Configuration Examples
Configure guardrails to run before MCP tool calls to validate and sanitize inputs:
```yaml title="config.yaml" showLineNumbers
guardrails:
- guardrail_name: "mcp-input-validation"
litellm_params:
guardrail: presidio # or other supported guardrails
mode: "pre_mcp_call" # or during_mcp_call
pii_entities_config:
CREDIT_CARD: "BLOCK" # Will block requests containing credit card numbers
EMAIL_ADDRESS: "MASK" # Will mask email addresses
PHONE_NUMBER: "MASK" # Will mask phone numbers
default_on: true
```
### Usage Examples
#### Testing Pre-MCP Call Guardrails
Test your MCP guardrails with a request that includes sensitive information:
```bash title="Test MCP Guardrail" showLineNumbers
curl http://localhost:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "My credit card is 4111-1111-1111-1111 and my email is john@example.com"}
],
"guardrails": ["mcp-input-validation"]
}'
```
The request will be processed as follows:
1. Credit card number will be blocked (request rejected)
2. Email address will be masked (e.g., replaced with `<EMAIL_ADDRESS>`)
#### Using with MCP Tools
When using MCP tools, guardrails will be applied to the tool inputs:
```python title="Python Example with MCP Guardrails" showLineNumbers
import openai
client = openai.OpenAI(
api_key="your-api-key",
base_url="http://localhost:4000"
)
# This request will trigger MCP guardrails
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Send an email to 555-123-4567 with my SSN 123-45-6789"}
],
tools=[{"type": "mcp", "server_label": "litellm", "server_url": "litellm_proxy"}],
guardrails=["mcp-input-validation"]
)
```
### Supported Guardrail Providers
MCP guardrails work with all LiteLLM-supported guardrail providers:
- **Presidio**: PII detection and masking
- **Bedrock**: AWS Bedrock guardrails
- **Lakera**: Content moderation
- **Aporia**: Custom guardrails
- **Custom**: Your own guardrail implementations
## MCP Permission Management
LiteLLM supports managing permissions for MCP Servers by Keys, Teams, Organizations (entities) on LiteLLM. When a MCP client attempts to list tools, LiteLLM will only return the tools the entity has permissions to access.
When Creating a Key, Team, or Organization, you can select the allowed MCP Servers that the entity has access to.
<Image
img={require('../img/mcp_key.png')}
style={{width: '80%', display: 'block', margin: '0'}}
/>
## LiteLLM Proxy - Walk through MCP Gateway
LiteLLM exposes an MCP Gateway for admins to add all their MCP servers to LiteLLM. The key benefits of using LiteLLM Proxy with MCP are:

45
docs/my-website/docs/mcp_control.md Normal file
View file

@ -0,0 +1,45 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# MCP Permission Management
Control which MCP servers and tools can be accessed by specific keys, teams, or organizations in LiteLLM. When a client attempts to list or call tools, LiteLLM enforces access controls based on configured permissions.
## Overview
LiteLLM provides fine-grained permission management for MCP servers, allowing you to:
- **Restrict MCP access by entity**: Control which keys, teams, or organizations can access specific MCP servers
- **Tool-level filtering**: Automatically filter available tools based on entity permissions
- **Centralized control**: Manage all MCP permissions from the LiteLLM Admin UI or API
This ensures that only authorized entities can discover and use MCP tools, providing an additional security layer for your MCP infrastructure.
:::info Related Documentation
- [MCP Overview](./mcp.md) - Learn about MCP in LiteLLM
- [MCP Cost Tracking](./mcp_cost.md) - Track costs for MCP tool calls
- [MCP Guardrails](./mcp_guardrail.md) - Apply security guardrails to MCP calls
- [Using MCP](./mcp_usage.md) - How to use MCP with LiteLLM
:::
## How It Works
LiteLLM supports managing permissions for MCP Servers by Keys, Teams, Organizations (entities) on LiteLLM. When a MCP client attempts to list tools, LiteLLM will only return the tools the entity has permissions to access.
When Creating a Key, Team, or Organization, you can select the allowed MCP Servers that the entity has access to.
<Image
img={require('../img/mcp_key.png')}
style={{width: '80%', display: 'block', margin: '0'}}
/>
## Set Allowed Tools for a Key, Team, or Organization
Control which tools different teams can access from the same MCP server. For example, give your Engineering team access to `list_repositories`, `create_issue`, and `search_code`, while Sales only gets `search_code` and `close_issue`.
This video shows how to set allowed tools for a Key, Team, or Organization.
<iframe width="840" height="500" src="https://www.loom.com/embed/7464d444c3324078892367272fe50745" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>

121
docs/my-website/docs/mcp_cost.md Normal file
View file

@ -0,0 +1,121 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# MCP Cost Tracking
LiteLLM provides two ways to track costs for MCP tool calls:
| Method | When to Use | What It Does |
|--------|-------------|--------------|
| **Config-based Cost Tracking** | Simple cost tracking with fixed costs per tool/server | Automatically tracks costs based on configuration |
| **Custom Post-MCP Hook** | Dynamic cost tracking with custom logic | Allows custom cost calculations and response modifications |
### Config-based Cost Tracking
Configure fixed costs for MCP servers directly in your config.yaml:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
mcp_servers:
zapier_server:
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
mcp_info:
mcp_server_cost_info:
# Default cost for all tools in this server
default_cost_per_query: 0.01
# Custom cost for specific tools
tool_name_to_cost_per_query:
send_email: 0.05
create_document: 0.03
expensive_api_server:
url: "https://api.expensive-service.com/mcp"
mcp_info:
mcp_server_cost_info:
default_cost_per_query: 1.50
```
### Custom Post-MCP Hook
Use this when you need dynamic cost calculation or want to modify the MCP response before it's returned to the user.
#### 1. Create a custom MCP hook file
```python title="custom_mcp_hook.py" showLineNumbers
from typing import Optional
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.mcp import MCPPostCallResponseObject
class CustomMCPCostTracker(CustomLogger):
"""
Custom handler for MCP cost tracking and response modification
"""
async def async_post_mcp_tool_call_hook(
self,
kwargs,
response_obj: MCPPostCallResponseObject,
start_time,
end_time
) -> Optional[MCPPostCallResponseObject]:
"""
Called after each MCP tool call.
Modify costs and response before returning to user.
"""
# Extract tool information from kwargs
tool_name = kwargs.get("name", "")
server_name = kwargs.get("server_name", "")
# Calculate custom cost based on your logic
custom_cost = 42.00
# Set the response cost
response_obj.hidden_params.response_cost = custom_cost
return response_obj
# Create instance for LiteLLM to use
custom_mcp_cost_tracker = CustomMCPCostTracker()
```
#### 2. Configure in config.yaml
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
# Add your custom MCP hook
callbacks:
- custom_mcp_hook.custom_mcp_cost_tracker
mcp_servers:
zapier_server:
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
```
#### 3. Start the proxy
```shell
$ litellm --config /path/to/config.yaml
```
When MCP tools are called, your custom hook will:
1. Calculate costs based on your custom logic
2. Modify the response if needed
3. Track costs in LiteLLM's logging system

88
docs/my-website/docs/mcp_guardrail.md Normal file
View file

@ -0,0 +1,88 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# MCP Guardrails
LiteLLM supports applying guardrails to MCP tool calls to ensure security and compliance. You can configure guardrails to run before or during MCP calls to validate inputs and block or mask sensitive information.
### Supported MCP Guardrail Modes
MCP guardrails support the following modes:
- `pre_mcp_call`: Run **before** MCP call, on **input**. Use this mode when you want to apply validation/masking/blocking for MCP requests
- `during_mcp_call`: Run **during** MCP call execution. Use this mode for real-time monitoring and intervention
### Configuration Examples
Configure guardrails to run before MCP tool calls to validate and sanitize inputs:
```yaml title="config.yaml" showLineNumbers
guardrails:
- guardrail_name: "mcp-input-validation"
litellm_params:
guardrail: presidio # or other supported guardrails
mode: "pre_mcp_call" # or during_mcp_call
pii_entities_config:
CREDIT_CARD: "BLOCK" # Will block requests containing credit card numbers
EMAIL_ADDRESS: "MASK" # Will mask email addresses
PHONE_NUMBER: "MASK" # Will mask phone numbers
default_on: true
```
### Usage Examples
#### Testing Pre-MCP Call Guardrails
Test your MCP guardrails with a request that includes sensitive information:
```bash title="Test MCP Guardrail" showLineNumbers
curl http://localhost:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "My credit card is 4111-1111-1111-1111 and my email is john@example.com"}
],
"guardrails": ["mcp-input-validation"]
}'
```
The request will be processed as follows:
1. Credit card number will be blocked (request rejected)
2. Email address will be masked (e.g., replaced with `<EMAIL_ADDRESS>`)
#### Using with MCP Tools
When using MCP tools, guardrails will be applied to the tool inputs:
```python title="Python Example with MCP Guardrails" showLineNumbers
import openai
client = openai.OpenAI(
api_key="your-api-key",
base_url="http://localhost:4000"
)
# This request will trigger MCP guardrails
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Send an email to 555-123-4567 with my SSN 123-45-6789"}
],
tools=[{"type": "mcp", "server_label": "litellm", "server_url": "litellm_proxy"}],
guardrails=["mcp-input-validation"]
)
```
### Supported Guardrail Providers
MCP guardrails work with all LiteLLM-supported guardrail providers:
- **Presidio**: PII detection and masking
- **Bedrock**: AWS Bedrock guardrails
- **Lakera**: Content moderation
- **Aporia**: Custom guardrails
- **Custom**: Your own guardrail implementations

209
docs/my-website/docs/mcp_usage.md Normal file
View file

@ -0,0 +1,209 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# Using your MCP
This document covers how to use LiteLLM as an MCP Gateway. You can see how to use it with Responses API, Cursor IDE, and OpenAI SDK.
### Use on LiteLLM UI
Follow this walkthrough to use your MCP on LiteLLM UI
<iframe width="840" height="500" src="https://www.loom.com/embed/57e0763267254bc79dbe6658d0b8758c" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
### Use with Responses API
Replace `http://localhost:4000` with your LiteLLM Proxy base URL.
Demo Video Using Responses API with LiteLLM Proxy: [Demo video here](https://www.loom.com/share/34587e618c5c47c0b0d67b4e4d02718f?sid=2caf3d45-ead4-4490-bcc1-8d6dd6041c02)
<Tabs>
<TabItem value="curl" label="cURL">
```bash title="cURL Example" showLineNumbers
curl --location 'http://localhost:4000/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer sk-1234" \
--data '{
"model": "gpt-5",
"input": [
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
"tools": [
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy",
"require_approval": "never"
}
],
"stream": true,
"tool_choice": "required"
}'
```
</TabItem>
<TabItem value="python" label="Python SDK">
```python title="Python SDK Example" showLineNumbers
"""
Use LiteLLM Proxy MCP Gateway to call MCP tools.
When using LiteLLM Proxy, you can use the same MCP tools across all your LLM providers.
"""
import openai
client = openai.OpenAI(
api_key="sk-1234", # paste your litellm proxy api key here
base_url="http://localhost:4000" # paste your litellm proxy base url here
)
print("Making API request to Responses API with MCP tools")
response = client.responses.create(
model="gpt-5",
input=[
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
tools=[
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy",
"require_approval": "never"
}
],
stream=True,
tool_choice="required"
)
for chunk in response:
print("response chunk: ", chunk)
```
</TabItem>
</Tabs>
#### Specifying MCP Tools
You can specify which MCP tools are available by using the `allowed_tools` parameter. This allows you to restrict access to specific tools within an MCP server.
To get the list of allowed tools when using LiteLLM MCP Gateway, you can naigate to the LiteLLM UI on MCP Servers > MCP Tools > Click the Tool > Copy Tool Name.
<Tabs>
<TabItem value="curl" label="cURL">
```bash title="cURL Example with allowed_tools" showLineNumbers
curl --location 'http://localhost:4000/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer sk-1234" \
--data '{
"model": "gpt-5",
"input": [
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
"tools": [
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy/mcp",
"require_approval": "never",
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
}
],
"stream": true,
"tool_choice": "required"
}'
```
</TabItem>
<TabItem value="python" label="Python SDK">
```python title="Python SDK Example with allowed_tools" showLineNumbers
import openai
client = openai.OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000"
)
response = client.responses.create(
model="gpt-5",
input=[
{
"role": "user",
"content": "give me TLDR of what BerriAI/litellm repo is about",
"type": "message"
}
],
tools=[
{
"type": "mcp",
"server_label": "litellm",
"server_url": "litellm_proxy/mcp",
"require_approval": "never",
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
}
],
stream=True,
tool_choice="required"
)
print(response)
```
</TabItem>
</Tabs>
### Use with Cursor IDE
Use tools directly from Cursor IDE with LiteLLM MCP:
**Setup Instructions:**
1. **Open Cursor Settings**: Use `⇧+⌘+J` (Mac) or `Ctrl+Shift+J` (Windows/Linux)
2. **Navigate to MCP Tools**: Go to the "MCP Tools" tab and click "New MCP Server"
3. **Add Configuration**: Copy and paste the JSON configuration below, then save with `Cmd+S` or `Ctrl+S`
```json title="Basic Cursor MCP Configuration" showLineNumbers
{
"mcpServers": {
"LiteLLM": {
"url": "litellm_proxy",
"headers": {
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
}
}
}
}
```
#### How it works when server_url="litellm_proxy"
When server_url="litellm_proxy", LiteLLM bridges non-MCP providers to your MCP tools.
- Tool Discovery: LiteLLM fetches MCP tools and converts them to OpenAI-compatible definitions
- LLM Call: Tools are sent to the LLM with your input; LLM selects which tools to call
- Tool Execution: LiteLLM automatically parses arguments, routes calls to MCP servers, executes tools, and retrieves results
- Response Integration: Tool results are sent back to LLM for final response generation
- Output: Complete response combining LLM reasoning with tool execution results
This enables MCP tool usage with any LiteLLM-supported provider, regardless of native MCP support.
#### Auto-execution for require_approval: "never"
Setting require_approval: "never" triggers automatic tool execution, returning the final response in a single API call without additional user interaction.

45
docs/my-website/docs/observability/posthog_integration.md
View file

@ -55,6 +55,26 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}'
```
### Team-Based Logging
Configure different PostHog credentials per team using the team callback settings:
```bash
curl -X POST 'http://localhost:4000/team/{team_id}/callback' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"callback_name": "posthog",
"callback_type": "success",
"callback_vars": {
"posthog_api_key": "ph_team_specific_key",
"posthog_api_url": "https://custom.posthog.com"
}
}'
```
Now all requests from that team will be logged to their specific PostHog project.
## Usage with LiteLLM Python SDK
### Quick Start
@ -142,6 +162,31 @@ response = client.chat.completions.create(
)
```
#### Per-Request Credentials
You can override PostHog credentials on a per-request basis:
```python
import litellm
litellm.success_callback = ["posthog"]
# Use custom PostHog credentials for this specific request
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Hello world"}
],
posthog_api_key="ph_custom_project_key",
posthog_api_url="https://custom.posthog.com"
)
```
This is useful when you need to:
- Log different teams/projects to separate PostHog instances
- Use different PostHog projects for staging vs production
- Route logs based on customer or tenant
#### Disable Logging for Specific Calls
Use the `no-log` flag to prevent logging for specific calls:

1
docs/my-website/docs/providers/bedrock_embedding.md
View file

@ -265,6 +265,7 @@ print(response)
| TwelveLabs Marengo Embed 2.7 | `embedding(model="bedrock/us.twelvelabs.marengo-embed-2-7-v1:0", input=input)` | Supports multimodal input (text, video, audio, image) |
| Cohere Embeddings - English | `embedding(model="bedrock/cohere.embed-english-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18)
| Cohere Embeddings - Multilingual | `embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18)
| Cohere Embed v4 | `embedding(model="bedrock/cohere.embed-v4:0", input=input)` | Supports text and image input, configurable dimensions (256, 512, 1024, 1536), 128k context length |
### Advanced - [Drop Unsupported Params](https://docs.litellm.ai/docs/completion/drop_params#openai-proxy-usage)

30
docs/my-website/docs/providers/oci.md
View file

@ -6,18 +6,27 @@ LiteLLM supports the following models for OCI on-demand GenAI API.
Check the [OCI Models List](https://docs.oracle.com/en-us/iaas/Content/generative-ai/pretrained-models.htm) to see if the model is available for your region.
## Supported Models
### Meta Llama Models
- `meta.llama-4-maverick-17b-128e-instruct-fp8`
- `meta.llama-4-scout-17b-16e-instruct`
- `meta.llama-3.3-70b-instruct`
- `meta.llama-3.2-90b-vision-instruct`
- `meta.llama-3.1-405b-instruct`
### xAI Grok Models
- `xai.grok-4`
- `xai.grok-3`
- `xai.grok-3-fast`
- `xai.grok-3-mini`
- `xai.grok-3-mini-fast`
### Cohere Models
- `cohere.command-latest`
- `cohere.command-a-03-2025`
- `cohere.command-plus-latest`
## Authentication
LiteLLM uses OCI signing key authentication. Follow the [official Oracle tutorial](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm) to create a signing key and obtain the following parameters:
@ -83,3 +92,24 @@ response = completion(
for chunk in response:
print(chunk["choices"][0]["delta"]["content"]) # same as openai format
```
## Usage Examples by Model Type
### Using Cohere Models
```python
from litellm import completion
messages = [{"role": "user", "content": "Explain quantum computing"}]
response = completion(
model="oci/cohere.command-latest",
messages=messages,
oci_region="us-chicago-1",
oci_user=<your_oci_user>,
oci_fingerprint=<your_oci_fingerprint>,
oci_tenancy=<your_oci_tenancy>,
oci_key=<string_with_content_of_oci_key>,
oci_compartment_id=<oci_compartment_id>,
)
print(response)
```

21
docs/my-website/docs/providers/openai.md
View file

@ -171,6 +171,7 @@ os.environ["OPENAI_BASE_URL"] = "https://your_host/v1" # OPTIONAL
| gpt-5-2025-08-07 | `response = completion(model="gpt-5-2025-08-07", messages=messages)` |
| gpt-5-mini-2025-08-07 | `response = completion(model="gpt-5-mini-2025-08-07", messages=messages)` |
| gpt-5-nano-2025-08-07 | `response = completion(model="gpt-5-nano-2025-08-07", messages=messages)` |
| gpt-5-pro | `response = completion(model="gpt-5-pro", messages=messages)` |
| gpt-4.1 | `response = completion(model="gpt-4.1", messages=messages)` |
| gpt-4.1-mini | `response = completion(model="gpt-4.1-mini", messages=messages)` |
| gpt-4.1-nano | `response = completion(model="gpt-4.1-nano", messages=messages)` |
@ -749,4 +750,24 @@ In your logs you should see the forwarded org id
```bash
LiteLLM:DEBUG: utils.py:255 - Request to litellm:
LiteLLM:DEBUG: utils.py:255 - litellm.acompletion(... organization='my-special-org',)
```
## GPT-5 Pro Special Notes
GPT-5 Pro is OpenAI's most advanced reasoning model with unique characteristics:
- **Responses API Only**: GPT-5 Pro is only available through the `/v1/responses` endpoint
- **No Streaming**: Does not support streaming responses
- **High Reasoning**: Designed for complex reasoning tasks with highest effort reasoning
- **Context Window**: 400,000 tokens input, 272,000 tokens output
- **Pricing**: $15.00 input / $120.00 output per 1M tokens (Standard), $7.50 input / $60.00 output (Batch)
- **Tools**: Supports Web Search, File Search, Image Generation, MCP (but not Code Interpreter or Computer Use)
- **Modalities**: Text and Image input, Text output only
```python
# GPT-5 Pro usage example
response = completion(
model="gpt-5-pro",
messages=[{"role": "user", "content": "Solve this complex reasoning problem..."}]
)
```

50
docs/my-website/docs/providers/openai/responses_api.md
View file

@ -37,6 +37,29 @@ for event in response:
print(event)
```
#### Image Generation with Streaming
```python showLineNumbers title="OpenAI Streaming Image Generation"
import litellm
import base64
# Streaming image generation with partial images
stream = litellm.responses(
model="gpt-4.1", # Use an actual image generation model
input="Generate a gorgeous image of a river made of white owl feathers",
stream=True,
tools=[{"type": "image_generation", "partial_images": 2}],
)
for event in stream:
if event.type == "response.image_generation_call.partial_image":
idx = event.partial_image_index
image_base64 = event.partial_image_b64
image_bytes = base64.b64decode(image_base64)
with open(f"river{idx}.png", "wb") as f:
f.write(image_bytes)
```
#### GET a Response
```python showLineNumbers title="Get Response by ID"
import litellm
@ -150,6 +173,33 @@ for event in response:
print(event)
```
#### Image Generation with Streaming
```python showLineNumbers title="OpenAI Proxy Streaming Image Generation"
from openai import OpenAI
import base64
# Initialize client with your proxy URL
client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000")
stream = client.responses.create(
model="gpt-4.1",
input="Draw a gorgeous image of a river made of white owl feathers, snaking its way through a serene winter landscape",
stream=True,
tools=[{"type": "image_generation", "partial_images": 2}],
)
for event in stream:
print(f"event: {event}")
if event.type == "response.image_generation_call.partial_image":
idx = event.partial_image_index
image_base64 = event.partial_image_b64
image_bytes = base64.b64decode(image_base64)
with open(f"river{idx}.png", "wb") as f:
f.write(image_bytes)
```
#### GET a Response
```python showLineNumbers title="Get Response by ID with OpenAI SDK"
from openai import OpenAI

159
docs/my-website/docs/providers/vertex.md
View file

@ -191,7 +191,7 @@ print(json.loads(completion.choices[0].message.content))
model_list:
- model_name: gemini-2.5-pro
litellm_params:
model: vertex_ai/gemini-1.5-pro
model: vertex_ai/gemini-2.5-pro
vertex_project: "project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json" # [OPTIONAL] Do this OR `!gcloud auth application-default login` - run this to add vertex credentials to your env
@ -277,7 +277,7 @@ except JSONSchemaValidationError as e:
model_list:
- model_name: gemini-2.5-pro
litellm_params:
model: vertex_ai/gemini-1.5-pro
model: vertex_ai/gemini-2.5-pro
vertex_project: "project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json" # [OPTIONAL] Do this OR `!gcloud auth application-default login` - run this to add vertex credentials to your env
@ -981,11 +981,158 @@ curl http://0.0.0.0:4000/v1/chat/completions \
### **Context Caching**
Use Vertex AI context caching is supported by calling provider api directly. (Unified Endpoint support coming soon.).
#### Unified Endpoint
Use Vertex AI context caching in the same way as [**Google AI Studio - Context Caching**](../providers/gemini.md#context-caching)
##### Example usage
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
for _ in range(2):
resp = completion(
model="vertex_ai/gemini-2.5-pro",
messages=[
# System Message
{
"role": "system",
"content": [
{
"type": "text",
"text": "Here is the full text of a complex legal agreement" * 4000,
"cache_control": {"type": "ephemeral"}, # 👈 KEY CHANGE
}
],
},
# marked for caching with the cache_control parameter, so that this checkpoint can read from the previous cache.
{
"role": "user",
"content": [
{
"type": "text",
"text": "What are the key terms and conditions in this agreement?",
"cache_control": {"type": "ephemeral"},
}
],
}]
)
print(resp.usage) # 👈 2nd usage block will be less, since cached tokens used
```
</TabItem>
<TabItem value="sdk-ttl" label="SDK with Custom TTL">
```python
from litellm import completion
# Cache for 2 hours (7200 seconds)
resp = completion(
model="vertex_ai/gemini-2.5-pro",
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "Here is the full text of a complex legal agreement" * 4000,
"cache_control": {
"type": "ephemeral",
"ttl": "7200s" # 👈 Cache for 2 hours
},
}
],
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "What are the key terms and conditions in this agreement?",
"cache_control": {
"type": "ephemeral",
"ttl": "3600s" # 👈 This TTL will be ignored (first one is used)
},
}
],
}
]
)
print(resp.usage)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
1. Setup config.yaml
```yaml
model_list:
- model_name: gemini-2.5-pro
litellm_params:
model: vertex_ai/gemini-2.5-pro
vertex_project: "project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gemini-2.5-flash",
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Long cache message (must be >= 1024 tokens)",
"cache_control": {
"type": "ephemeral",
"ttl": "7200s"
}
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is the text about?"
}
]
}
]
}'
```
</TabItem>
</Tabs>
#### Calling provider api directly
[**Go straight to provider**](../pass_through/vertex_ai.md#context-caching)
#### 1. Create the Cache
##### 1. Create the Cache
First, create the cache by sending a `POST` request to the `cachedContents` endpoint via the LiteLLM proxy.
@ -1011,7 +1158,7 @@ curl http://0.0.0.0:4000/vertex_ai/v1/projects/{project_id}/locations/{location}
</TabItem>
</Tabs>
#### 2. Get the Cache Name from the Response
##### 2. Get the Cache Name from the Response
Vertex AI will return a response containing the `name` of the cached content. This name is the identifier for your cached data.
@ -1030,7 +1177,7 @@ Vertex AI will return a response containing the `name` of the cached content. Th
}
```
#### 3. Use the Cached Content
##### 3. Use the Cached Content
Use the `name` from the response as `cachedContent` or `cached_content` in subsequent API calls to reuse the cached information. This is passed in the body of your request to `/chat/completions`.

2
docs/my-website/docs/providers/vertex_batch.md
View file

@ -1,7 +1,7 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
## **Batch APIs**
# Vertex Batch APIs
Just add the following Vertex env vars to your environment.

110
docs/my-website/docs/providers/vertex_partner.md
View file

@ -16,7 +16,6 @@ import TabItem from '@theme/TabItem';
| AI21 (Jamba) | `vertex_ai/jamba-*` | [Vertex AI - AI21 Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/ai21) |
| Qwen | `vertex_ai/qwen/*` | [Vertex AI - Qwen Models](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/qwen) |
| OpenAI (GPT-OSS) | `vertex_ai/openai/gpt-oss-*` | [Vertex AI - GPT-OSS Models](https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/) |
| Model Garden | `vertex_ai/openai/{MODEL_ID}` or `vertex_ai/{MODEL_ID}` | [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) |
## Vertex AI - Anthropic (Claude)
@ -793,112 +792,3 @@ curl http://0.0.0.0:4000/v1/chat/completions \
</TabItem>
</Tabs>
## Model Garden
:::tip
All OpenAI compatible models from Vertex Model Garden are supported.
:::
#### Using Model Garden
**Almost all Vertex Model Garden models are OpenAI compatible.**
<Tabs>
<TabItem value="openai" label="OpenAI Compatible Models">
| Property | Details |
|----------|---------|
| Provider Route | `vertex_ai/openai/{MODEL_ID}` |
| Vertex Documentation | [Model Garden LiteLLM Inference](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb), [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) |
| Supported Operations | `/chat/completions`, `/embeddings` |
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import os
## set ENV variables
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
os.environ["VERTEXAI_LOCATION"] = "us-central1"
response = completion(
model="vertex_ai/openai/<your-endpoint-id>",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)
```
</TabItem>
<TabItem value="proxy" label="Proxy">
**1. Add to config**
```yaml
model_list:
- model_name: llama3-1-8b-instruct
litellm_params:
model: vertex_ai/openai/5464397967697903616
vertex_ai_project: "my-test-project"
vertex_ai_location: "us-east-1"
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000
```
**3. Test it!**
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"model": "llama3-1-8b-instruct", # 👈 the 'model_name' in config
"messages": [
{
"role": "user",
"content": "what llm are you"
}
],
}'
```
</TabItem>
</Tabs>
</TabItem>
<TabItem value="non-openai" label="Non-OpenAI Compatible Models">
```python
from litellm import completion
import os
## set ENV variables
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
os.environ["VERTEXAI_LOCATION"] = "us-central1"
response = completion(
model="vertex_ai/<your-endpoint-id>",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)
```
</TabItem>
</Tabs>

229
docs/my-website/docs/providers/vertex_self_deployed.md Normal file
View file

@ -0,0 +1,229 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Vertex AI - Self Deployed Models
Deploy and use your own models on Vertex AI through Model Garden or custom endpoints.
## Model Garden
:::tip
All OpenAI compatible models from Vertex Model Garden are supported.
:::
### Using Model Garden
**Almost all Vertex Model Garden models are OpenAI compatible.**
<Tabs>
<TabItem value="openai" label="OpenAI Compatible Models">
| Property | Details |
|----------|---------|
| Provider Route | `vertex_ai/openai/{MODEL_ID}` |
| Vertex Documentation | [Model Garden LiteLLM Inference](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb), [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) |
| Supported Operations | `/chat/completions`, `/embeddings` |
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import os
## set ENV variables
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
os.environ["VERTEXAI_LOCATION"] = "us-central1"
response = completion(
model="vertex_ai/openai/<your-endpoint-id>",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)
```
</TabItem>
<TabItem value="proxy" label="Proxy">
**1. Add to config**
```yaml
model_list:
- model_name: llama3-1-8b-instruct
litellm_params:
model: vertex_ai/openai/5464397967697903616
vertex_ai_project: "my-test-project"
vertex_ai_location: "us-east-1"
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000
```
**3. Test it!**
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"model": "llama3-1-8b-instruct", # 👈 the 'model_name' in config
"messages": [
{
"role": "user",
"content": "what llm are you"
}
],
}'
```
</TabItem>
</Tabs>
</TabItem>
<TabItem value="non-openai" label="Non-OpenAI Compatible Models">
```python
from litellm import completion
import os
## set ENV variables
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
os.environ["VERTEXAI_LOCATION"] = "us-central1"
response = completion(
model="vertex_ai/<your-endpoint-id>",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)
```
</TabItem>
</Tabs>
## Gemma Models (Custom Endpoints)
Deploy Gemma models on custom Vertex AI prediction endpoints with OpenAI-compatible format.
| Property | Details |
|----------|---------|
| Provider Route | `vertex_ai/gemma/{MODEL_NAME}` |
| Vertex Documentation | [Vertex AI Prediction](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions) |
| Required Parameter | `api_base` - Full prediction endpoint URL |
**Proxy Usage:**
**1. Add to config.yaml**
```yaml
model_list:
- model_name: gemma-model
litellm_params:
model: vertex_ai/gemma/gemma-3-12b-it-1222199011122
api_base: https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict
vertex_project: "my-project-id"
vertex_location: "us-central1"
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
```
**3. Test it**
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gemma-model",
"messages": [{"role": "user", "content": "What is machine learning?"}],
"max_tokens": 100
}'
```
**SDK Usage:**
```python
from litellm import completion
response = completion(
model="vertex_ai/gemma/gemma-3-12b-it-1222199011122",
messages=[{"role": "user", "content": "What is machine learning?"}],
api_base="https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict",
vertex_project="my-project-id",
vertex_location="us-central1",
)
```
## MedGemma Models (Custom Endpoints)
Deploy MedGemma models on custom Vertex AI prediction endpoints with OpenAI-compatible format. MedGemma models use the same `vertex_ai/gemma/` route.
| Property | Details |
|----------|---------|
| Provider Route | `vertex_ai/gemma/{MODEL_NAME}` |
| Vertex Documentation | [Vertex AI Prediction](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions) |
| Required Parameter | `api_base` - Full prediction endpoint URL |
**Proxy Usage:**
**1. Add to config.yaml**
```yaml
model_list:
- model_name: medgemma-model
litellm_params:
model: vertex_ai/gemma/medgemma-2b-v1
api_base: https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict
vertex_project: "my-project-id"
vertex_location: "us-central1"
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
```
**3. Test it**
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "medgemma-model",
"messages": [{"role": "user", "content": "What are the symptoms of hypertension?"}],
"max_tokens": 100
}'
```
**SDK Usage:**
```python
from litellm import completion
response = completion(
model="vertex_ai/gemma/medgemma-2b-v1",
messages=[{"role": "user", "content": "What are the symptoms of hypertension?"}],
api_base="https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict",
vertex_project="my-project-id",
vertex_location="us-central1",
)
```

196
docs/my-website/docs/providers/wandb_inference.md Normal file
View file

@ -0,0 +1,196 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Weights & Biases Inference
https://weave-docs.wandb.ai/quickstart-inference
:::tip
Litellm provides support to all models from W&B Inference service. To use a model, set `model=wandb/<any-model-on-wandb-inference-dashboard>` as a prefix for litellm requests. The full list of supported models is provided at https://docs.wandb.ai/guides/inference/models/
:::
## API Key
You can get an API key for W&B Inference at - https://wandb.ai/authorize
```python
import os
# env variable
os.environ['WANDB_API_KEY']
```
## Sample Usage: Text Generation
```python
from litellm import completion
import os
os.environ['WANDB_API_KEY'] = "insert-your-wandb-api-key"
response = completion(
model="wandb/Qwen/Qwen3-235B-A22B-Instruct-2507",
messages=[
{
"role": "user",
"content": "What character was Wall-e in love with?",
}
],
max_tokens=10,
response_format={ "type": "json_object" },
seed=123,
temperature=0.6, # either set temperature or `top_p`
top_p=0.01, # to get as deterministic results as possible
)
print(response)
```
## Sample Usage - Streaming
```python
from litellm import completion
import os
os.environ['WANDB_API_KEY'] = ""
response = completion(
model="wandb/Qwen/Qwen3-235B-A22B-Instruct-2507",
messages=[
{
"role": "user",
"content": "What character was Wall-e in love with?",
}
],
stream=True,
max_tokens=10,
response_format={ "type": "json_object" },
seed=123,
temperature=0.6, # either set temperature or `top_p`
top_p=0.01, # to get as deterministic results as possible
)
for chunk in response:
print(chunk)
```
:::tip
The above examples may not work if the model has been taken offline. Check the full list of available models at https://docs.wandb.ai/guides/inference/models/.
:::
## Usage with LiteLLM Proxy Server
Here's how to call a W&B Inference model with the LiteLLM Proxy Server
1. Modify the config.yaml
```yaml
model_list:
- model_name: my-model
litellm_params:
model: wandb/<your-model-name> # add wandb/ prefix to use W&B Inference as provider
api_key: api-key # api key to send your model
```
2. Start the proxy
```bash
$ litellm --config /path/to/config.yaml
```
3. Send Request to LiteLLM Proxy Server
<Tabs>
<TabItem value="openai" label="OpenAI Python v1.0.0+">
```python
import openai
client = openai.OpenAI(
api_key="litellm-proxy-key", # pass litellm proxy key, if you're using virtual keys
base_url="http://0.0.0.0:4000" # litellm-proxy-base url
)
response = client.chat.completions.create(
model="my-model",
messages = [
{
"role": "user",
"content": "What character was Wall-e in love with?"
}
],
)
print(response)
```
</TabItem>
<TabItem value="curl" label="curl">
```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Authorization: litellm-proxy-key' \
--header 'Content-Type: application/json' \
--data '{
"model": "my-model",
"messages": [
{
"role": "user",
"content": "What character was Wall-e in love with?"
}
],
}'
```
</TabItem>
</Tabs>
## Supported Parameters
The W&B Inference provider supports the following parameters:
### Chat Completion Parameters
| Parameter | Type | Description |
| --------- | ---- | ----------- |
| frequency_penalty | number | Penalizes new tokens based on their frequency in the text |
| function_call | string/object | Controls how the model calls functions |
| functions | array | List of functions for which the model may generate JSON inputs |
| logit_bias | map | Modifies the likelihood of specified tokens |
| max_tokens | integer | Maximum number of tokens to generate |
| n | integer | Number of completions to generate |
| presence_penalty | number | Penalizes tokens based on if they appear in the text so far |
| response_format | object | Format of the response, e.g., `{"type": "json"}` |
| seed | integer | Sampling seed for deterministic results |
| stop | string/array | Sequences where the API will stop generating tokens |
| stream | boolean | Whether to stream the response |
| temperature | number | Controls randomness (0-2) |
| top_p | number | Controls nucleus sampling |
## Error Handling
The integration uses the standard LiteLLM error handling. Further, here's a list of commonly encountered errors with the W&B Inference API -
| Error Code | Message | Cause | Solution |
| ---------- | ------- | ----- | -------- |
| 401 | Authentication failed | Your authentication credentials are incorrect or your W&B project entity and/or name are incorrect. | Ensure you're using the correct API key and that your W&B project name and entity are correct. |
| 403 | Country, region, or territory not supported | Accessing the API from an unsupported location. | Please see [Geographic restrictions](https://docs.wandb.ai/guides/inference/usage-limits/#geographic-restrictions) |
| 429 | Concurrency limit reached for requests | Too many concurrent requests. | Reduce the number of concurrent requests or increase your limits. For more information, see [Usage information and limits](https://docs.wandb.ai/guides/inference/usage-limits/). |
| 429 | You exceeded your current quota, please check your plan and billing details | Out of credits or reached monthly spending cap. | Get more credits or increase your limits. For more information, see [Usage information and limits](https://docs.wandb.ai/guides/inference/usage-limits/). |
| 429 | W&B Inference isn't available for personal accounts. | Switch to a non-personal account. | Follow [the instructions below](#error-429-personal-entities-unsupported) for a work around. |
| 500 | The server had an error while processing your request | Internal server error. | Retry after a brief wait and contact support if it persists. |
| 503 | The engine is currently overloaded, please try again later | Server is experiencing high traffic. | Retry your request after a short delay. |
### Error 429: Personal entities unsupported
The user is on a personal account, which doesn't have access to W&B Inference. If one isn't available, create a Team to create a non-personal account.
Once done, add the `openai-project` header to your request as shown below:
```python
response = completion(
model="...",
extra_headers={"openai-project": "team_name/project_name"},
...
```
For more information, see [Personal entities unsupported](https://docs.wandb.ai/guides/inference/usage-limits/#personal-entities-unsupported).
You can find more ways of using custom headers with LiteLLM here - https://docs.litellm.ai/docs/proxy/request_headers.

17
docs/my-website/docs/proxy/admin_ui_sso.md
View file

@ -81,6 +81,23 @@ MICROSOFT_TENANT="5a39737
http://localhost:4000/sso/callback
```
**Using App Roles for User Permissions**
You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token and assign the corresponding role to the user.
Supported roles:
- `proxy_admin` - Admin over the platform
- `proxy_admin_viewer` - Can login, view all keys, view all spend (read-only)
- `internal_user` - Normal user. Can login, view spend and depending on team-member permissions - view/create/delete their own keys.
To set up app roles:
1. Navigate to your App Registration on https://portal.azure.com/
2. Go to "App roles" and create a new app role
3. Use one of the supported role names above (e.g., `proxy_admin`)
4. Assign users to these roles in your Enterprise Application
5. When users sign in via SSO, LiteLLM will automatically assign them the corresponding role
</TabItem>
<TabItem value="Generic" label="Generic SSO Provider">

2
docs/my-website/docs/proxy/caching.md
View file

@ -278,6 +278,8 @@ Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable cac
REDIS_HOST = "" # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
REDIS_PORT = "" # REDIS_PORT='18841'
REDIS_PASSWORD = "" # REDIS_PASSWORD='liteLlmIsAmazing'
REDIS_USERNAME = "" # REDIS_USERNAME='my-redis-username' [OPTIONAL] if your redis server requires a username
REDIS_SSL = "True" # REDIS_SSL='True' to enable SSL by default is False
```
**Additional kwargs**

11
docs/my-website/docs/proxy/config_settings.md
View file

@ -224,6 +224,7 @@ router_settings:
| service_account_settings | List[Dict[str, Any]] | Set `service_account_settings` if you want to create settings that only apply to service account keys (Doc on service accounts)[./service_accounts.md] |
| image_generation_model | str | The default model to use for image generation - ignores model set in request |
| store_model_in_db | boolean | If true, enables storing model + credential information in the DB. |
| supported_db_objects | List[str] | Fine-grained control over which object types to load from the database when `store_model_in_db` is True. Available types: `"models"`, `"mcp"`, `"guardrails"`, `"vector_stores"`, `"pass_through_endpoints"`, `"prompts"`, `"model_cost_map"`. If not set, all object types are loaded (default behavior). Example: `supported_db_objects: ["mcp"]` to only load MCP servers from DB. |
| store_prompts_in_spend_logs | boolean | If true, allows prompts and responses to be stored in the spend logs table. |
| max_request_size_mb | int | The maximum size for requests in MB. Requests above this size will be rejected. |
| max_response_size_mb | int | The maximum size for responses in MB. LLM Responses above this size will not be sent. |
@ -352,7 +353,10 @@ router_settings:
| AGENTOPS_SERVICE_NAME | Service Name for AgentOps logging integration
| AISPEND_ACCOUNT_ID | Account ID for AI Spend
| AISPEND_API_KEY | API Key for AI Spend
| AIOHTTP_CONNECTOR_LIMIT | Connection limit for aiohttp connector. When set to 0, no limit is applied. **Default is 0**
| AIOHTTP_KEEPALIVE_TIMEOUT | Keep-alive timeout for aiohttp connections in seconds. **Default is 120**
| AIOHTTP_TRUST_ENV | Flag to enable aiohttp trust environment. When this is set to True, aiohttp will respect HTTP(S)_PROXY env vars. **Default is False**
| AIOHTTP_TTL_DNS_CACHE | DNS cache time-to-live for aiohttp in seconds. **Default is 300**
| ALLOWED_EMAIL_DOMAINS | List of email domains allowed for access
| ARIZE_API_KEY | API key for Arize platform integration
| ARIZE_SPACE_KEY | Space key for Arize platform
@ -505,6 +509,8 @@ router_settings:
| EMAIL_SIGNATURE | Custom HTML footer/signature for all emails. Can include HTML tags for formatting and links.
| EMAIL_SUBJECT_INVITATION | Custom subject template for invitation emails.
| EMAIL_SUBJECT_KEY_CREATED | Custom subject template for key creation emails.
| ENKRYPTAI_API_BASE | Base URL for EnkryptAI Guardrails API. **Default is https://api.enkryptai.com**
| ENKRYPTAI_API_KEY | API key for EnkryptAI Guardrails service
| EXPERIMENTAL_MULTI_INSTANCE_RATE_LIMITING | Flag to enable new multi-instance rate limiting. **Default is False**
| FIREWORKS_AI_4_B | Size parameter for Fireworks AI 4B model. Default is 4
| FIREWORKS_AI_16_B | Size parameter for Fireworks AI 16B model. Default is 16
@ -628,6 +634,7 @@ router_settings:
| LITELLM_MODE | Operating mode for LiteLLM (e.g., production, development)
| LITELLM_RATE_LIMIT_WINDOW_SIZE | Rate limit window size for LiteLLM. Default is 60
| LITELLM_SALT_KEY | Salt key for encryption in LiteLLM
| LITELLM_SSL_CIPHERS | SSL/TLS cipher configuration for faster handshakes. Controls cipher suite preferences for OpenSSL connections.
| LITELLM_SECRET_AWS_KMS_LITELLM_LICENSE | AWS KMS encrypted license for LiteLLM
| LITELLM_TOKEN | Access token for LiteLLM integration
| LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD | If true, prints the standard logging payload to the console - useful for debugging
@ -775,4 +782,6 @@ router_settings:
| WEBHOOK_URL | URL for receiving webhooks from external services
| SPEND_LOG_RUN_LOOPS | Constant for setting how many runs of 1000 batch deletes should spend_log_cleanup task run |
| SPEND_LOG_CLEANUP_BATCH_SIZE | Number of logs deleted per batch during cleanup. Default is 1000 |
| COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY | Maximum size for CoroutineChecker in-memory cache. Default is 1000 |
| COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY | Maximum size for CoroutineChecker in-memory cache. Default is 1000 |
DEFAULT_SHARED_HEALTH_CHECK_TTL | Time-to-live in seconds for cached health check results in shared health check mode. Default is 300 (5 minutes)
DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL | Time-to-live in seconds for health check lock in shared health check mode. Default is 60 (1 minute)

3
docs/my-website/docs/proxy/dynamic_rate_limit.md
View file

@ -141,6 +141,7 @@ litellm_settings:
"dev": 0.1 # 10% reserved for development (1 RPM)
priority_reservation_settings:
default_priority: 0 # Weight (0%) assigned to keys without explicit priority metadata
saturation_threshold: 0.50 # A model is saturated if it has hit 50% of its RPM limit
general_settings:
master_key: sk-1234 # OR set `LITELLM_MASTER_KEY=".."` in your .env
@ -156,6 +157,8 @@ general_settings:
`priority_reservation_settings`: Object (Optional)
- **default_priority (float)**: Weight/percentage (0.0 to 1.0) assigned to API keys that have no priority metadata set (defaults to 0.5)
- **saturation_threshold (float)**: Saturation level (0.0 to 1.0) at which strict priority enforcement begins for a model. Saturation is calculated as `max(current_rpm/max_rpm, current_tpm/max_tpm)`. Below this threshold, generous mode allows priority borrowing from unused capacity. Above this threshold, strict mode enforces normalized priority limits.
- Example: When model usage is low, keys can use more than their allocated share. When model usage is high, keys are strictly limited to their allocated share.
**Start Proxy**

276
docs/my-website/docs/proxy/guardrails/enkryptai.md Normal file
View file

@ -0,0 +1,276 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# EnkryptAI Guardrails
LiteLLM supports EnkryptAI guardrails for content moderation and safety checks on LLM inputs and outputs.
## Quick Start
### 1. Define Guardrails on your LiteLLM config.yaml
Define your guardrails under the `guardrails` section:
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "enkryptai-guard"
litellm_params:
guardrail: enkryptai
mode: "pre_call"
api_key: os.environ/ENKRYPTAI_API_KEY
detectors:
toxicity:
enabled: true
nsfw:
enabled: true
pii:
enabled: true
entities: ["email", "phone", "secrets"]
injection_attack:
enabled: true
```
#### Supported values for `mode`
- `pre_call` - Run **before** LLM call, on **input**
- `post_call` - Run **after** LLM call, on **output**
- `during_call` - Run **during** LLM call, on **input**. Same as `pre_call` but runs in parallel as LLM call
#### Available Detectors
EnkryptAI supports multiple content detection types:
- **toxicity** - Detect toxic language
- **nsfw** - Detect NSFW (Not Safe For Work) content
- **pii** - Detect personally identifiable information
- Configure entities: `["pii", "email", "phone", "secrets", "ip_address", "url"]`
- **injection_attack** - Detect prompt injection attempts
- **keyword_detector** - Detect custom keywords/phrases
- **policy_violation** - Detect policy violations
- **bias** - Detect biased content
- **sponge_attack** - Detect sponge attacks
### 2. Set Environment Variables
```bash
export ENKRYPTAI_API_KEY="your-api-key"
```
### 3. Start LiteLLM Gateway
```shell
litellm --config config.yaml --detailed_debug
```
### 4. Test Request
**[Langchain, OpenAI SDK Usage Examples](../proxy/user_keys#request-format)**
<Tabs>
<TabItem label="Successful Call" value="allowed">
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "Hello, how can you help me today?"}
],
"guardrails": ["enkryptai-guard"]
}'
```
**Response: HTTP 200 Success**
Content passes all detector checks and is allowed through.
</TabItem>
<TabItem label="Unsuccessful Call" value="not-allowed">
Expect this to fail if content violates detector policies:
```shell
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "My email is test@example.com and my SSN is 123-45-6789"}
],
"guardrails": ["enkryptai-guard"]
}'
```
**Expected Response on Failure: HTTP 400 Error**
```json
{
"error": {
"message": {
"error": "Content blocked by EnkryptAI guardrail",
"detected": true,
"violations": ["pii"],
"response": {
"summary": {
"pii": 1
},
"details": {
"pii": {
"detected": ["email", "ssn"]
}
}
}
},
"type": "None",
"param": "None",
"code": "400"
}
}
```
</TabItem>
</Tabs>
## Video Walkthrough
<iframe width="840" height="500" src="https://www.loom.com/embed/ff222211e0864937aee4aeef0f28c3b7" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
## Advanced Configuration
### Using Custom Policies
You can specify a custom EnkryptAI policy:
```yaml
guardrails:
- guardrail_name: "enkryptai-custom"
litellm_params:
guardrail: enkryptai
mode: "pre_call"
api_key: os.environ/ENKRYPTAI_API_KEY
policy_name: "my-custom-policy" # Sent via x-enkrypt-policy header
detectors:
toxicity:
enabled: true
```
### Using Deployments
Specify an EnkryptAI deployment:
```yaml
guardrails:
- guardrail_name: "enkryptai-deployment"
litellm_params:
guardrail: enkryptai
mode: "pre_call"
api_key: os.environ/ENKRYPTAI_API_KEY
deployment_name: "production" # Sent via X-Enkrypt-Deployment header
detectors:
toxicity:
enabled: true
```
### Monitor Mode (Logging Without Blocking)
Set `block_on_violation: false` to log violations without blocking requests:
```yaml
guardrails:
- guardrail_name: "enkryptai-monitor"
litellm_params:
guardrail: enkryptai
mode: "pre_call"
api_key: os.environ/ENKRYPTAI_API_KEY
block_on_violation: false # Log violations but don't block
detectors:
toxicity:
enabled: true
nsfw:
enabled: true
```
In monitor mode, all violations are logged but requests are never blocked.
### Input and Output Guardrails
Configure separate guardrails for input and output:
```yaml
guardrails:
# Input guardrail
- guardrail_name: "enkryptai-input"
litellm_params:
guardrail: enkryptai
mode: "pre_call"
api_key: os.environ/ENKRYPTAI_API_KEY
detectors:
pii:
enabled: true
entities: ["email", "phone", "ssn"]
injection_attack:
enabled: true
# Output guardrail
- guardrail_name: "enkryptai-output"
litellm_params:
guardrail: enkryptai
mode: "post_call"
api_key: os.environ/ENKRYPTAI_API_KEY
detectors:
toxicity:
enabled: true
nsfw:
enabled: true
```
## Configuration Options
| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `api_key` | string | EnkryptAI API key | `ENKRYPTAI_API_KEY` env var |
| `api_base` | string | EnkryptAI API base URL | `https://api.enkryptai.com` |
| `policy_name` | string | Custom policy name (sent via `x-enkrypt-policy` header) | None |
| `deployment_name` | string | Deployment name (sent via `X-Enkrypt-Deployment` header) | None |
| `detectors` | object | Detector configuration | `{}` |
| `block_on_violation` | boolean | Block requests on violations | `true` |
| `mode` | string | When to run: `pre_call`, `post_call`, or `during_call` | Required |
## Observability
EnkryptAI guardrail logs include:
- **guardrail_status**: `success`, `guardrail_intervened`, or `guardrail_failed_to_respond`
- **guardrail_provider**: `enkryptai`
- **guardrail_json_response**: Full API response with detection details
- **duration**: Time taken for guardrail check
- **start_time** and **end_time**: Timestamps
These logs are available through your configured LiteLLM logging callbacks.
## Error Handling
The guardrail handles errors gracefully:
- **API Failures**: Logs error and raises exception
- **Rate Limits (429)**: Logs error and raises exception
- **Invalid Configuration**: Raises `ValueError` on initialization
Set `block_on_violation: false` to continue processing even when violations are detected (monitor mode).
## Support
For more information about EnkryptAI:
- Documentation: [https://docs.enkryptai.com](https://docs.enkryptai.com)
- Website: [https://enkryptai.com](https://enkryptai.com)

21
docs/my-website/docs/proxy/health.md
View file

@ -9,13 +9,32 @@ Use this to health check all LLMs defined in your config.yaml
| `/health/readiness` | **Load balancer health checks** | Ready to accept traffic - includes DB connection status |
| `/health` | **Model health monitoring** | Comprehensive LLM model health - makes actual API calls |
| `/health/services` | **Service debugging** | Check specific integrations (datadog, langfuse, etc.) |
| `/health/shared-status` | **Multi-pod coordination** | Monitor shared health check state across pods |
## Summary
The proxy exposes:
* a /health endpoint which returns the health of the LLM APIs
* a /health/readiness endpoint for returning if the proxy is ready to accept requests
* a /health/liveliness endpoint for returning if the proxy is alive
* a /health/liveliness endpoint for returning if the proxy is alive
* a /health/shared-status endpoint for monitoring shared health check coordination across pods
## Shared Health Check State
When running multiple LiteLLM proxy pods, you can enable shared health check state to coordinate health checks across pods and avoid duplicate API calls. This is especially beneficial for expensive models like Gemini 2.5-pro.
**Key Benefits:**
- Reduces duplicate health checks across pods
- Saves costs on expensive model API calls
- Reduces monitoring noise and logging
- Improves resource efficiency
**Requirements:**
- Redis for shared state coordination
- Background health checks enabled
- Multiple proxy pods
For detailed configuration and usage, see [Shared Health Check State](./shared_health_check.md).
## `/health`
#### Request

354
docs/my-website/docs/proxy/security_encryption_faq.md Normal file
View file

@ -0,0 +1,354 @@
# LiteLLM Self-Hosted Security & Encryption FAQ
## Data in Transit Encryption
### Does the product encrypt data in transit?
**Yes**, LiteLLM encrypts data in transit using TLS/SSL.
### Available in both OSS and Enterprise?
**Yes**, TLS encryption is available in both Open Source and Enterprise versions.
### In transit between the calling client and the product?
**Yes**, HTTPS/TLS is supported through SSL certificate configuration.
**Configuration:**
```bash
# CLI
litellm --ssl_keyfile_path /path/to/key.pem --ssl_certfile_path /path/to/cert.pem
# Environment Variables
export SSL_KEYFILE_PATH="/path/to/key.pem"
export SSL_CERTFILE_PATH="/path/to/cert.pem"
```
**Documentation Reference:** `docs/my-website/docs/guides/security_settings.md`
### In transit between the product and the LLM providers?
**Yes**, all connections to LLM providers use TLS encryption by default.
**Implementation Details:**
- Uses Python's `ssl.create_default_context()`
- Leverages HTTPX and aiohttp libraries with SSL/TLS enabled
- Uses certifi CA bundle by default for SSL verification
**Code Reference:** `litellm/llms/custom_httpx/http_handler.py` (lines 43-105)
### Are TCP sessions to the LLM providers shared?
**Yes**, TCP connections are pooled and reused.
**Details:**
- Connection pooling is enabled by default
- Default: 1000 max concurrent connections with keepalive
- Sessions are maintained across requests to the same provider
- Reduces overhead of TLS handshakes
**Code Reference:** `litellm/llms/custom_httpx/http_handler.py` (lines 704-712)
### Or does the product negotiate a new TLS session with the same LLM provider for every sequential call?
**No**, TLS sessions are reused through connection pooling. New TLS handshakes are not performed for every request.
### How is it encrypted?
**TLS 1.2 and TLS 1.3**
Uses Python's default SSL context which supports both TLS 1.2 and TLS 1.3. The specific version negotiated depends on:
- Python version
- System SSL library (typically OpenSSL)
- Server capabilities
**Implementation:** `ssl.create_default_context()` in Python
### How are these added to the product's configuration?
#### x.509 Certificate
**Method 1: CLI Arguments**
```bash
litellm --ssl_certfile_path /path/to/certificate.pem
```
**Method 2: Environment Variable**
```bash
export SSL_CERTFILE_PATH="/path/to/certificate.pem"
```
#### Private Key
**Method 1: CLI Arguments**
```bash
litellm --ssl_keyfile_path /path/to/private_key.pem
```
**Method 2: Environment Variable**
```bash
export SSL_KEYFILE_PATH="/path/to/private_key.pem"
```
#### Certificate Bundle/Chain
**For client-to-proxy connections:**
Use standard SSL certificate setup with intermediate certificates bundled in the certfile.
**For proxy-to-LLM provider connections:**
**Method 1: Config YAML**
```yaml
litellm_settings:
ssl_verify: "/path/to/ca_bundle.pem"
```
**Method 2: Environment Variable**
```bash
export SSL_CERT_FILE="/path/to/ca_bundle.pem"
```
**Method 3: Client Certificate Authentication**
```yaml
litellm_settings:
ssl_certificate: "/path/to/client_certificate.pem"
```
or
```bash
export SSL_CERTIFICATE="/path/to/client_certificate.pem"
```
### Documentation Coverage
**Primary Documentation:**
- `docs/my-website/docs/guides/security_settings.md` - SSL/TLS configuration guide
**Additional References:**
- `litellm/proxy/proxy_cli.py` (lines 455-467) - CLI options
- `docs/my-website/docs/completion/http_handler_config.md` - Custom HTTP handler configuration
---
## Data at Rest Encryption
### Does the product encrypt data at rest?
**Partially**. Only specific sensitive data is encrypted at rest.
### What data is stored in encrypted form?
#### Encrypted Data:
1. **LLM API Keys** - Model credentials in `LiteLLM_ProxyModelTable.litellm_params`
2. **Provider Credentials** - Stored in `LiteLLM_CredentialsTable.credential_values`
3. **Configuration Secrets** - Sensitive config values in `LiteLLM_Config` table
4. **Virtual Keys** - When using secret managers (optional feature)
#### NOT Encrypted:
1. **Spend Logs** - Request/response data in `LiteLLM_SpendLogs`
2. **Audit Logs** - Change history in `LiteLLM_AuditLog`
3. **User/Team/Organization Data** - Metadata and configuration
4. **Cached Prompts and Completions** - Cache data is stored in plaintext
### Cached prompts and completions?
**No**, cached prompts and completions are **NOT encrypted**.
Cache backends (Redis, S3, local disk) store data as plaintext JSON.
**Code References:**
- `litellm/caching/redis_cache.py`
- `litellm/caching/s3_cache.py`
- `litellm/caching/caching.py`
### Configuration data?
**Partially encrypted**.
#### What IS Encrypted:
- LLM API keys and credentials in model configurations
- Sensitive values in `LiteLLM_Config` table
- Credential values in `LiteLLM_CredentialsTable`
#### What is NOT Encrypted:
- Model names and aliases
- Rate limits and budget settings
- User/team/organization metadata
- Non-sensitive configuration parameters
**Code Reference:** `litellm/proxy/management_endpoints/model_management_endpoints.py` (lines 275-308)
### Log data?
**No**, log data is **NOT encrypted**.
Log data stored in database tables is in plaintext:
- `LiteLLM_SpendLogs` - Contains request/response data, tokens, spend
- `LiteLLM_ErrorLogs` - Error information
- `LiteLLM_AuditLog` - Audit trail of changes
**Note:** You can disable logging to avoid storing sensitive data:
```yaml
general_settings:
disable_spend_logs: True # Disable writing spend logs to DB
disable_error_logs: True # Disable writing error logs to DB
```
**Documentation:** `docs/my-website/docs/proxy/db_info.md` (lines 52-60)
### Where is it stored?
#### In the DB?
**Yes**, encrypted data is stored in PostgreSQL database.
**Key Tables with Encrypted Data:**
- `LiteLLM_ProxyModelTable` - Model configurations with encrypted API keys
- `LiteLLM_CredentialsTable` - Credential values
- `LiteLLM_Config` - Configuration secrets
**Schema Reference:** `schema.prisma`
#### In the filesystem?
**No**, encrypted data is not stored in the filesystem by default.
**Note:** If using disk cache (`disk_cache_dir`), cached data is stored unencrypted.
#### Somewhere else?
**Optional:** When using secret managers (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault), encrypted data can be stored externally.
**Configuration:**
```yaml
general_settings:
key_management_system: "aws_secret_manager" # or "azure_key_vault", "hashicorp_vault"
```
**Documentation:** `docs/my-website/docs/secret.md`
### How is it encrypted?
**Algorithm:** NaCl SecretBox (XSalsa20-Poly1305 AEAD)
**NOT AES-256** - LiteLLM uses NaCl (Networking and Cryptography Library) which provides:
- XSalsa20 stream cipher
- Poly1305 MAC for authentication
- Equivalent security to AES-256
**Key Derivation:**
1. Takes `LITELLM_SALT_KEY` (or `LITELLM_MASTER_KEY` if salt key not set)
2. Hashes with SHA-256 to derive 256-bit encryption key
3. Uses NaCl SecretBox for authenticated encryption
**Code Reference:** `litellm/proxy/common_utils/encrypt_decrypt_utils.py` (lines 69-112)
**Implementation:**
```python
import hashlib
import nacl.secret
# Derive 256-bit key from salt
hash_object = hashlib.sha256(signing_key.encode())
hash_bytes = hash_object.digest()
# Create SecretBox and encrypt
box = nacl.secret.SecretBox(hash_bytes)
encrypted = box.encrypt(value_bytes)
```
### Setting the Encryption Key
**Required Environment Variable:**
```bash
export LITELLM_SALT_KEY="your-strong-random-key-here"
```
**Important Notes:**
- ⚠️ **Must be set before adding any models**
- ⚠️ **Never change this key** - encrypted data becomes unrecoverable
- ⚠️ Use a strong random key (recommended: https://1password.com/password-generator/)
- If not set, falls back to `LITELLM_MASTER_KEY`
**Documentation:** `docs/my-website/docs/proxy/prod.md` (section 8, lines 184-196)
### Documentation Coverage
**Primary Documentation:**
- `docs/my-website/docs/proxy/prod.md` (section 8) - LITELLM_SALT_KEY setup
- `docs/my-website/docs/secret.md` - Secret management systems
- `docs/my-website/docs/proxy/db_info.md` - Database information
**Additional References:**
- `security.md` - General security measures
- `docs/my-website/docs/data_security.md` - Data privacy overview
- `schema.prisma` - Database schema with encrypted fields
---
## Summary of Security Features
### ✅ Provided Out of the Box
1. **TLS/SSL encryption** for client-to-proxy connections
2. **TLS encryption** for proxy-to-LLM provider connections (with connection pooling)
3. **Encrypted storage** of LLM API keys and credentials
4. **Support for TLS 1.2 and TLS 1.3**
5. **Connection pooling** to reduce TLS handshake overhead
### ⚠️ Important Limitations
1. **Cached data is NOT encrypted** (Redis, S3, disk cache)
2. **Log data is NOT encrypted** (spend logs, audit logs)
3. **Request/response payloads in logs are NOT encrypted**
4. **Uses NaCl SecretBox, NOT AES-256** (equivalent security)
5. **TLS version not explicitly configured** - uses Python/system defaults
### 🔧 Configuration Requirements
**For Production Deployments:**
1. **Set LITELLM_SALT_KEY** before adding any models
2. **Configure SSL certificates** for HTTPS client connections
3. **Consider disabling logs** if they contain sensitive data
4. **Use secret managers** for enhanced security (optional)
5. **Configure CA bundles** if using custom certificates
---
## Quick Start Security Checklist
```bash
# 1. Generate a strong salt key
export LITELLM_SALT_KEY="$(openssl rand -base64 32)"
# 2. Set up SSL certificates (for HTTPS)
export SSL_KEYFILE_PATH="/path/to/private_key.pem"
export SSL_CERTFILE_PATH="/path/to/certificate.pem"
# 3. Configure database
export DATABASE_URL="postgresql://user:password@host:port/dbname"
# 4. (Optional) Disable logs if they contain sensitive data
# Add to config.yaml:
# general_settings:
# disable_spend_logs: True
# disable_error_logs: True
# 5. Start LiteLLM Proxy
litellm --config config.yaml
```
---
## Additional Resources
- **LiteLLM Documentation:** https://docs.litellm.ai/
- **Security Settings Guide:** https://docs.litellm.ai/docs/guides/security_settings
- **Production Deployment:** https://docs.litellm.ai/docs/proxy/prod
- **Secret Management:** https://docs.litellm.ai/docs/secret
For security inquiries: support@berri.ai

310
docs/my-website/docs/proxy/shared_health_check.md Normal file
View file

@ -0,0 +1,310 @@
# Shared Health Check State Across Pods
This feature enables coordination of health checks across multiple LiteLLM proxy pods to avoid duplicate health checks and reduce costs.
## Overview
When running multiple LiteLLM proxy pods (e.g., in Kubernetes), each pod typically runs its own independent health checks on every model. This can result in:
- **Duplicate health checks** across pods
- **Increased costs** for expensive models (e.g., Gemini 2.5-pro)
- **Redundant monitoring/logging noise**
- **Inefficient resource usage**
The shared health check state feature solves this by:
- **Coordinating health checks** across pods using Redis
- **Caching results** with configurable TTL
- **Using distributed locks** to ensure only one pod runs health checks at a time
- **Allowing other pods** to read cached results instead of running redundant checks
## How It Works
### 1. Lock Acquisition
When a pod needs to run health checks:
- It attempts to acquire a Redis lock
- If successful, it runs the health checks
- If failed, it waits briefly and checks for cached results
### 2. Result Caching
After running health checks:
- Results are cached in Redis with a configurable TTL
- Other pods can read these cached results
- Cache includes timestamp and pod ID for tracking
### 3. Fallback Behavior
If Redis is unavailable or cache is expired:
- Pods fall back to running health checks locally
- System continues to function normally
## Configuration
### Enable Shared Health Check
Add to your `proxy_config.yaml`:
```yaml
general_settings:
# Enable background health checks (required)
background_health_checks: true
# Enable shared health check state across pods
use_shared_health_check: true
# Health check interval (seconds)
health_check_interval: 300 # 5 minutes
# Redis configuration (required for shared health check)
litellm_settings:
cache: true
cache_params:
type: redis
host: your-redis-host
port: 6379
password: your-redis-password
```
### Environment Variables
You can also configure using environment variables:
```bash
# Enable shared health check
export USE_SHARED_HEALTH_CHECK=true
# Health check TTL (seconds)
export DEFAULT_SHARED_HEALTH_CHECK_TTL=300
# Lock TTL (seconds)
export DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL=60
```
## Requirements
- **Redis**: Required for shared state coordination
- **Background Health Checks**: Must be enabled (`background_health_checks: true`)
- **Multiple Pods**: Most beneficial with 2+ proxy instances
## API Endpoints
### Check Shared Health Check Status
```bash
GET /health/shared-status
```
Returns information about the shared health check coordination:
```json
{
"shared_health_check_enabled": true,
"status": {
"pod_id": "pod_1703123456789",
"redis_available": true,
"lock_ttl": 60,
"cache_ttl": 300,
"lock_owner": "pod_1703123456788",
"lock_in_progress": true,
"cache_available": true,
"cache_age_seconds": 45.2,
"last_checked_by": "pod_1703123456788"
}
}
```
## Monitoring
### Health Check Status
Monitor the shared health check status to ensure proper coordination:
```bash
curl -H "Authorization: Bearer your-api-key" \
http://your-proxy-host/health/shared-status
```
### Logs
Look for these log messages:
```
INFO: Initialized shared health check manager
INFO: Pod pod_123 acquired health check lock
INFO: Pod pod_123 released health check lock
INFO: Cached health check results for 5 healthy and 0 unhealthy endpoints
DEBUG: Using cached health check results
```
## Troubleshooting
### Common Issues
#### 1. Shared Health Check Not Working
**Symptoms**: Each pod still runs independent health checks
**Solutions**:
- Verify Redis is configured and accessible
- Check that `use_shared_health_check: true` is set
- Ensure `background_health_checks: true` is enabled
- Check Redis connectivity in logs
#### 2. Redis Connection Issues
**Symptoms**: Health checks fall back to local execution
**Solutions**:
- Verify Redis host, port, and credentials
- Check network connectivity between pods and Redis
- Monitor Redis server logs for errors
#### 3. Lock Not Released
**Symptoms**: One pod holds the lock indefinitely
**Solutions**:
- Lock has automatic TTL (default 60 seconds)
- Check pod logs for lock release messages
- Verify Redis TTL settings
### Debug Mode
Enable debug logging to see detailed coordination:
```yaml
general_settings:
set_verbose: true
```
## Performance Impact
### Benefits
- **Reduced API calls**: Only one pod runs health checks per interval
- **Lower costs**: Especially significant for expensive models
- **Better resource utilization**: Less redundant work across pods
- **Cleaner monitoring**: Reduced noise in logs and metrics
### Overhead
- **Redis operations**: Minimal overhead for lock/cache operations
- **Network latency**: Small delay for Redis communication
- **Memory usage**: Negligible additional memory usage
## Best Practices
### 1. Redis Configuration
- Use Redis with persistence enabled
- Configure appropriate memory limits
- Set up Redis monitoring and alerts
### 2. TTL Settings
- Set `health_check_interval` to your desired check frequency
- Use default TTL values unless you have specific requirements
- Consider model-specific timeouts for expensive models
### 3. Monitoring
- Monitor shared health check status endpoint
- Set up alerts for Redis connectivity issues
- Track health check costs and frequency
### 4. Scaling
- Feature works with any number of pods
- More pods = better coordination benefits
- Consider Redis cluster for high availability
## Example Configuration
### Complete Example
```yaml
# proxy_config.yaml
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
model_info:
health_check_timeout: 30 # 30 second timeout for health checks
general_settings:
# Enable background health checks
background_health_checks: true
# Enable shared health check coordination
use_shared_health_check: true
# Health check interval (5 minutes)
health_check_interval: 300
# Health check details
health_check_details: true
litellm_settings:
# Redis configuration
cache: true
cache_params:
type: redis
host: redis-cluster.example.com
port: 6379
password: os.environ/REDIS_PASSWORD
ssl: true
```
### Kubernetes Example
```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: litellm-proxy
spec:
replicas: 3 # Multiple pods for coordination
template:
spec:
containers:
- name: litellm-proxy
image: ghcr.io/berriai/litellm:latest
env:
- name: USE_SHARED_HEALTH_CHECK
value: "true"
- name: REDIS_HOST
value: "redis-service"
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-secret
key: password
```
## Migration
### From Independent Health Checks
1. **Enable Redis**: Ensure Redis is configured and accessible
2. **Enable Background Health Checks**: Set `background_health_checks: true`
3. **Enable Shared Health Check**: Set `use_shared_health_check: true`
4. **Deploy**: Update your proxy configuration
5. **Monitor**: Check `/health/shared-status` endpoint
### Rollback
To disable shared health check:
```yaml
general_settings:
use_shared_health_check: false
# background_health_checks can remain true for independent checks
```
## Related Features
- [Background Health Checks](./health.md#background-health-checks)
- [Redis Caching](./caching.md)
- [High Availability Setup](./db_deadlocks.md)
- [Health Check Endpoints](./health.md#health-endpoints)

277
docs/my-website/docs/proxy/tag_budgets.md Normal file
View file

@ -0,0 +1,277 @@
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Setting Tag Budgets
Track spend and set budgets for your API requests using tags. Tags allow you to categorize and monitor costs across different cost centers, projects, and departments.
## Pre-Requisites
- You must set up a Postgres database (e.g. Supabase, Neon, etc.)
## What are Tags?
Tags are labels you can attach to your LLM requests to track and limit spending by category.
**Common Use Cases:**
- **Cost Center Tracking**: Allocate LLM costs to specific departments or business units (e.g., "engineering", "marketing", "customer-support")
- **Project-based Budgeting**: Set budgets for different projects or initiatives (e.g., "project-alpha", "chatbot-v2")
- **Customer Attribution**: Track spend per customer or client (e.g., "customer-acme", "customer-techcorp")
- **Feature Monitoring**: Monitor costs for specific features (e.g., "feature-chat", "feature-summarization")
Tags are added to each request in the `metadata` field to track and enforce budget limits.
## Setting Tag Budgets
### 1. Create a tag with budget
Create a tag to represent a cost center, project, or any budget category. Set `max_budget` ($ value allowed) and `budget_duration` (how frequently the budget resets).
**Example:** Create a tag for your Engineering department with a monthly $500 budget
#### API
Create a new tag and set `max_budget` and `budget_duration`
```shell
curl -X POST 'http://0.0.0.0:4000/tag/new' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"name": "engineering",
"description": "Engineering department cost center",
"max_budget": 500.0,
"budget_duration": "30d"
}'
```
**Request Body Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `name` | string | Yes | Unique name for the tag (e.g., cost center name) |
| `description` | string | No | Description of what this tag tracks |
| `models` | list[string] | No | Restrict tag to specific models |
| `max_budget` | float | No | Maximum budget in USD |
| `budget_duration` | string | No | How often budget resets (e.g., "30d", "1d") |
| `soft_budget` | float | No | Soft budget limit for warnings |
**Response:**
```json
{
"name": "engineering",
"description": "Engineering department cost center",
"max_budget": 500.0,
"budget_duration": "30d",
"budget_reset_at": "2025-11-10T00:00:00Z",
"created_at": "2025-10-11T00:00:00Z"
}
```
#### LiteLLM Admin UI
Navigate to the **Tag Management** page and click **Create New Tag**. Fill in the tag details and set your budget:
<Image
img={require('../../img/tag_budget1.png')}
style={{width: '80%', display: 'block', margin: '0'}}
/>
<br />
**Possible values for `budget_duration`:**
| `budget_duration` | When Budget will reset |
| --- | --- |
| `budget_duration="1s"` | every 1 second |
| `budget_duration="1m"` | every 1 minute |
| `budget_duration="1h"` | every 1 hour |
| `budget_duration="1d"` | every 1 day |
| `budget_duration="7d"` | every 1 week |
| `budget_duration="30d"` | every 1 month |
### 2. Use the tag in your requests
Add tags to your API requests in the `metadata` field:
:::info Tags Budgets on API Keys
Currently, tag budget enforcement is only supported per request. If you'd like to set tags on API keys so all requests automatically inherit the tags budgets, please [create a feature request on GitHub](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeat%5D%3A).
:::
<Tabs>
<TabItem value="openai" label="OpenAI SDK">
```python
import openai
client = openai.OpenAI(
api_key="sk-1234", # Your LiteLLM proxy key
base_url="http://0.0.0.0:4000"
)
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}],
extra_body={
"metadata": {
"tags": ["engineering"]
}
}
)
```
</TabItem>
<TabItem value="curl" label="cURL">
```shell
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}],
"metadata": {
"tags": ["engineering"]
}
}'
```
</TabItem>
</Tabs>
### 3. Test It
Make requests until the budget is exceeded:
```shell
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}],
"metadata": {
"tags": ["engineering"]
}
}'
```
**When budget is exceeded, you'll see:**
```json
{
"error": {
"message": "Budget has been exceeded! Tag=engineering Current cost: 505.50, Max budget: 500.0",
"type": "budget_exceeded",
"param": null,
"code": "400"
}
}
```
## Managing Tags
### View Tag Information
Get information about specific tags:
```shell
curl -X POST 'http://0.0.0.0:4000/tag/info' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"names": ["engineering", "marketing"]
}'
```
**Response:**
```json
{
"engineering": {
"name": "engineering",
"description": "Engineering department cost center",
"spend": 245.50,
"max_budget": 500.0,
"budget_duration": "30d",
"budget_reset_at": "2025-11-10T00:00:00Z",
"created_at": "2025-10-11T00:00:00Z",
"updated_at": "2025-10-11T12:30:00Z"
},
"marketing": {
"name": "marketing",
"description": "Marketing department cost center",
"spend": 89.20,
"max_budget": 300.0,
"budget_duration": "30d",
"budget_reset_at": "2025-11-10T00:00:00Z",
"created_at": "2025-10-11T00:00:00Z",
"updated_at": "2025-10-11T12:30:00Z"
}
}
```
### Update Tag Budget
Update an existing tag's budget:
```shell
curl -X POST 'http://0.0.0.0:4000/tag/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"name": "engineering",
"max_budget": 750.0,
"budget_duration": "30d"
}'
```
### Delete Tag
```shell
curl -X POST 'http://0.0.0.0:4000/tag/delete' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"name": "engineering"
}'
```
## Multiple Tags per Request
You can apply multiple tags to a single request to track costs across different dimensions simultaneously. For example, track both the cost center and the specific project:
```python
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}],
extra_body={
"metadata": {
"tags": ["engineering", "project-alpha", "customer-acme"]
}
}
)
```
```shell
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}],
"metadata": {
"tags": ["engineering", "project-alpha", "customer-acme"]
}
}'
```
**Budget Enforcement:** If any tag exceeds its budget, the request will be rejected.

50
docs/my-website/docs/response_api.md
View file

@ -14,6 +14,7 @@ Requests to /chat/completions may be bridged here automatically when the provide
| Logging | ✅ | Works across all integrations |
| End-user Tracking | ✅ | |
| Streaming | ✅ | |
| Image Generation Streaming | ✅ | Progressive image generation with partial images (1-3) |
| Fallbacks | ✅ | Works between supported models |
| Loadbalancing | ✅ | Works between supported models |
| Supported operations | Create a response, Get a response, Delete a response | |
@ -56,6 +57,29 @@ for event in response:
print(event)
```
#### Image Generation with Streaming
```python showLineNumbers title="OpenAI Streaming Image Generation"
import litellm
import base64
# Streaming image generation with partial images
stream = litellm.responses(
model="gpt-4.1", # Use an actual image generation model
input="Generate a gorgeous image of a river made of white owl feathers",
stream=True,
tools=[{"type": "image_generation", "partial_images": 2}],
)
for event in stream:
if event.type == "response.image_generation_call.partial_image":
idx = event.partial_image_index
image_base64 = event.partial_image_b64
image_bytes = base64.b64decode(image_base64)
with open(f"river{idx}.png", "wb") as f:
f.write(image_bytes)
```
#### GET a Response
```python showLineNumbers title="Get Response by ID"
import litellm
@ -380,6 +404,32 @@ for event in response:
print(event)
```
#### Image Generation with Streaming
```python showLineNumbers title="OpenAI Proxy Streaming Image Generation"
from openai import OpenAI
import base64
client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000")
stream = client.responses.create(
model="gpt-4.1",
input="Draw a gorgeous image of a river made of white owl feathers, snaking its way through a serene winter landscape",
stream=True,
tools=[{"type": "image_generation", "partial_images": 2}],
)
for event in stream:
print(f"event: {event}")
if event.type == "response.image_generation_call.partial_image":
idx = event.partial_image_index
image_base64 = event.partial_image_b64
image_bytes = base64.b64decode(image_base64)
with open(f"river{idx}.png", "wb") as f:
f.write(image_bytes)
```
#### GET a Response
```python showLineNumbers title="Get Response by ID with OpenAI SDK"
from openai import OpenAI

48
docs/my-website/docs/tutorials/msft_sso.md
View file

@ -140,6 +140,54 @@ litellm_settings:
<Image img={require('../../img/msft_default_settings.png')} style={{ width: '900px', height: 'auto' }} />
## 4. Using Entra ID App Roles for User Permissions
You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token during SSO sign-in and assign the corresponding role to the user.
### 4.1 Supported Roles
LiteLLM supports the following app roles (case-insensitive):
- `proxy_admin` - Admin over the entire LiteLLM platform
- `proxy_admin_viewer` - Read-only admin access (can view all keys and spend)
- `org_admin` - Admin over a specific organization (can create teams and users within their org)
- `internal_user` - Standard user (can create/view/delete their own keys and view their own spend)
### 4.2 Create App Roles in Entra ID
1. Navigate to your App Registration on https://portal.azure.com/
2. Go to **App roles** > **Create app role**
3. Configure the app role:
- **Display name**: Proxy Admin (or your preferred display name)
- **Value**: `proxy_admin` (use one of the supported role values above)
- **Description**: Administrator access to LiteLLM proxy
- **Allowed member types**: Users/Groups
4. Click **Apply** to save the role
### 4.3 Assign Users to App Roles
1. Navigate to **Enterprise Applications** on https://portal.azure.com/
2. Select your LiteLLM application
3. Go to **Users and groups** > **Add user/group**
4. Select the user and assign them to one of the app roles you created
### 4.4 Test the Role Assignment
1. Sign in to LiteLLM UI via SSO as a user with an assigned app role
2. LiteLLM will automatically extract the app role from the JWT token
3. The user will be assigned the corresponding LiteLLM role in the database
4. The user's permissions will reflect their assigned role
**How it works:**
- When a user signs in via Microsoft SSO, LiteLLM extracts the `roles` claim from the JWT `id_token`
- If any of the roles match a valid LiteLLM role (case-insensitive), that role is assigned to the user
- If multiple roles are present, LiteLLM uses the first valid role it finds
- This role assignment persists in the LiteLLM database and determines the user's access level
## Video Walkthrough
This walks through setting up sso auto-add for **Microsoft Entra ID**

BIN
docs/my-website/img/tag_budget1.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 305 KiB

BIN
docs/my-website/img/tag_budget2.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 416 KiB

35
docs/my-website/sidebars.js
View file

@ -36,6 +36,7 @@ const sidebars = {
"proxy/guardrails/aporia_api",
"proxy/guardrails/azure_content_guardrail",
"proxy/guardrails/bedrock",
"proxy/guardrails/enkryptai",
"proxy/guardrails/lasso_security",
"proxy/guardrails/guardrails_ai",
"proxy/guardrails/lakera_ai",
@ -188,12 +189,13 @@ const sidebars = {
type: "category",
label: "Budgets + Rate Limits",
items: [
"proxy/users",
"proxy/team_budgets",
"proxy/tag_budgets",
"proxy/customers",
"proxy/dynamic_rate_limit",
"proxy/rate_limit_tiers",
"proxy/team_budgets",
"proxy/temporary_budget_increase",
"proxy/users"
],
},
"proxy/caching",
@ -333,7 +335,17 @@ const sidebars = {
"image_variations",
]
},
"mcp",
{
type: "category",
label: "/mcp - Model Context Protocol",
items: [
"mcp",
"mcp_usage",
"mcp_control",
"mcp_cost",
"mcp_guardrail",
]
},
"moderation",
{
type: "category",
@ -412,6 +424,7 @@ const sidebars = {
items: [
"providers/vertex",
"providers/vertex_partner",
"providers/vertex_self_deployed",
"providers/vertex_image",
"providers/vertex_batch",
]
@ -522,22 +535,17 @@ const sidebars = {
"providers/oci",
"providers/datarobot",
"providers/ovhcloud",
"providers/wandb_inference",
],
},
{
type: "category",
label: "Guides",
items: [
{
type: "category",
label: "Tools",
items: [
"completion/computer_use",
"completion/web_search",
"completion/web_fetch",
"completion/function_call",
]
},
"completion/computer_use",
"completion/web_search",
"completion/web_fetch",
"completion/function_call",
"completion/audio",
"completion/document_understanding",
"completion/drop_params",
@ -674,6 +682,7 @@ const sidebars = {
items: [
"data_security",
"data_retention",
"proxy/security_encryption_faq",
"migration_policy",
{
type: "category",

1
docs/my-website/src/pages/completion/supported.md
View file

@ -8,6 +8,7 @@
| gpt-3.5-turbo-16k | `completion('gpt-3.5-turbo-16k', messages)` | `os.environ['OPENAI_API_KEY']` |
| gpt-3.5-turbo-16k-0613 | `completion('gpt-3.5-turbo-16k-0613', messages)` | `os.environ['OPENAI_API_KEY']` |
| gpt-4 | `completion('gpt-4', messages)` | `os.environ['OPENAI_API_KEY']` |
| gpt-5-pro | `completion('gpt-5-pro', messages)` | `os.environ['OPENAI_API_KEY']` |
## Azure OpenAI Chat Completion Models
For Azure calls add the `azure/` prefix to `model`. If your azure deployment name is `gpt-v-2` set `model` = `azure/gpt-v-2`

2
litellm-proxy-extras/litellm_proxy_extras/migrations/20250918083359_drop_spec_version_column_from_mcp_table/migration.sql
View file

@ -5,4 +5,4 @@
*/
-- AlterTable
ALTER TABLE "public"."LiteLLM_MCPServerTable" DROP COLUMN "spec_version";
ALTER TABLE "LiteLLM_MCPServerTable" DROP COLUMN "spec_version";

3
litellm-proxy-extras/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql Normal file
View file

@ -0,0 +1,3 @@
-- AlterTable
ALTER TABLE "LiteLLM_ObjectPermissionTable" ADD COLUMN "mcp_tool_permissions" JSONB;

16
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -25,6 +25,7 @@ model LiteLLM_BudgetTable {
organization LiteLLM_OrganizationTable[] // multiple orgs can have the same budget
keys LiteLLM_VerificationToken[] // multiple keys can have the same budget
end_users LiteLLM_EndUserTable[] // multiple end-users can have the same budget
tags LiteLLM_TagTable[] // multiple tags can have the same budget
team_membership LiteLLM_TeamMembership[] // budgets of Users within a Team
organization_membership LiteLLM_OrganizationMembership[] // budgets of Users within a Organization
}
@ -156,6 +157,7 @@ model LiteLLM_ObjectPermissionTable {
object_permission_id String @id @default(uuid())
mcp_servers String[] @default([])
mcp_access_groups String[] @default([])
mcp_tool_permissions Json? // Tool-level permissions for MCP servers. Format: {"server_id": ["tool_name_1", "tool_name_2"]}
vector_stores String[] @default([])
teams LiteLLM_TeamTable[]
verification_tokens LiteLLM_VerificationToken[]
@ -244,6 +246,20 @@ model LiteLLM_EndUserTable {
blocked Boolean @default(false)
}
// Track tags with budgets and spend
model LiteLLM_TagTable {
tag_name String @id
description String?
models String[]
model_info Json? // maps model_id to model_name
spend Float @default(0.0)
budget_id String?
litellm_budget_table LiteLLM_BudgetTable? @relation(fields: [budget_id], references: [budget_id])
created_at DateTime @default(now()) @map("created_at")
created_by String?
updated_at DateTime @default(now()) @updatedAt @map("updated_at")
}
// store proxy config.yaml
model LiteLLM_Config {
param_name String @id

3
litellm/__init__.py
View file

@ -1190,6 +1190,9 @@ from .llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
from .llms.azure.responses.o_series_transformation import (
AzureOpenAIOSeriesResponsesAPIConfig,
)
from .llms.litellm_proxy.responses.transformation import (
LiteLLMProxyResponsesAPIConfig,
)
from .llms.openai.chat.o_series_transformation import (
OpenAIOSeriesConfig as OpenAIO1Config, # maintain backwards compatibility
OpenAIOSeriesConfig,

19
litellm/_redis.py
View file

@ -177,14 +177,21 @@ def get_redis_url_from_environment():
raise ValueError(
"Either 'REDIS_URL' or both 'REDIS_HOST' and 'REDIS_PORT' must be specified for Redis."
)
if "REDIS_PASSWORD" in os.environ:
redis_password = f":{os.environ['REDIS_PASSWORD']}@"
if "REDIS_SSL" in os.environ and os.environ["REDIS_SSL"].lower() == "true":
redis_protocol = "rediss"
else:
redis_password = ""
redis_protocol = "redis"
# Build authentication part of URL
auth_part = ""
if "REDIS_USERNAME" in os.environ and "REDIS_PASSWORD" in os.environ:
auth_part = f"{os.environ['REDIS_USERNAME']}:{os.environ['REDIS_PASSWORD']}@"
elif "REDIS_PASSWORD" in os.environ:
auth_part = f"{os.environ['REDIS_PASSWORD']}@"
return (
f"redis://{redis_password}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
f"{redis_protocol}://{auth_part}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
)

16
litellm/caching/caching_handler.py
View file

@ -14,10 +14,10 @@ It utilizes the (RedisCache, s3Cache, RedisSemanticCache, QdrantSemanticCache, I
In each method it will call the appropriate method from caching.py
"""
import time
import asyncio
import datetime
import inspect
import time
from typing import (
TYPE_CHECKING,
Any,
@ -62,12 +62,10 @@ else:
LiteLLMLoggingObj = Any
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.litellm_core_utils.core_helpers import (
_get_parent_otel_span_from_kwargs,
_get_parent_otel_span_from_kwargs,
)
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
class CachingHandlerResponse(BaseModel):
@ -214,9 +212,7 @@ class LLMCachingHandler:
end_time=end_time,
cache_hit=cache_hit,
)
cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
**kwargs
)
cache_key = litellm.cache.get_cache_key(**kwargs)
if (
isinstance(cached_result, BaseModel)
or isinstance(cached_result, CustomStreamWrapper)
@ -330,9 +326,7 @@ class LLMCachingHandler:
end_time=end_time,
cache_hit=cache_hit
)
cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
**kwargs
)
cache_key = litellm.cache.get_cache_key(**kwargs)
if (
isinstance(cached_result, BaseModel)
or isinstance(cached_result, CustomStreamWrapper)

36
litellm/constants.py
View file

@ -87,6 +87,35 @@ MAX_TOKEN_TRIMMING_ATTEMPTS = int(
########## Networking constants ##############################################################
_DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 # 1 hour, re-use the same httpx client for 1 hour
# Aiohttp connection pooling constants
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 0))
AIOHTTP_KEEPALIVE_TIMEOUT = int(os.getenv("AIOHTTP_KEEPALIVE_TIMEOUT", 120))
AIOHTTP_TTL_DNS_CACHE = int(os.getenv("AIOHTTP_TTL_DNS_CACHE", 300))
# SSL/TLS cipher configuration for faster handshakes
# Strategy: Strongly prefer fast modern ciphers, but allow fallback to commonly supported ones
# This balances performance with broad compatibility
DEFAULT_SSL_CIPHERS = os.getenv(
"LITELLM_SSL_CIPHERS",
# Priority 1: TLS 1.3 ciphers (fastest, ~50ms handshake)
"TLS_AES_256_GCM_SHA384:" # Fastest observed in testing
"TLS_AES_128_GCM_SHA256:" # Slightly faster than 256-bit
"TLS_CHACHA20_POLY1305_SHA256:" # Fast on ARM/mobile
# Priority 2: TLS 1.2 ECDHE+GCM (fast, ~100ms handshake, widely supported)
"ECDHE-RSA-AES256-GCM-SHA384:"
"ECDHE-RSA-AES128-GCM-SHA256:"
"ECDHE-ECDSA-AES256-GCM-SHA384:"
"ECDHE-ECDSA-AES128-GCM-SHA256:"
# Priority 3: Additional modern ciphers (good balance)
"ECDHE-RSA-CHACHA20-POLY1305:"
"ECDHE-ECDSA-CHACHA20-POLY1305:"
# Priority 4: Widely compatible fallbacks (slower but universally supported)
"ECDHE-RSA-AES256-SHA384:" # Common fallback
"ECDHE-RSA-AES128-SHA256:" # Very widely supported
"AES256-GCM-SHA384:" # Non-PFS fallback (compatibility)
"AES128-GCM-SHA256", # Last resort (maximum compatibility)
)
########### v2 Architecture constants for managing writing updates to the database ###########
REDIS_UPDATE_BUFFER_KEY = "litellm_spend_update_buffer"
REDIS_DAILY_SPEND_UPDATE_BUFFER_KEY = "litellm_daily_spend_update_buffer"
@ -871,6 +900,7 @@ bedrock_embedding_models: set = set(
"amazon.titan-embed-text-v1",
"cohere.embed-english-v3",
"cohere.embed-multilingual-v3",
"cohere.embed-v4:0",
"twelvelabs.marengo-embed-2-7-v1:0",
]
)
@ -1027,6 +1057,12 @@ PROXY_BATCH_WRITE_AT = int(os.getenv("PROXY_BATCH_WRITE_AT", 10)) # in seconds
DEFAULT_HEALTH_CHECK_INTERVAL = int(
os.getenv("DEFAULT_HEALTH_CHECK_INTERVAL", 300)
) # 5 minutes
DEFAULT_SHARED_HEALTH_CHECK_TTL = int(
os.getenv("DEFAULT_SHARED_HEALTH_CHECK_TTL", 300)
) # 5 minutes - TTL for cached health check results
DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL = int(
os.getenv("DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL", 60)
) # 1 minute - TTL for health check lock
PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS = int(
os.getenv("PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS", 9)
)

48
litellm/experimental_mcp_client/client.py
View file

@ -5,8 +5,9 @@ LiteLLM Proxy uses this MCP Client to connnect to other MCP servers.
import asyncio
import base64
from datetime import timedelta
from typing import Dict, List, Optional, Union
from typing import Callable, Dict, List, Optional, Union
import httpx
from mcp import ClientSession, StdioServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
@ -17,6 +18,8 @@ from mcp.types import TextContent
from mcp.types import Tool as MCPTool
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import get_ssl_configuration
from litellm.types.llms.custom_http import VerifyTypes
from litellm.types.mcp import (
MCPAuth,
MCPAuthType,
@ -48,6 +51,7 @@ class MCPClient:
timeout: float = 60.0,
stdio_config: Optional[MCPStdioConfig] = None,
extra_headers: Optional[Dict[str, str]] = None,
ssl_verify: Optional[VerifyTypes] = None,
):
self.server_url: str = server_url
self.transport_type: MCPTransport = transport_type
@ -62,6 +66,7 @@ class MCPClient:
self._task: Optional[asyncio.Task] = None
self.stdio_config: Optional[MCPStdioConfig] = stdio_config
self.extra_headers: Optional[Dict[str, str]] = extra_headers
self.ssl_verify: Optional[VerifyTypes] = ssl_verify
# handle the basic auth value if provided
if auth_value:
self.update_auth_value(auth_value)
@ -104,10 +109,12 @@ class MCPClient:
await self._session.initialize()
elif self.transport_type == MCPTransport.sse:
headers = self._get_auth_headers()
httpx_client_factory = self._create_httpx_client_factory()
self._transport_ctx = sse_client(
url=self.server_url,
timeout=self.timeout,
headers=headers,
httpx_client_factory=httpx_client_factory,
)
self._transport = await self._transport_ctx.__aenter__()
self._session_ctx = ClientSession(
@ -117,13 +124,15 @@ class MCPClient:
await self._session.initialize()
else: # http
headers = self._get_auth_headers()
httpx_client_factory = self._create_httpx_client_factory()
verbose_logger.debug(
"litellm headers for streamablehttp_client: ", headers
"litellm headers for streamablehttp_client: %s", headers
)
self._transport_ctx = streamablehttp_client(
url=self.server_url,
timeout=timedelta(seconds=self.timeout),
headers=headers,
httpx_client_factory=httpx_client_factory,
)
self._transport = await self._transport_ctx.__aenter__()
self._session_ctx = ClientSession(
@ -215,6 +224,41 @@ class MCPClient:
return headers
def _create_httpx_client_factory(self) -> Callable[..., httpx.AsyncClient]:
"""
Create a custom httpx client factory that uses LiteLLM's SSL configuration.
This factory follows the same CA bundle path logic as http_handler.py:
1. Check ssl_verify parameter (can be SSLContext, bool, or path to CA bundle)
2. Check SSL_VERIFY environment variable
3. Check SSL_CERT_FILE environment variable
4. Fall back to certifi CA bundle
"""
def factory(
*,
headers: Optional[Dict[str, str]] = None,
timeout: Optional[httpx.Timeout] = None,
auth: Optional[httpx.Auth] = None,
) -> httpx.AsyncClient:
"""Create an httpx.AsyncClient with LiteLLM's SSL configuration."""
# Get unified SSL configuration using the same logic as http_handler.py
ssl_config = get_ssl_configuration(self.ssl_verify)
verbose_logger.debug(
f"MCP client using SSL configuration: {type(ssl_config).__name__}"
)
return httpx.AsyncClient(
headers=headers,
timeout=timeout,
auth=auth,
verify=ssl_config,
follow_redirects=True,
)
return factory
async def list_tools(self) -> List[MCPTool]:
"""List available tools from the server."""
if not self._session:

9
litellm/files/main.py
View file

@ -18,6 +18,7 @@ from litellm import get_secret_str
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.files.handler import AzureOpenAIFilesAPI
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.llms.openai.openai import FileDeleted, FileObject, OpenAIFilesAPI
from litellm.llms.vertex_ai.files.handler import VertexAIFilesHandler
@ -268,6 +269,7 @@ def create_file(
raise e
@client
async def afile_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -308,6 +310,7 @@ async def afile_retrieve(
raise e
@client
def file_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -422,6 +425,7 @@ def file_retrieve(
# Delete file
@client
async def afile_delete(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -462,6 +466,7 @@ async def afile_delete(
raise e
@client
def file_delete(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -577,6 +582,7 @@ def file_delete(
# List files
@client
async def afile_list(
custom_llm_provider: Literal["openai", "azure"] = "openai",
purpose: Optional[str] = None,
@ -617,6 +623,7 @@ async def afile_list(
raise e
@client
def file_list(
custom_llm_provider: Literal["openai", "azure"] = "openai",
purpose: Optional[str] = None,
@ -729,6 +736,7 @@ def file_list(
raise e
@client
async def afile_content(
file_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
@ -771,6 +779,7 @@ async def afile_content(
raise e
@client
def file_content(
file_id: str,
model: Optional[str] = None,

95
litellm/integrations/posthog.py
View file

@ -26,7 +26,7 @@ from litellm.types.integrations.posthog import (
POSTHOG_MAX_BATCH_SIZE,
PostHogEventPayload,
)
from litellm.types.utils import StandardLoggingPayload
from litellm.types.utils import StandardCallbackDynamicParams, StandardLoggingPayload
class PostHogLogger(CustomBatchLogger):
@ -72,17 +72,19 @@ class PostHogLogger(CustomBatchLogger):
verbose_logger.debug(
"PostHog: Sync logging - Enters logging function for model %s", kwargs
)
api_key, api_url = self._get_credentials_for_request(kwargs)
event_payload = self.create_posthog_event_payload(kwargs)
headers = {
"Content-Type": "application/json",
}
payload = self._create_posthog_payload([event_payload])
payload = self._create_posthog_payload([event_payload], api_key)
capture_url = f"{api_url.rstrip('/')}/batch/"
response = self.sync_client.post(
url=self.capture_url,
url=capture_url,
json=payload,
headers=headers,
)
@ -92,9 +94,9 @@ class PostHogLogger(CustomBatchLogger):
raise Exception(
f"Response from PostHog API status_code: {response.status_code}, text: {response.text}"
)
verbose_logger.debug("PostHog: Sync event successfully sent")
except Exception as e:
verbose_logger.exception(f"PostHog Sync Layer Error - {str(e)}")
@ -122,9 +124,15 @@ class PostHogLogger(CustomBatchLogger):
async def _log_async_event(self, kwargs, response_obj=None, start_time=0.0, end_time=0.0):
# Note: response_obj, start_time, end_time not used - all data comes from kwargs
api_key, api_url = self._get_credentials_for_request(kwargs)
event_payload = self.create_posthog_event_payload(kwargs)
self.log_queue.append(event_payload)
# Store event with its credentials for batch sending
self.log_queue.append({
"event": event_payload,
"api_key": api_key,
"api_url": api_url
})
verbose_logger.debug(
f"PostHog, event added to queue. Will flush in {self.flush_interval} seconds..."
)
@ -257,16 +265,42 @@ class PostHogLogger(CustomBatchLogger):
metadata = self._extract_metadata(kwargs)
user_id = self._safe_get(metadata, "user_id")
if user_id:
return str(user_id)
return str(user_id)
end_user = self._safe_get(standard_logging_object, "end_user")
if end_user:
return str(end_user)
trace_id = self._safe_get(standard_logging_object, "trace_id")
if trace_id:
return str(trace_id)
return str(trace_id)
return self._safe_uuid()
def _get_credentials_for_request(self, kwargs: Dict[str, Any]) -> tuple[str, str]:
"""
Get PostHog credentials for this request.
Checks for per-request credentials in standard_callback_dynamic_params,
falls back to instance defaults from environment variables.
Args:
kwargs: Request kwargs containing standard_callback_dynamic_params
Returns:
tuple[str, str]: (api_key, api_url)
"""
standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
kwargs.get("standard_callback_dynamic_params", None)
)
if standard_callback_dynamic_params is not None:
api_key = standard_callback_dynamic_params.get("posthog_api_key") or self.POSTHOG_API_KEY
api_url = standard_callback_dynamic_params.get("posthog_api_url") or self.posthog_host
else:
api_key = self.POSTHOG_API_KEY
api_url = self.posthog_host
return api_key, api_url
async def async_send_batch(self):
"""
Sends the in memory logs queue to PostHog API
@ -282,23 +316,34 @@ class PostHogLogger(CustomBatchLogger):
f"PostHog: Sending batch of {len(self.log_queue)} events"
)
headers = {
"Content-Type": "application/json",
}
# Group events by credentials for batch sending
batches_by_credentials: Dict[tuple[str, str], list] = {}
for item in self.log_queue:
key = (item["api_key"], item["api_url"])
if key not in batches_by_credentials:
batches_by_credentials[key] = []
batches_by_credentials[key].append(item["event"])
payload = self._create_posthog_payload(list(self.log_queue))
# Send each batch to its respective PostHog instance
for (api_key, api_url), events in batches_by_credentials.items():
headers = {
"Content-Type": "application/json",
}
response = await self.async_client.post(
url=self.capture_url,
json=payload,
headers=headers,
)
response.raise_for_status()
payload = self._create_posthog_payload(events, api_key)
capture_url = f"{api_url.rstrip('/')}/batch/"
if response.status_code != 200:
raise Exception(
f"Response from PostHog API status_code: {response.status_code}, text: {response.text}"
response = await self.async_client.post(
url=capture_url,
json=payload,
headers=headers,
)
response.raise_for_status()
if response.status_code != 200:
raise Exception(
f"Response from PostHog API status_code: {response.status_code}, text: {response.text}"
)
verbose_logger.debug(
f"PostHog: Batch of {len(self.log_queue)} events successfully sent"
@ -324,8 +369,8 @@ class PostHogLogger(CustomBatchLogger):
def _safe_uuid(self) -> str:
return str(uuid.uuid4())
def _create_posthog_payload(self, events: list) -> Dict[str, Any]:
return {"api_key": self.POSTHOG_API_KEY, "batch": events}
def _create_posthog_payload(self, events: list, api_key: str) -> Dict[str, Any]:
return {"api_key": api_key, "batch": events}
def _safe_get(self, obj: Any, key: str, default: Any = None) -> Any:
if obj is None or not hasattr(obj, 'get'):

2
litellm/litellm_core_utils/litellm_logging.py
View file

@ -81,12 +81,12 @@ from litellm.types.llms.openai import (
)
from litellm.types.mcp import MCPPostCallResponseObject
from litellm.types.rerank import RerankResponse
from litellm.types.router import CustomPricingLiteLLMParams
from litellm.types.utils import (
CachingDetails,
CallTypes,
CostBreakdown,
CostResponseTypes,
CustomPricingLiteLLMParams,
DynamicPromptManagementParamLiteral,
EmbeddingResponse,
GuardrailStatus,

290
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -2,7 +2,6 @@ import copy
import json
import mimetypes
import re
from litellm._uuid import uuid
import xml.etree.ElementTree as ET
from enum import Enum
from typing import Any, List, Optional, Tuple, cast, overload
@ -13,6 +12,7 @@ import litellm
import litellm.types
import litellm.types.llms
from litellm import verbose_logger
from litellm._uuid import uuid
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_async_httpx_client
from litellm.types.files import get_file_extension_from_mime_type
from litellm.types.llms.anthropic import *
@ -232,7 +232,6 @@ def ollama_pt(
## MERGE CONSECUTIVE ASSISTANT CONTENT ##
while msg_i < len(messages) and messages[msg_i]["role"] == "assistant":
assistant_content_str += convert_content_list_to_str(messages[msg_i])
msg_i += 1
tool_calls = messages[msg_i].get("tool_calls")
ollama_tool_calls = []
@ -258,7 +257,7 @@ def ollama_pt(
f"Tool Calls: {json.dumps(ollama_tool_calls, indent=2)}"
)
msg_i += 1
msg_i += 1
if assistant_content_str:
prompt += f"### Assistant:\n{assistant_content_str}\n\n"
@ -365,62 +364,20 @@ def phind_codellama_pt(messages):
return prompt
def hf_chat_template( # noqa: PLR0915
model: str, messages: list, chat_template: Optional[Any] = None
):
# Define Jinja2 environment
env = ImmutableSandboxedEnvironment()
def raise_exception(message):
raise Exception(f"Error message - {message}")
# Create a template object from the template text
env.globals["raise_exception"] = raise_exception
## get the tokenizer config from huggingface
bos_token = ""
eos_token = ""
if chat_template is None:
def _get_tokenizer_config(hf_model_name):
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
# Make a GET request to fetch the JSON data
client = HTTPHandler(concurrent_limit=1)
response = client.get(url)
except Exception as e:
raise e
if response.status_code == 200:
# Parse the JSON data
tokenizer_config = json.loads(response.content)
return {"status": "success", "tokenizer": tokenizer_config}
else:
return {"status": "failure"}
if model in litellm.known_tokenizer_config:
tokenizer_config = litellm.known_tokenizer_config[model]
else:
tokenizer_config = _get_tokenizer_config(model)
litellm.known_tokenizer_config.update({model: tokenizer_config})
if (
tokenizer_config["status"] == "failure"
or "chat_template" not in tokenizer_config["tokenizer"]
):
raise Exception("No chat template found")
## read the bos token, eos token and chat template from the json
tokenizer_config = tokenizer_config["tokenizer"] # type: ignore
bos_token = tokenizer_config["bos_token"] # type: ignore
if bos_token is not None and not isinstance(bos_token, str):
if isinstance(bos_token, dict):
bos_token = bos_token.get("content", None)
eos_token = tokenizer_config["eos_token"] # type: ignore
if eos_token is not None and not isinstance(eos_token, str):
if isinstance(eos_token, dict):
eos_token = eos_token.get("content", None)
chat_template = tokenizer_config["chat_template"] # type: ignore
def _render_chat_template(env, chat_template: str, bos_token: str, eos_token: str, messages: list) -> str:
"""
Shared template rendering logic for both sync and async hf_chat_template
Args:
env: Jinja2 environment
chat_template: Chat template string
bos_token: Beginning of sequence token
eos_token: End of sequence token
messages: Messages to render
Returns:
Rendered template string
"""
try:
template = env.from_string(chat_template) # type: ignore
except Exception as e:
@ -435,7 +392,6 @@ def hf_chat_template( # noqa: PLR0915
bos_token="<bos>",
)
return True
# This will be raised if Jinja attempts to render the system message and it can't
except Exception:
return False
@ -469,7 +425,7 @@ def hf_chat_template( # noqa: PLR0915
)
except Exception as e:
if "Conversation roles must alternate user/assistant" in str(e):
# reformat messages to ensure user/assistant are alternating, if there's either 2 consecutive 'user' messages or 2 consecutive 'assistant' message, add a blank 'user' or 'assistant' message to ensure compatibility
# reformat messages to ensure user/assistant are alternating
new_messages = []
for i in range(len(reformatted_messages) - 1):
new_messages.append(reformatted_messages[i])
@ -495,6 +451,188 @@ def hf_chat_template( # noqa: PLR0915
) # don't use verbose_logger.exception, if exception is raised
async def _afetch_and_extract_template(
model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
) -> Tuple[str, str, str]:
"""
Async version: Fetch template and tokens from HuggingFace.
Returns: (chat_template, bos_token, eos_token)
"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_extract_token_value,
)
bos_token = ""
eos_token = ""
if chat_template is None:
# Fetch or retrieve cached tokenizer config
if model in litellm.known_tokenizer_config:
tokenizer_config = litellm.known_tokenizer_config[model]
else:
tokenizer_config = await get_config_fn(hf_model_name=model)
litellm.known_tokenizer_config.update({model: tokenizer_config})
# Try to get chat template from tokenizer_config.json first
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
and "chat_template" in tokenizer_config["tokenizer"]
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
chat_template = tokenizer_data["chat_template"]
else:
# Fallback: Try to fetch chat template from separate .jinja file
template_result = await get_template_fn(hf_model_name=model)
if template_result.get("status") == "success":
chat_template = template_result["chat_template"]
# Still try to get tokens from tokenizer_config if available
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
else:
raise Exception("No chat template found")
return chat_template, bos_token, eos_token # type: ignore
def _fetch_and_extract_template(
model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
) -> Tuple[str, str, str]:
"""
Sync version: Fetch template and tokens from HuggingFace.
Returns: (chat_template, bos_token, eos_token)
"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_extract_token_value,
)
bos_token = ""
eos_token = ""
if chat_template is None:
# Fetch or retrieve cached tokenizer config
if model in litellm.known_tokenizer_config:
tokenizer_config = litellm.known_tokenizer_config[model]
else:
tokenizer_config = get_config_fn(hf_model_name=model)
litellm.known_tokenizer_config.update({model: tokenizer_config})
# Try to get chat template from tokenizer_config.json first
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
and "chat_template" in tokenizer_config["tokenizer"]
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
chat_template = tokenizer_data["chat_template"]
else:
# Fallback: Try to fetch chat template from separate .jinja file
template_result = get_template_fn(hf_model_name=model)
if template_result.get("status") == "success":
chat_template = template_result["chat_template"]
# Still try to get tokens from tokenizer_config if available
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
else:
raise Exception("No chat template found")
return chat_template, bos_token, eos_token # type: ignore
async def ahf_chat_template(
model: str, messages: list, chat_template: Optional[Any] = None
):
"""HuggingFace chat template (async version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_aget_chat_template_file,
_aget_tokenizer_config,
strftime_now,
)
env = ImmutableSandboxedEnvironment()
env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
env.globals["strftime_now"] = strftime_now
template, bos_token, eos_token = await _afetch_and_extract_template(
model=model,
chat_template=chat_template,
get_config_fn=_aget_tokenizer_config,
get_template_fn=_aget_chat_template_file,
)
return _render_chat_template(
env=env,
chat_template=template,
bos_token=bos_token,
eos_token=eos_token,
messages=messages,
)
def hf_chat_template(
model: str, messages: list, chat_template: Optional[Any] = None
):
"""HuggingFace chat template (sync version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_get_chat_template_file,
_get_tokenizer_config,
strftime_now,
)
env = ImmutableSandboxedEnvironment()
env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
env.globals["strftime_now"] = strftime_now
template, bos_token, eos_token = _fetch_and_extract_template(
model=model,
chat_template=chat_template,
get_config_fn=_get_tokenizer_config,
get_template_fn=_get_chat_template_file,
)
return _render_chat_template(
env=env,
chat_template=template,
bos_token=bos_token,
eos_token=eos_token,
messages=messages,
)
def deepseek_r1_pt(messages):
return hf_chat_template(
model="deepseek-r1/deepseek-r1-7b-instruct", messages=messages
@ -4032,33 +4170,9 @@ def prompt_factory(
elif custom_llm_provider == "azure_text":
return azure_text_pt(messages=messages)
elif custom_llm_provider == "watsonx":
if "granite" in model and "chat" in model:
# granite-13b-chat-v1 and granite-13b-chat-v2 use a specific prompt template
return ibm_granite_pt(messages=messages)
elif "ibm-mistral" in model and "instruct" in model:
# models like ibm-mistral/mixtral-8x7b-instruct-v01-q use the mistral instruct prompt template
return mistral_instruct_pt(messages=messages)
elif "meta-llama/llama-3" in model and "instruct" in model:
# https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/
return custom_prompt(
role_dict={
"system": {
"pre_message": "<|start_header_id|>system<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"user": {
"pre_message": "<|start_header_id|>user<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"assistant": {
"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
},
messages=messages,
initial_prompt_value="<|begin_of_text|>",
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
)
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
return IBMWatsonXChatConfig.apply_prompt_template(model=model, messages=messages)
try:
if "meta-llama/llama-2" in model and "chat" in model:
return llama_2_chat_pt(messages=messages)

139
litellm/litellm_core_utils/prompt_templates/huggingface_template_handler.py Normal file
View file

@ -0,0 +1,139 @@
import json
from datetime import datetime
from typing import Any, Dict, Union
from litellm.llms.custom_httpx.http_handler import (
_get_httpx_client,
get_async_httpx_client,
)
from litellm.types.llms.custom_http import httpxSpecialProvider
def strftime_now(fmt: str) -> str:
"""
Custom function for templates that need current date/time formatting (e.g., gpt-oss)
Args:
fmt: Format string for datetime.now().strftime()
Returns:
Formatted string
"""
return datetime.now().strftime(fmt)
def _get_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch tokenizer_config.json from HuggingFace (sync)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'tokenizer' keys
"""
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
client = _get_httpx_client()
response = client.get(url=url)
except Exception as e:
raise e
if response.status_code == 200:
tokenizer_config = json.loads(response.content)
return {"status": "success", "tokenizer": tokenizer_config}
else:
return {"status": "failure"}
async def _aget_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch tokenizer_config.json from HuggingFace (async)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'tokenizer' keys
"""
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.PromptFactory,
)
response = await client.get(url=url)
except Exception as e:
raise e
if response.status_code == 200:
tokenizer_config = json.loads(response.content)
return {"status": "success", "tokenizer": tokenizer_config}
else:
return {"status": "failure"}
def _get_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch chat template from separate .jinja file (sync)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'chat_template' keys
"""
template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
client = _get_httpx_client()
for filename in template_filenames:
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
response = client.get(url=url)
if response.status_code == 200:
return {"status": "success", "chat_template": response.content.decode("utf-8")}
except Exception:
continue
return {"status": "failure"}
async def _aget_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch chat template from separate .jinja file (async)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'chat_template' keys
"""
template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.PromptFactory,
)
for filename in template_filenames:
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
response = await client.get(url=url)
if response.status_code == 200:
return {"status": "success", "chat_template": response.content.decode("utf-8")}
except Exception:
continue
return {"status": "failure"}
def _extract_token_value(token_value: Union[None, str, Dict[str, Any]]) -> str:
"""
Extract token string from various formats (string, dict, etc.)
Args:
token_value: Token value in various formats (None, str, or dict with 'content' key)
Returns:
Extracted token string
"""
if token_value is None or isinstance(token_value, str):
return token_value or ""
if isinstance(token_value, dict):
return token_value.get("content", "")
return ""

8
litellm/litellm_core_utils/sensitive_data_masker.py
View file

@ -75,7 +75,7 @@ class SensitiveDataMasker:
masked_data[k] = self._mask_value(str_value)
else:
masked_data[k] = (
v if isinstance(v, (int, float, bool, str)) else str(v)
v if isinstance(v, (int, float, bool, str, list)) else str(v)
)
except Exception:
masked_data[k] = "<unable to serialize>"
@ -89,12 +89,14 @@ masker = SensitiveDataMasker()
data = {
"api_key": "sk-1234567890abcdef",
"redis_password": "very_secret_pass",
"port": 6379
"port": 6379,
"tags": ["East US 2", "production", "test"]
}
masked = masker.mask_dict(data)
# Result: {
# "api_key": "sk-1****cdef",
# "redis_password": "very****pass",
# "port": 6379
# "port": 6379,
# "tags": ["East US 2", "production", "test"]
# }
"""

2
litellm/llms/anthropic/chat/transformation.py
View file

@ -878,7 +878,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
prompt_tokens_details = PromptTokensDetailsWrapper(
cached_tokens=cache_read_input_tokens,
cache_creation_tokens=cache_read_input_tokens,
cache_creation_tokens=cache_creation_input_tokens,
cache_creation_token_details=cache_creation_token_details,
)
completion_token_details = (

8
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
View file

@ -270,7 +270,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
processed_chunk.get("delta", {}).get("stop_reason")
is not None
):
self.holding_stop_reason_chunk = processed_chunk
else:
self.chunk_queue.append(processed_chunk)
@ -380,4 +379,11 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
self.current_content_block_start = content_block_start
return True
# For parallel tool calls, we'll necessarily have a new content block
# if we get a function name since it signals a new tool call
if block_type == "tool_use" and content_block_start.get("name"):
self.current_content_block_type = block_type
self.current_content_block_start = content_block_start
return True
return False

15
litellm/llms/azure/common_utils.py
View file

@ -365,6 +365,11 @@ def get_azure_ad_token(
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
except ValueError:
verbose_logger.debug("Azure AD Token Provider could not be used.")
except Exception as e:
verbose_logger.error(
f"Error calling Azure AD token provider: {str(e)}. Follow docs - https://docs.litellm.ai/docs/providers/azure/#azure-ad-token-refresh---defaultazurecredential"
)
raise e
#########################################################
# If litellm.enable_azure_ad_token_refresh is True and no other token provider is available,
@ -561,7 +566,9 @@ class BaseAzureLLM(BaseOpenAILLM):
"Using Azure AD token provider based on Service Principal with Secret workflow for Azure Auth"
)
try:
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
azure_ad_token_provider = get_azure_ad_token_provider(
azure_scope=scope,
)
except ValueError:
verbose_logger.debug("Azure AD Token Provider could not be used.")
if api_version is None:
@ -665,10 +672,6 @@ class BaseAzureLLM(BaseOpenAILLM):
) -> dict:
litellm_params = litellm_params or GenericLiteLLMParams()
# If api-key is already in headers, preserve it
if "api-key" in headers:
return headers
api_key = (
litellm_params.api_key
or litellm.api_key
@ -693,7 +696,7 @@ class BaseAzureLLM(BaseOpenAILLM):
def _get_base_azure_url(
api_base: Optional[str],
litellm_params: Optional[Union[GenericLiteLLMParams, Dict[str, Any]]],
route: Literal["/openai/responses", "/openai/vector_stores"],
route: Union[Literal["/openai/responses", "/openai/vector_stores"], str],
default_api_version: Optional[Union[str, Literal["latest", "preview"]]] = None,
) -> str:
"""

85
litellm/llms/azure/passthrough/transformation.py Normal file
View file

@ -0,0 +1,85 @@
from typing import TYPE_CHECKING, List, Optional, Tuple
import httpx
from litellm.llms.azure.common_utils import BaseAzureLLM
from litellm.llms.base_llm.passthrough.transformation import BasePassthroughConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
from litellm.types.router import GenericLiteLLMParams
if TYPE_CHECKING:
from httpx import URL
class AzurePassthroughConfig(BasePassthroughConfig):
def is_streaming_request(self, endpoint: str, request_data: dict) -> bool:
return "stream" in request_data
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
endpoint: str,
request_query_params: Optional[dict],
litellm_params: dict,
) -> Tuple["URL", str]:
base_target_url = self.get_api_base(api_base)
if base_target_url is None:
raise Exception("Azure api base not found")
litellm_metadata = litellm_params.get("litellm_metadata") or {}
model_group = litellm_metadata.get("model_group")
if model_group and model_group in endpoint:
endpoint = endpoint.replace(model_group, model)
complete_url = BaseAzureLLM._get_base_azure_url(
api_base=base_target_url,
litellm_params=litellm_params,
route=endpoint,
default_api_version=litellm_params.get("api_version"),
)
return (
httpx.URL(complete_url),
base_target_url,
)
def validate_environment(
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
return BaseAzureLLM._base_validate_azure_environment(
headers=headers,
litellm_params=GenericLiteLLMParams(
**{**litellm_params, "api_key": api_key}
),
)
@staticmethod
def get_api_base(
api_base: Optional[str] = None,
) -> Optional[str]:
return api_base or get_secret_str("AZURE_API_BASE")
@staticmethod
def get_api_key(
api_key: Optional[str] = None,
) -> Optional[str]:
return api_key or get_secret_str("AZURE_API_KEY")
@staticmethod
def get_base_model(model: str) -> Optional[str]:
return model
def get_models(
self, api_key: Optional[str] = None, api_base: Optional[str] = None
) -> List[str]:
return super().get_models(api_key, api_base)

64
litellm/llms/azure/responses/transformation.py
View file

@ -1,6 +1,7 @@
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Tuple
import httpx
from openai.types.responses import ResponseReasoningItem
from litellm._logging import verbose_logger
from litellm.llms.azure.common_utils import BaseAzureLLM
@ -38,6 +39,50 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
model = model.replace("o_series/", "")
return model
def _handle_reasoning_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
"""
Handle reasoning items specifically to filter out status=None using OpenAI's model.
Issue: https://github.com/BerriAI/litellm/issues/13484
OpenAI API does not accept ReasoningItem(status=None), so we need to:
1. Check if the item is a reasoning type
2. Create a ResponseReasoningItem object with the item data
3. Convert it back to dict with exclude_none=True to filter None values
"""
if item.get("type") == "reasoning":
try:
# Ensure required fields are present for ResponseReasoningItem
item_data = dict(item)
if "id" not in item_data:
item_data["id"] = f"rs_{hash(str(item_data))}"
if "summary" not in item_data:
item_data["summary"] = (
item_data.get("reasoning_content", "")[:100] + "..."
if len(item_data.get("reasoning_content", "")) > 100
else item_data.get("reasoning_content", "")
)
# Create ResponseReasoningItem object from the item data
reasoning_item = ResponseReasoningItem(**item_data)
# Convert back to dict with exclude_none=True to exclude None fields
dict_reasoning_item = reasoning_item.model_dump(exclude_none=True)
dict_reasoning_item.pop("status", None)
return dict_reasoning_item
except Exception as e:
verbose_logger.debug(
f"Failed to create ResponseReasoningItem, falling back to manual filtering: {e}"
)
# Fallback: manually filter out known None fields
filtered_item = {
k: v
for k, v in item.items()
if v is not None
or k not in {"status", "content", "encrypted_content"}
}
return filtered_item
return item
def transform_responses_api_request(
self,
model: str,
@ -48,12 +93,13 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
) -> Dict:
"""No transform applied since inputs are in OpenAI spec already"""
stripped_model_name = self.get_stripped_model_name(model)
return dict(
ResponsesAPIRequestParams(
model=stripped_model_name,
input=input,
**response_api_optional_request_params,
)
return super().transform_responses_api_request(
model=stripped_model_name,
input=input,
response_api_optional_request_params=response_api_optional_request_params,
litellm_params=litellm_params,
headers=headers,
)
def get_complete_url(
@ -217,15 +263,15 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
at the correct location (before any query parameters).
"""
from urllib.parse import urlparse, urlunparse
# Parse the URL to separate its components
parsed_url = urlparse(api_base)
# Insert the response_id and /cancel at the end of the path component
# Remove trailing slash if present to avoid double slashes
path = parsed_url.path.rstrip("/")
new_path = f"{path}/{response_id}/cancel"
# Reconstruct the URL with all original components but with the modified path
cancel_url = urlunparse(
(

82
litellm/llms/azure_ai/chat/transformation.py
View file

@ -14,6 +14,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.llms.base_llm.chat.transformation import LiteLLMLoggingObj
from litellm.llms.openai.common_utils import drop_params_from_unprocessable_entity_error
from litellm.llms.openai.openai import OpenAIConfig
from litellm.llms.xai.chat.transformation import XAIChatConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse, ProviderField
@ -35,9 +36,24 @@ class AzureAIStudioConfig(OpenAIConfig):
for param in supported_params:
if param != "tool_choice":
filtered_supported_params.append(param)
return filtered_supported_params
supported_params = filtered_supported_params
# Filter out unsupported parameters for specific models
if not self._supports_stop_reason(model):
supported_params = [param for param in supported_params if param != "stop"]
return supported_params
def _supports_stop_reason(self, model: str) -> bool:
"""
Check if the model supports stop tokens.
"""
if "grok" in model:
# Reuse Xai method for Grok model
xai_config = XAIChatConfig()
return xai_config._supports_stop_reason(model)
return True
def validate_environment(
self,
headers: dict,
@ -53,9 +69,7 @@ class AzureAIStudioConfig(OpenAIConfig):
else:
headers["Authorization"] = f"Bearer {api_key}"
headers["Content-Type"] = (
"application/json" # tell Azure AI Studio to expect JSON
)
headers["Content-Type"] = "application/json" # tell Azure AI Studio to expect JSON
return headers
@ -65,10 +79,7 @@ class AzureAIStudioConfig(OpenAIConfig):
"""
parsed_url = urlparse(api_base)
host = parsed_url.hostname
if host and (
host.endswith(".services.ai.azure.com")
or host.endswith(".openai.azure.com")
):
if host and (host.endswith(".services.ai.azure.com") or host.endswith(".openai.azure.com")):
return True
return False
@ -115,13 +126,9 @@ class AzureAIStudioConfig(OpenAIConfig):
# Add the path to the base URL
if "services.ai.azure.com" in api_base:
new_url = _add_path_to_api_base(
api_base=api_base, ending_path="/models/chat/completions"
)
new_url = _add_path_to_api_base(api_base=api_base, ending_path="/models/chat/completions")
else:
new_url = _add_path_to_api_base(
api_base=api_base, ending_path="/chat/completions"
)
new_url = _add_path_to_api_base(api_base=api_base, ending_path="/chat/completions")
# Use the new query_params dictionary
final_url = httpx.URL(new_url).copy_with(params=query_params)
@ -191,11 +198,7 @@ class AzureAIStudioConfig(OpenAIConfig):
dynamic_api_key = api_key or get_secret_str("AZURE_AI_API_KEY")
if self._is_azure_openai_model(model=model, api_base=api_base):
verbose_logger.debug(
"Model={} is Azure OpenAI model. Setting custom_llm_provider='azure'.".format(
model
)
)
verbose_logger.debug("Model={} is Azure OpenAI model. Setting custom_llm_provider='azure'.".format(model))
custom_llm_provider = "azure"
return api_base, dynamic_api_key, custom_llm_provider
@ -211,9 +214,7 @@ class AzureAIStudioConfig(OpenAIConfig):
if extra_body and isinstance(extra_body, dict):
optional_params.update(extra_body)
optional_params.pop("max_retries", None)
return super().transform_request(
model, messages, optional_params, litellm_params, headers
)
return super().transform_request(model, messages, optional_params, litellm_params, headers)
def transform_response(
self,
@ -252,47 +253,30 @@ class AzureAIStudioConfig(OpenAIConfig):
if should_drop_params and "Extra inputs are not permitted" in error_text:
return True
elif (
"unknown field: parameter index is not a valid field" in error_text
): # remove index from tool calls
elif "unknown field: parameter index is not a valid field" in error_text: # remove index from tool calls
return True
elif (
AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value
in error_text
AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value in error_text
): # remove extra-parameters from tool calls
return True
return super().should_retry_llm_api_inside_llm_translation_on_http_error(
e=e, litellm_params=litellm_params
)
return super().should_retry_llm_api_inside_llm_translation_on_http_error(e=e, litellm_params=litellm_params)
@property
def max_retry_on_unprocessable_entity_error(self) -> int:
return 2
def transform_request_on_unprocessable_entity_error(
self, e: httpx.HTTPStatusError, request_data: dict
) -> dict:
def transform_request_on_unprocessable_entity_error(self, e: httpx.HTTPStatusError, request_data: dict) -> dict:
_messages = cast(Optional[List[AllMessageValues]], request_data.get("messages"))
if (
"unknown field: parameter index is not a valid field" in e.response.text
and _messages is not None
):
if "unknown field: parameter index is not a valid field" in e.response.text and _messages is not None:
litellm.remove_index_from_tool_calls(
messages=_messages,
)
elif (
AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value
in e.response.text
):
request_data = self._drop_extra_params_from_request_data(
request_data, e.response.text
)
elif AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value in e.response.text:
request_data = self._drop_extra_params_from_request_data(request_data, e.response.text)
data = drop_params_from_unprocessable_entity_error(e=e, data=request_data)
return data
def _drop_extra_params_from_request_data(
self, request_data: dict, error_text: str
) -> dict:
def _drop_extra_params_from_request_data(self, request_data: dict, error_text: str) -> dict:
params_to_drop = self._extract_params_to_drop_from_error_text(error_text)
if params_to_drop:
for param in params_to_drop:
@ -300,9 +284,7 @@ class AzureAIStudioConfig(OpenAIConfig):
request_data.pop(param, None)
return request_data
def _extract_params_to_drop_from_error_text(
self, error_text: str
) -> Optional[List[str]]:
def _extract_params_to_drop_from_error_text(self, error_text: str) -> Optional[List[str]]:
"""
Error text looks like this"
"Extra parameters ['stream_options', 'extra-parameters'] are not allowed when extra-parameters is not set or set to be 'error'.

4
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -1092,10 +1092,8 @@ class AmazonConverseConfig(BaseConfig):
cache_read_input_tokens = usage["cacheReadInputTokens"]
input_tokens += cache_read_input_tokens
if "cacheWriteInputTokens" in usage:
"""
Do not increment prompt_tokens with cacheWriteInputTokens
"""
cache_creation_input_tokens = usage["cacheWriteInputTokens"]
input_tokens += cache_creation_input_tokens
prompt_tokens_details = PromptTokensDetailsWrapper(
cached_tokens=cache_read_input_tokens

2
litellm/llms/bedrock/common_utils.py
View file

@ -440,7 +440,7 @@ class BedrockModelInfo(BaseLLMModelInfo):
"""
Abbreviations of regions AWS Bedrock supports for cross region inference
"""
return ["us", "eu", "apac", "jp"]
return ["global", "us", "eu", "apac", "jp", "au"]
@staticmethod
def get_bedrock_route(

128
litellm/llms/custom_httpx/aiohttp_transport.py
View file

@ -152,6 +152,16 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
# If we don't have a client or it's not a ClientSession, create one
if not isinstance(self.client, ClientSession):
if hasattr(self, "_client_factory") and callable(self._client_factory):
self.client = self._client_factory()
else:
self.client = ClientSession()
# Don't return yet - check if the newly created session is valid
# Check if the session itself is closed
if self.client.closed:
verbose_logger.debug("Session is closed, creating new session")
# Create a new session
if hasattr(self, "_client_factory") and callable(self._client_factory):
self.client = self._client_factory()
else:
@ -169,14 +179,17 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
or session_loop != current_loop
or session_loop.is_closed()
):
# Clean up the old session
# Close old session to prevent leaks
old_session = self.client
try:
# Note: not awaiting close() here as it might be from a different loop
# The session will be garbage collected
pass
if not old_session.closed:
try:
asyncio.create_task(old_session.close())
except RuntimeError:
# Different event loop - can't schedule task, rely on GC
verbose_logger.debug("Old session from different loop, relying on GC")
except Exception as e:
verbose_logger.debug(f"Error closing old session: {e}")
pass
# Create a new session in the current event loop
if hasattr(self, "_client_factory") and callable(self._client_factory):
@ -193,13 +206,58 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
return self.client
async def _make_aiohttp_request(
self,
client_session: ClientSession,
request: httpx.Request,
timeout: dict,
proxy: Optional[str],
sni_hostname: Optional[str],
) -> ClientResponse:
"""
Helper function to make an aiohttp request with the given parameters.
Args:
client_session: The aiohttp ClientSession to use
request: The httpx Request to send
timeout: Timeout settings dict with 'connect', 'read', 'pool' keys
proxy: Optional proxy URL
sni_hostname: Optional SNI hostname for SSL
Returns:
ClientResponse from aiohttp
"""
from aiohttp import ClientTimeout
from yarl import URL as YarlURL
try:
data = request.content
except httpx.RequestNotRead:
data = request.stream # type: ignore
request.headers.pop("transfer-encoding", None) # handled by aiohttp
response = await client_session.request(
method=request.method,
url=YarlURL(str(request.url), encoded=True),
headers=request.headers,
data=data,
allow_redirects=False,
auto_decompress=False,
timeout=ClientTimeout(
sock_connect=timeout.get("connect"),
sock_read=timeout.get("read"),
connect=timeout.get("pool"),
),
proxy=proxy,
server_hostname=sni_hostname,
).__aenter__()
return response
async def handle_async_request(
self,
request: httpx.Request,
) -> httpx.Response:
from aiohttp import ClientTimeout
from yarl import URL as YarlURL
timeout = request.extensions.get("timeout", {})
sni_hostname = request.extensions.get("sni_hostname")
@ -209,28 +267,38 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
# Resolve proxy settings from environment variables
proxy = await self._get_proxy_settings(request)
with map_aiohttp_exceptions():
try:
data = request.content
except httpx.RequestNotRead:
data = request.stream # type: ignore
request.headers.pop("transfer-encoding", None) # handled by aiohttp
response = await client_session.request(
method=request.method,
url=YarlURL(str(request.url), encoded=True),
headers=request.headers,
data=data,
allow_redirects=False,
auto_decompress=False,
timeout=ClientTimeout(
sock_connect=timeout.get("connect"),
sock_read=timeout.get("read"),
connect=timeout.get("pool"),
),
proxy=proxy,
server_hostname=sni_hostname,
).__aenter__()
try:
with map_aiohttp_exceptions():
response = await self._make_aiohttp_request(
client_session=client_session,
request=request,
timeout=timeout,
proxy=proxy,
sni_hostname=sni_hostname,
)
except RuntimeError as e:
# Handle the case where session was closed between our check and actual use
if "Session is closed" in str(e):
verbose_logger.debug(f"Session closed during request, retrying with new session: {e}")
# Force creation of a new session
if hasattr(self, "_client_factory") and callable(self._client_factory):
self.client = self._client_factory()
else:
self.client = ClientSession()
client_session = self.client
# Retry the request with the new session
with map_aiohttp_exceptions():
response = await self._make_aiohttp_request(
client_session=client_session,
request=request,
timeout=timeout,
proxy=proxy,
sni_hostname=sni_hostname,
)
else:
# Re-raise if it's a different RuntimeError
raise
return httpx.Response(
status_code=response.status,

51
litellm/llms/custom_httpx/http_handler.py
View file

@ -12,7 +12,13 @@ from httpx._types import RequestFiles
import litellm
from litellm._logging import verbose_logger
from litellm.constants import _DEFAULT_TTL_FOR_HTTPX_CLIENTS
from litellm.constants import (
_DEFAULT_TTL_FOR_HTTPX_CLIENTS,
AIOHTTP_CONNECTOR_LIMIT,
AIOHTTP_KEEPALIVE_TIMEOUT,
AIOHTTP_TTL_DNS_CACHE,
DEFAULT_SSL_CIPHERS
)
from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
from litellm.types.llms.custom_http import *
@ -94,10 +100,19 @@ def get_ssl_configuration(
if ssl_verify is not False:
custom_ssl_context = ssl.create_default_context(cafile=cafile)
# If security level is set, apply it to the SSL context
# Optimize SSL handshake performance
# Set minimum TLS version to 1.2 for better performance
custom_ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2
# Configure cipher suites for optimal performance
if ssl_security_level and isinstance(ssl_security_level, str):
# Create a custom SSL context with reduced security level
# User provided custom cipher configuration (e.g., via SSL_SECURITY_LEVEL env var)
custom_ssl_context.set_ciphers(ssl_security_level)
else:
# Use optimized cipher list that strongly prefers fast ciphers
# but falls back to widely compatible ones
custom_ssl_context.set_ciphers(DEFAULT_SSL_CIPHERS)
# Use our custom SSL context instead of the original ssl_verify value
return custom_ssl_context
@ -164,7 +179,7 @@ class AsyncHTTPHandler:
self,
timeout: Optional[Union[float, httpx.Timeout]] = None,
event_hooks: Optional[Mapping[str, List[Callable[..., Any]]]] = None,
concurrent_limit=1000,
concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
client_alias: Optional[str] = None, # name for client in logs
ssl_verify: Optional[VerifyTypes] = None,
shared_session: Optional["ClientSession"] = None,
@ -173,7 +188,6 @@ class AsyncHTTPHandler:
self.event_hooks = event_hooks
self.client = self.create_client(
timeout=timeout,
concurrent_limit=concurrent_limit,
event_hooks=event_hooks,
ssl_verify=ssl_verify,
shared_session=shared_session,
@ -183,7 +197,6 @@ class AsyncHTTPHandler:
def create_client(
self,
timeout: Optional[Union[float, httpx.Timeout]],
concurrent_limit: int,
event_hooks: Optional[Mapping[str, List[Callable[..., Any]]]],
ssl_verify: Optional[VerifyTypes] = None,
shared_session: Optional["ClientSession"] = None,
@ -209,10 +222,6 @@ class AsyncHTTPHandler:
transport=transport,
event_hooks=event_hooks,
timeout=timeout,
limits=httpx.Limits(
max_connections=concurrent_limit,
max_keepalive_connections=concurrent_limit,
),
verify=ssl_config,
cert=cert,
headers=headers,
@ -286,7 +295,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@ -352,7 +361,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@ -412,7 +421,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@ -471,7 +480,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@ -657,7 +666,13 @@ class AsyncHTTPHandler:
)
return LiteLLMAiohttpTransport(
client=lambda: ClientSession(
connector=TCPConnector(**connector_kwargs),
connector=TCPConnector(
limit=AIOHTTP_CONNECTOR_LIMIT,
keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
enable_cleanup_closed=True,
**connector_kwargs
),
trust_env=trust_env,
),
)
@ -680,7 +695,7 @@ class HTTPHandler:
def __init__(
self,
timeout: Optional[Union[float, httpx.Timeout]] = None,
concurrent_limit=1000,
concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
client: Optional[httpx.Client] = None,
ssl_verify: Optional[Union[bool, str]] = None,
):
@ -701,10 +716,6 @@ class HTTPHandler:
self.client = httpx.Client(
transport=transport,
timeout=timeout,
limits=httpx.Limits(
max_connections=concurrent_limit,
max_keepalive_connections=concurrent_limit,
),
verify=ssl_config,
cert=cert,
headers=headers,

5
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -13,13 +13,13 @@ from typing import (
cast,
)
from litellm._logging import verbose_logger
import httpx # type: ignore
import litellm
import litellm.litellm_core_utils
import litellm.types
import litellm.types.utils
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
from litellm.llms.base_llm.anthropic_messages.transformation import (
BaseAnthropicMessagesConfig,
@ -239,7 +239,7 @@ class BaseLLMHTTPHandler:
json_mode: bool = False,
signed_json_body: Optional[bytes] = None,
shared_session: Optional["ClientSession"] = None,
):
):
if client is None:
verbose_logger.debug(
f"Creating HTTP client with shared_session: {id(shared_session) if shared_session else None}"
@ -426,6 +426,7 @@ class BaseLLMHTTPHandler:
),
json_mode=json_mode,
signed_json_body=signed_json_body,
shared_session=shared_session,
)
if stream is True:

48
litellm/llms/litellm_proxy/responses/transformation.py Normal file
View file

@ -0,0 +1,48 @@
"""
Responses API transformation for LiteLLM Proxy provider.
LiteLLM Proxy supports the OpenAI Responses API natively when the underlying model supports it.
This config enables pass-through behavior to the proxy's /v1/responses endpoint.
"""
from typing import Optional
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import LlmProviders
class LiteLLMProxyResponsesAPIConfig(OpenAIResponsesAPIConfig):
"""
Configuration for LiteLLM Proxy Responses API support.
Extends OpenAI's config since the proxy follows OpenAI's API spec,
but uses LITELLM_PROXY_API_BASE for the base URL.
"""
@property
def custom_llm_provider(self) -> LlmProviders:
return LlmProviders.LITELLM_PROXY
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the endpoint for LiteLLM Proxy responses API.
Uses LITELLM_PROXY_API_BASE environment variable if api_base is not provided.
"""
api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
if api_base is None:
raise ValueError(
"api_base not set for LiteLLM Proxy responses API. "
"Set via api_base parameter or LITELLM_PROXY_API_BASE environment variable"
)
# Remove trailing slashes
api_base = api_base.rstrip("/")
return f"{api_base}/responses"

433
litellm/llms/oci/chat/transformation.py
View file

@ -19,6 +19,13 @@ from litellm.llms.custom_httpx.http_handler import (
)
from litellm.llms.oci.common_utils import OCIError
from litellm.types.llms.oci import (
CohereChatRequest,
CohereMessage,
CohereChatResult,
CohereParameterDefinition,
CohereStreamChunk,
CohereTool,
CohereToolCall,
OCIChatRequestPayload,
OCICompletionPayload,
OCICompletionResponse,
@ -37,13 +44,13 @@ from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import (
Delta,
LlmProviders,
ModelResponse,
ModelResponseStream,
StreamingChoices,
)
from litellm.utils import (
ChatCompletionMessageToolCall,
CustomStreamWrapper,
ModelResponse,
Usage,
)
@ -170,13 +177,16 @@ class OCIChatConfig(BaseConfig):
"web_search_options": False,
}
# Cohere and Gemini use the same parameter mapping as GENERIC
self.openai_to_oci_cohere_param_map = self.openai_to_oci_generic_param_map.copy()
def get_supported_openai_params(self, model: str) -> List[str]:
supported_params = []
vendor = get_vendor_from_model(model)
if vendor == OCIVendors.COHERE:
raise ValueError(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
)
open_ai_to_oci_param_map = self.openai_to_oci_cohere_param_map
open_ai_to_oci_param_map.pop("tool_choice")
open_ai_to_oci_param_map.pop("max_retries")
else:
open_ai_to_oci_param_map = self.openai_to_oci_generic_param_map
for key, value in open_ai_to_oci_param_map.items():
@ -195,9 +205,7 @@ class OCIChatConfig(BaseConfig):
adapted_params = {}
vendor = get_vendor_from_model(model)
if vendor == OCIVendors.COHERE:
raise ValueError(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
)
open_ai_to_oci_param_map = self.openai_to_oci_cohere_param_map
else:
open_ai_to_oci_param_map = self.openai_to_oci_generic_param_map
@ -416,21 +424,130 @@ class OCIChatConfig(BaseConfig):
def _get_optional_params(self, vendor: OCIVendors, optional_params: dict) -> Dict:
selected_params = {}
if vendor == OCIVendors.COHERE:
raise ValueError(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
)
open_ai_to_oci_param_map = self.openai_to_oci_cohere_param_map
# remove tool_choice from the map
open_ai_to_oci_param_map.pop("tool_choice")
# Add default values for Cohere API
selected_params = {
"maxTokens": 600,
"temperature": 1,
"topK": 0,
"topP": 0.75,
"frequencyPenalty": 0
}
else:
open_ai_to_oci_param_map = self.openai_to_oci_generic_param_map
for value in open_ai_to_oci_param_map.values():
if value in optional_params:
selected_params[value] = optional_params[value]
# Map OpenAI params to OCI params
for openai_key, oci_key in open_ai_to_oci_param_map.items():
if oci_key and openai_key in optional_params:
selected_params[oci_key] = optional_params[openai_key] # type: ignore[index]
# Also check for already-mapped OCI params (for backward compatibility)
for oci_value in open_ai_to_oci_param_map.values():
if oci_value and oci_value in optional_params and oci_value not in selected_params:
selected_params[oci_value] = optional_params[oci_value] # type: ignore[index]
if "tools" in selected_params:
selected_params["tools"] = adapt_tool_definition_to_oci_standard(
selected_params["tools"], vendor
)
if vendor == OCIVendors.COHERE:
selected_params["tools"] = self.adapt_tool_definitions_to_cohere_standard( # type: ignore[assignment]
selected_params["tools"] # type: ignore[arg-type]
)
else:
selected_params["tools"] = adapt_tool_definition_to_oci_standard( # type: ignore[assignment]
selected_params["tools"], vendor # type: ignore[arg-type]
)
return selected_params
def adapt_messages_to_cohere_standard(self, messages: List[AllMessageValues]) -> List[CohereMessage]:
"""Build chat history for Cohere models."""
chat_history = []
for msg in messages[:-1]: # All messages except the last one
role = msg.get("role")
content = msg.get("content")
if isinstance(content, list):
# Extract text from content array
text_content = ""
for content_item in content:
if isinstance(content_item, dict) and content_item.get("type") == "text":
text_content += content_item.get("text", "")
content = text_content
# Ensure content is a string
if not isinstance(content, str):
content = str(content) if content is not None else ""
# Handle tool calls
tool_calls: Optional[List[CohereToolCall]] = None
if role == "assistant" and "tool_calls" in msg and msg.get("tool_calls"): # type: ignore[union-attr,typeddict-item]
tool_calls = []
for tool_call in msg["tool_calls"]: # type: ignore[union-attr,typeddict-item]
# Parse arguments if they're a JSON string
raw_arguments: Any = tool_call.get("function", {}).get("arguments", {})
if isinstance(raw_arguments, str):
try:
arguments: Dict[str, Any] = json.loads(raw_arguments)
except json.JSONDecodeError:
arguments = {}
else:
arguments = raw_arguments
tool_calls.append(CohereToolCall(
name=str(tool_call.get("function", {}).get("name", "")),
parameters=arguments
))
if role == "user":
chat_history.append(CohereMessage(role="USER", message=content))
elif role == "assistant":
chat_history.append(CohereMessage(role="CHATBOT", message=content, toolCalls=tool_calls))
elif role == "tool":
# Tool messages need special handling
chat_history.append(CohereMessage(
role="TOOL",
message=content,
toolCalls=None # Tool messages don't have tool calls
))
return chat_history
def adapt_tool_definitions_to_cohere_standard(self, tools: List[Dict[str, Any]]) -> List[CohereTool]:
"""Adapt tool definitions to Cohere format."""
cohere_tools = []
for tool in tools:
function_def = tool.get("function", {})
parameters = function_def.get("parameters", {}).get("properties", {})
required = function_def.get("parameters", {}).get("required", [])
parameter_definitions = {}
for param_name, param_schema in parameters.items():
parameter_definitions[param_name] = CohereParameterDefinition(
description=param_schema.get("description", ""),
type=param_schema.get("type", "string"),
isRequired=param_name in required
)
cohere_tools.append(CohereTool(
name=function_def.get("name", ""),
description=function_def.get("description", ""),
parameterDefinitions=parameter_definitions
))
return cohere_tools
def _extract_text_content(self, content: Any) -> str:
"""Extract text content from message content."""
if isinstance(content, str):
return content
elif isinstance(content, list):
text_content = ""
for content_item in content:
if isinstance(content_item, dict) and content_item.get("type") == "text":
text_content += content_item.get("text", "")
return text_content
return str(content)
def transform_request(
self,
model: str,
@ -445,28 +562,47 @@ class OCIChatConfig(BaseConfig):
vendor = get_vendor_from_model(model)
if vendor == OCIVendors.COHERE:
oci_serving_mode = optional_params.get("oci_serving_mode", "ON_DEMAND")
if oci_serving_mode not in ["ON_DEMAND", "DEDICATED"]:
raise Exception(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
"kwarg `oci_serving_mode` must be either 'ON_DEMAND' or 'DEDICATED'"
)
if oci_serving_mode == "DEDICATED":
servingMode = OCIServingMode(
servingType="DEDICATED",
endpointId=model,
)
else:
oci_serving_mode = optional_params.get("oci_serving_mode", "ON_DEMAND")
if oci_serving_mode not in ["ON_DEMAND", "DEDICATED"]:
raise Exception(
"kwarg `oci_serving_mode` must be either 'ON_DEMAND' or 'DEDICATED'"
)
servingMode = OCIServingMode(
servingType="ON_DEMAND",
modelId=model,
)
if oci_serving_mode == "DEDICATED":
servingMode = OCIServingMode(
servingType="DEDICATED",
endpointId=model,
)
else:
servingMode = OCIServingMode(
servingType="ON_DEMAND",
modelId=model,
)
# Build request based on vendor type
if vendor == OCIVendors.COHERE:
# For Cohere, we need to use the specific Cohere format
# Extract the last user message as the main message
user_messages = [msg for msg in messages if msg.get("role") == "user"]
if not user_messages:
raise Exception("No user message found for Cohere model")
# Create Cohere-specific chat request
chat_request = CohereChatRequest(
apiFormat="COHERE",
message=self._extract_text_content(user_messages[-1]["content"]),
chatHistory=self.adapt_messages_to_cohere_standard(messages),
**self._get_optional_params(OCIVendors.COHERE, optional_params)
)
data = OCICompletionPayload(
compartmentId=oci_compartment_id,
servingMode=servingMode,
chatRequest=chat_request
)
else:
# Use generic format for other vendors
data = OCICompletionPayload(
compartmentId=oci_compartment_id,
servingMode=servingMode,
@ -479,6 +615,111 @@ class OCIChatConfig(BaseConfig):
return data.model_dump(exclude_none=True)
def _handle_cohere_response(
self,
json_response: dict,
model: str,
model_response: ModelResponse
) -> ModelResponse:
"""Handle Cohere-specific response format."""
cohere_response = CohereChatResult(**json_response)
# Cohere response format (uses camelCase)
model_id = model
# Set basic response info
model_response.model = model_id
model_response.created = int(datetime.datetime.now().timestamp())
# Extract the response text
response_text = cohere_response.chatResponse.text
oci_finish_reason = cohere_response.chatResponse.finishReason
# Map finish reason
if oci_finish_reason == "COMPLETE":
finish_reason = "stop"
elif oci_finish_reason == "MAX_TOKENS":
finish_reason = "length"
else:
finish_reason = "stop"
# Handle tool calls
tool_calls: Optional[List[Dict[str, Any]]] = None
if cohere_response.chatResponse.toolCalls:
tool_calls = []
for tool_call in cohere_response.chatResponse.toolCalls:
tool_calls.append({
"id": f"call_{len(tool_calls)}", # Generate a simple ID
"type": "function",
"function": {
"name": tool_call.name,
"arguments": json.dumps(tool_call.parameters)
}
})
# Create choice
from litellm.types.utils import Choices
choice = Choices(
index=0,
message={
"role": "assistant",
"content": response_text,
"tool_calls": tool_calls
},
finish_reason=finish_reason
)
model_response.choices = [choice]
# Extract usage info
usage_info = cohere_response.chatResponse.usage
from litellm.types.utils import Usage
model_response.usage = Usage( # type: ignore[attr-defined]
prompt_tokens=usage_info.promptTokens, # type: ignore[union-attr]
completion_tokens=usage_info.completionTokens, # type: ignore[union-attr]
total_tokens=usage_info.totalTokens # type: ignore[union-attr]
)
return model_response
def _handle_generic_response(
self,
json: dict,
model: str,
model_response: ModelResponse,
raw_response: httpx.Response
) -> ModelResponse:
"""Handle generic OCI response format."""
try:
completion_response = OCICompletionResponse(**json)
except TypeError as e:
raise OCIError(
message=f"Response cannot be casted to OCICompletionResponse: {str(e)}",
status_code=raw_response.status_code,
)
iso_str = completion_response.chatResponse.timeCreated
dt = datetime.datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
model_response.created = int(dt.timestamp())
model_response.model = completion_response.modelId
message = model_response.choices[0].message # type: ignore
response_message = completion_response.chatResponse.choices[0].message
if response_message.content and response_message.content[0].type == "TEXT":
message.content = response_message.content[0].text
if response_message.toolCalls:
message.tool_calls = adapt_tools_to_openai_standard(
response_message.toolCalls
)
usage = Usage(
prompt_tokens=completion_response.chatResponse.usage.promptTokens,
completion_tokens=completion_response.chatResponse.usage.completionTokens,
total_tokens=completion_response.chatResponse.usage.totalTokens,
)
model_response.usage = usage # type: ignore
return model_response
def transform_response(
self,
model: str,
@ -509,46 +750,13 @@ class OCIChatConfig(BaseConfig):
status_code=raw_response.status_code,
)
try:
completion_response = OCICompletionResponse(**json)
except TypeError as e:
raise OCIError(
message=f"Response cannot be casted to OCICompletionResponse: {str(e)}",
status_code=raw_response.status_code,
)
vendor = get_vendor_from_model(model)
# Handle response based on vendor type
if vendor == OCIVendors.COHERE:
raise ValueError(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
)
model_response = self._handle_cohere_response(json, model, model_response)
else:
iso_str = completion_response.chatResponse.timeCreated
dt = datetime.datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
model_response.created = int(dt.timestamp())
model_response.model = completion_response.modelId
message = model_response.choices[0].message # type: ignore
if vendor == OCIVendors.COHERE:
raise ValueError(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
)
else:
response_message = completion_response.chatResponse.choices[0].message
if response_message.content and response_message.content[0].type == "TEXT":
message.content = response_message.content[0].text
if response_message.toolCalls:
message.tool_calls = adapt_tools_to_openai_standard(
response_message.toolCalls
)
usage = Usage(
prompt_tokens=completion_response.chatResponse.usage.promptTokens,
completion_tokens=completion_response.chatResponse.usage.completionTokens,
total_tokens=completion_response.chatResponse.usage.totalTokens,
)
model_response.usage = usage # type: ignore
model_response = self._handle_generic_response(json, model, model_response, raw_response)
model_response._hidden_params["additional_headers"] = raw_response.headers
@ -818,26 +1026,21 @@ def adapt_messages_to_generic_oci_standard(
def adapt_tool_definition_to_oci_standard(tools: List[Dict], vendor: OCIVendors):
new_tools = []
if vendor == OCIVendors.COHERE:
raise ValueError(
"Cohere models are not yet supported in the litellm OCI chat completion endpoint. Use the Cohere API directly."
for tool in tools:
if tool["type"] != "function":
raise Exception("OCI only supports function tools")
tool_function = tool.get("function")
if not isinstance(tool_function, dict):
raise Exception("Prop `function` is not a dictionary")
new_tool = OCIToolDefinition(
type="FUNCTION",
name=tool_function.get("name"),
description=tool_function.get("description", ""),
parameters=tool_function.get("parameters", {}),
)
else:
for tool in tools:
if tool["type"] != "function":
raise Exception("OCI only supports function tools")
tool_function = tool.get("function")
if not isinstance(tool_function, dict):
raise Exception("Prop `function` is not a dictionary")
new_tool = OCIToolDefinition(
type="FUNCTION",
name=tool_function.get("name"),
description=tool_function.get("description", ""),
parameters=tool_function.get("parameters", {}),
)
new_tools.append(new_tool)
new_tools.append(new_tool)
return new_tools
@ -877,6 +1080,58 @@ class OCIStreamWrapper(CustomStreamWrapper):
if not chunk.startswith("data:"):
raise ValueError(f"Chunk does not start with 'data:': {chunk}")
dict_chunk = json.loads(chunk[5:]) # Remove 'data: ' prefix and parse JSON
# Check if this is a Cohere stream chunk
if "apiFormat" in dict_chunk and dict_chunk.get("apiFormat") == "COHERE":
return self._handle_cohere_stream_chunk(dict_chunk)
else:
return self._handle_generic_stream_chunk(dict_chunk)
def _handle_cohere_stream_chunk(self, dict_chunk: dict):
"""Handle Cohere-specific streaming chunks."""
try:
typed_chunk = CohereStreamChunk(**dict_chunk)
except TypeError as e:
raise ValueError(f"Chunk cannot be casted to CohereStreamChunk: {str(e)}")
if typed_chunk.index is None:
typed_chunk.index = 0
# Extract text content
text = typed_chunk.text or ""
# Map finish reason to standard format
finish_reason = typed_chunk.finishReason
if finish_reason == "COMPLETE":
finish_reason = "stop"
elif finish_reason == "MAX_TOKENS":
finish_reason = "length"
elif finish_reason is None:
finish_reason = None
else:
finish_reason = "stop"
# For Cohere, we don't have tool calls in the streaming format
tool_calls = None
return ModelResponseStream(
choices=[
StreamingChoices(
index=typed_chunk.index if typed_chunk.index else 0,
delta=Delta(
content=text,
tool_calls=tool_calls,
provider_specific_fields=None,
thinking_blocks=None,
reasoning_content=None,
),
finish_reason=finish_reason,
)
]
)
def _handle_generic_stream_chunk(self, dict_chunk: dict):
"""Handle generic OCI streaming chunks."""
try:
typed_chunk = OCIStreamChunk(**dict_chunk)
except TypeError as e:

1
litellm/llms/openai/chat/gpt_5_transformation.py
View file

@ -41,6 +41,7 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
"presence_penalty",
"frequency_penalty",
"top_logprobs",
"stop",
]
return [

8
litellm/llms/openai/chat/gpt_transformation.py
View file

@ -397,13 +397,13 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
)
from litellm.types.llms.openai import ChatCompletionToolParam
for message in messages:
message = cast(
for i, message in enumerate(messages):
messages[i] = cast(
AllMessageValues, filter_value_from_dict(message, "cache_control") # type: ignore
)
if tools is not None:
for tool in tools:
tool = cast(
for i, tool in enumerate(tools):
tools[i] = cast(
ChatCompletionToolParam,
filter_value_from_dict(tool, "cache_control"), # type: ignore
)

2
litellm/llms/openai/common_utils.py
View file

@ -207,7 +207,6 @@ class BaseOpenAILLM:
ssl_config = get_ssl_configuration()
return httpx.AsyncClient(
limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100),
verify=ssl_config,
transport=AsyncHTTPHandler._create_async_transport(
ssl_context=ssl_config
@ -228,7 +227,6 @@ class BaseOpenAILLM:
ssl_config = get_ssl_configuration()
return httpx.Client(
limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100),
verify=ssl_config,
follow_redirects=True,
)

4
litellm/llms/openai/openai.py
View file

@ -1125,6 +1125,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
api_base: Optional[str] = None,
client: Optional[AsyncOpenAI] = None,
max_retries=None,
shared_session: Optional["ClientSession"] = None,
):
try:
openai_aclient: AsyncOpenAI = self._get_openai_client( # type: ignore
@ -1134,6 +1135,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
timeout=timeout,
max_retries=max_retries,
client=client,
shared_session=shared_session,
)
headers, response = await self.make_openai_embedding_request(
openai_aclient=openai_aclient,
@ -1197,6 +1199,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
client=None,
aembedding=None,
max_retries: Optional[int] = None,
shared_session: Optional["ClientSession"] = None,
) -> EmbeddingResponse:
super().embedding()
try:
@ -1223,6 +1226,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
timeout=timeout,
client=client,
max_retries=max_retries,
shared_session=shared_session,
)
openai_client: OpenAI = self._get_openai_client( # type: ignore

16
litellm/llms/openai/responses/transformation.py
View file

@ -1,12 +1,4 @@
from typing import (
TYPE_CHECKING,
Any,
Dict,
Optional,
Union,
cast,
get_type_hints,
)
from typing import TYPE_CHECKING, Any, Dict, Optional, Union, cast, get_type_hints
import httpx
from openai.types.responses import ResponseReasoningItem
@ -127,13 +119,12 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
2. Create a ResponseReasoningItem object with the item data
3. Convert it back to dict with exclude_none=True to filter None values
"""
verbose_logger.debug(f"Handling reasoning item: {item}")
if item.get("type") == "reasoning":
try:
# Ensure required fields are present for ResponseReasoningItem
item_data = dict(item)
if "id" not in item_data:
item_data["id"] = f"reasoning_{hash(str(item_data))}"
item_data["id"] = f"rs_{hash(str(item_data))}"
if "summary" not in item_data:
item_data["summary"] = (
item_data.get("reasoning_content", "")[:100] + "..."
@ -178,7 +169,7 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
raise OpenAIError(
message=raw_response.text, status_code=raw_response.status_code
)
return ResponsesAPIResponse(**raw_response_json)
return ResponsesAPIResponse.model_construct(**raw_response_json)
def validate_environment(
self, headers: dict, model: str, litellm_params: Optional[GenericLiteLLMParams]
@ -280,6 +271,7 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
ResponsesAPIStreamEvents.MCP_CALL_ARGUMENTS_DONE: MCPCallArgumentsDoneEvent,
ResponsesAPIStreamEvents.MCP_CALL_COMPLETED: MCPCallCompletedEvent,
ResponsesAPIStreamEvents.MCP_CALL_FAILED: MCPCallFailedEvent,
ResponsesAPIStreamEvents.IMAGE_GENERATION_PARTIAL_IMAGE: ImageGenerationPartialImageEvent,
ResponsesAPIStreamEvents.ERROR: ErrorEvent,
}

71
litellm/llms/openrouter/chat/transformation.py
View file

@ -6,7 +6,8 @@ Calls done in OpenAI/openai.py as OpenRouter is openai-compatible.
Docs: https://openrouter.ai/docs/parameters
"""
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union
from enum import Enum
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union, cast
import httpx
@ -20,6 +21,12 @@ from ...openai.chat.gpt_transformation import OpenAIGPTConfig
from ..common_utils import OpenRouterException
class CacheControlSupportedModels(str, Enum):
"""Models that support cache_control in content blocks."""
CLAUDE = "claude"
GEMINI = "gemini"
class OpenrouterConfig(OpenAIGPTConfig):
def map_openai_params(
self,
@ -48,19 +55,76 @@ class OpenrouterConfig(OpenAIGPTConfig):
)
return mapped_openai_params
def _supports_cache_control_in_content(self, model: str) -> bool:
"""
Check if the model supports cache_control in content blocks.
Returns:
bool: True if model supports cache_control (Claude or Gemini models)
"""
model_lower = model.lower()
return any(
supported_model.value in model_lower
for supported_model in CacheControlSupportedModels
)
def remove_cache_control_flag_from_messages_and_tools(
self,
model: str,
messages: List[AllMessageValues],
tools: Optional[List["ChatCompletionToolParam"]] = None,
) -> Tuple[List[AllMessageValues], Optional[List["ChatCompletionToolParam"]]]:
if "claude" in model.lower(): # don't remove 'cache_control' flag
if self._supports_cache_control_in_content(model):
return messages, tools
else:
return super().remove_cache_control_flag_from_messages_and_tools(
model, messages, tools
)
def _move_cache_control_to_content(
self, messages: List[AllMessageValues]
) -> List[AllMessageValues]:
"""
Move cache_control from message level to content blocks.
OpenRouter requires cache_control to be inside content blocks, not at message level.
To avoid exceeding Anthropic's limit of 4 cache breakpoints, cache_control is only
added to the LAST content block in each message.
"""
transformed_messages: List[AllMessageValues] = []
for message in messages:
message_dict = dict(message)
cache_control = message_dict.pop("cache_control", None)
if cache_control is not None:
content = message_dict.get("content")
if isinstance(content, list):
# Content is already a list, add cache_control only to the last block
if len(content) > 0:
content_copy = []
for i, block in enumerate(content):
block_dict = dict(block)
# Only add cache_control to the last content block
if i == len(content) - 1:
block_dict["cache_control"] = cache_control
content_copy.append(block_dict)
message_dict["content"] = content_copy
else:
# Content is a string, convert to structured format
message_dict["content"] = [
{
"type": "text",
"text": content,
"cache_control": cache_control,
}
]
# Cast back to AllMessageValues after modification
transformed_messages.append(cast(AllMessageValues, message_dict))
return transformed_messages
def transform_request(
self,
model: str,
@ -75,6 +139,9 @@ class OpenrouterConfig(OpenAIGPTConfig):
Returns:
dict: The transformed request. Sent as the body of the API call.
"""
if self._supports_cache_control_in_content(model):
messages = self._move_cache_control_to_content(messages)
extra_body = optional_params.pop("extra_body", {})
response = super().transform_request(
model, messages, optional_params, litellm_params, headers

196
litellm/llms/snowflake/chat/transformation.py
View file

@ -1,14 +1,15 @@
"""
Support for Snowflake REST API
Support for Snowflake REST API
"""
from typing import TYPE_CHECKING, Any, List, Optional, Tuple
import json
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
from litellm.types.utils import ChatCompletionMessageToolCall, Function, ModelResponse
from ...openai_like.chat.transformation import OpenAIGPTConfig
@ -22,15 +23,25 @@ else:
class SnowflakeConfig(OpenAIGPTConfig):
"""
source: https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex
Reference: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-llm-rest-api
Snowflake Cortex LLM REST API supports function calling with specific models (e.g., Claude 3.5 Sonnet).
This config handles transformation between OpenAI format and Snowflake's tool_spec format.
"""
@classmethod
def get_config(cls):
return super().get_config()
def get_supported_openai_params(self, model: str) -> List:
return ["temperature", "max_tokens", "top_p", "response_format"]
def get_supported_openai_params(self, model: str) -> List[str]:
return [
"temperature",
"max_tokens",
"top_p",
"response_format",
"tools",
"tool_choice",
]
def map_openai_params(
self,
@ -56,6 +67,57 @@ class SnowflakeConfig(OpenAIGPTConfig):
optional_params[param] = value
return optional_params
def _transform_tool_calls_from_snowflake_to_openai(
self, content_list: List[Dict[str, Any]]
) -> Tuple[str, Optional[List[ChatCompletionMessageToolCall]]]:
"""
Transform Snowflake tool calls to OpenAI format.
Args:
content_list: Snowflake's content_list array containing text and tool_use items
Returns:
Tuple of (text_content, tool_calls)
Snowflake format in content_list:
{
"type": "tool_use",
"tool_use": {
"tool_use_id": "tooluse_...",
"name": "get_weather",
"input": {"location": "Paris"}
}
}
OpenAI format (returned tool_calls):
ChatCompletionMessageToolCall(
id="tooluse_...",
type="function",
function=Function(name="get_weather", arguments='{"location": "Paris"}')
)
"""
text_content = ""
tool_calls: List[ChatCompletionMessageToolCall] = []
for idx, content_item in enumerate(content_list):
if content_item.get("type") == "text":
text_content += content_item.get("text", "")
## TOOL CALLING
elif content_item.get("type") == "tool_use":
tool_use_data = content_item.get("tool_use", {})
tool_call = ChatCompletionMessageToolCall(
id=tool_use_data.get("tool_use_id", ""),
type="function",
function=Function(
name=tool_use_data.get("name", ""),
arguments=json.dumps(tool_use_data.get("input", {})),
),
)
tool_calls.append(tool_call)
return text_content, tool_calls if tool_calls else None
def transform_response(
self,
model: str,
@ -71,6 +133,7 @@ class SnowflakeConfig(OpenAIGPTConfig):
json_mode: Optional[bool] = None,
) -> ModelResponse:
response_json = raw_response.json()
logging_obj.post_call(
input=messages,
api_key="",
@ -78,6 +141,26 @@ class SnowflakeConfig(OpenAIGPTConfig):
additional_args={"complete_input_dict": request_data},
)
## RESPONSE TRANSFORMATION
# Snowflake returns content_list (not content) with tool_use objects
# We need to transform this to OpenAI's format with content + tool_calls
if "choices" in response_json and len(response_json["choices"]) > 0:
choice = response_json["choices"][0]
if "message" in choice and "content_list" in choice["message"]:
content_list = choice["message"]["content_list"]
(
text_content,
tool_calls,
) = self._transform_tool_calls_from_snowflake_to_openai(content_list)
# Update the choice message with OpenAI format
choice["message"]["content"] = text_content
if tool_calls:
choice["message"]["tool_calls"] = tool_calls
# Remove Snowflake-specific content_list
del choice["message"]["content_list"]
returned_response = ModelResponse(**response_json)
returned_response.model = "snowflake/" + (returned_response.model or "")
@ -150,6 +233,95 @@ class SnowflakeConfig(OpenAIGPTConfig):
return api_base
def _transform_tools(self, tools: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""
Transform OpenAI tool format to Snowflake tool format.
Args:
tools: List of tools in OpenAI format
Returns:
List of tools in Snowflake format
OpenAI format:
{
"type": "function",
"function": {
"name": "get_weather",
"description": "...",
"parameters": {...}
}
}
Snowflake format:
{
"tool_spec": {
"type": "generic",
"name": "get_weather",
"description": "...",
"input_schema": {...}
}
}
"""
snowflake_tools: List[Dict[str, Any]] = []
for tool in tools:
if tool.get("type") == "function":
function = tool.get("function", {})
snowflake_tool: Dict[str, Any] = {
"tool_spec": {
"type": "generic",
"name": function.get("name"),
"input_schema": function.get(
"parameters",
{"type": "object", "properties": {}},
),
}
}
# Add description if present
if "description" in function:
snowflake_tool["tool_spec"]["description"] = function[
"description"
]
snowflake_tools.append(snowflake_tool)
return snowflake_tools
def _transform_tool_choice(
self, tool_choice: Union[str, Dict[str, Any]]
) -> Union[str, Dict[str, Any]]:
"""
Transform OpenAI tool_choice format to Snowflake format.
Args:
tool_choice: Tool choice in OpenAI format (str or dict)
Returns:
Tool choice in Snowflake format
OpenAI format:
{"type": "function", "function": {"name": "get_weather"}}
Snowflake format:
{"type": "tool", "name": ["get_weather"]}
Note: String values ("auto", "required", "none") pass through unchanged.
"""
if isinstance(tool_choice, str):
# "auto", "required", "none" pass through as-is
return tool_choice
if isinstance(tool_choice, dict):
if tool_choice.get("type") == "function":
function_name = tool_choice.get("function", {}).get("name")
if function_name:
return {
"type": "tool",
"name": [function_name], # Snowflake expects array
}
return tool_choice
def transform_request(
self,
model: str,
@ -160,6 +332,18 @@ class SnowflakeConfig(OpenAIGPTConfig):
) -> dict:
stream: bool = optional_params.pop("stream", None) or False
extra_body = optional_params.pop("extra_body", {})
## TOOL CALLING
# Transform tools from OpenAI format to Snowflake's tool_spec format
tools = optional_params.pop("tools", None)
if tools:
optional_params["tools"] = self._transform_tools(tools)
# Transform tool_choice from OpenAI format to Snowflake's tool name array format
tool_choice = optional_params.pop("tool_choice", None)
if tool_choice:
optional_params["tool_choice"] = self._transform_tool_choice(tool_choice)
return {
"model": model,
"messages": messages,

63
litellm/llms/vertex_ai/common_utils.py
View file

@ -1,4 +1,5 @@
import re
from enum import Enum
from typing import Any, Dict, List, Literal, Optional, Set, Tuple, Union, get_type_hints
import httpx
@ -24,6 +25,68 @@ class VertexAIError(BaseLLMException):
super().__init__(message=message, status_code=status_code, headers=headers)
class VertexAIModelRoute(str, Enum):
"""Enum for Vertex AI model routing"""
PARTNER_MODELS = "partner_models"
GEMINI = "gemini"
GEMMA = "gemma"
MODEL_GARDEN = "model_garden"
NON_GEMINI = "non_gemini"
def get_vertex_ai_model_route(model: str, litellm_params: Optional[dict] = None) -> VertexAIModelRoute:
"""
Determine which handler to use for a Vertex AI model based on the model name.
Args:
model: The model name (e.g., "llama3-405b", "gemini-pro", "gemma/gemma-3-12b-it", "openai/gpt-oss-120b")
litellm_params: Optional litellm parameters dict that may contain base_model for routing
Returns:
VertexAIModelRoute: The route enum indicating which handler should be used
Examples:
>>> get_vertex_ai_model_route("llama3-405b")
VertexAIModelRoute.PARTNER_MODELS
>>> get_vertex_ai_model_route("gemini-pro")
VertexAIModelRoute.GEMINI
>>> get_vertex_ai_model_route("gemma/gemma-3-12b-it")
VertexAIModelRoute.GEMMA
>>> get_vertex_ai_model_route("openai/gpt-oss-120b")
VertexAIModelRoute.MODEL_GARDEN
"""
from litellm.llms.vertex_ai.vertex_ai_partner_models.main import (
VertexAIPartnerModels,
)
# Check base_model in litellm_params for gemini override
if litellm_params and litellm_params.get("base_model") is not None:
if "gemini" in litellm_params["base_model"]:
return VertexAIModelRoute.GEMINI
# Check for partner models (llama, mistral, claude, etc.)
if VertexAIPartnerModels.is_vertex_partner_model(model=model):
return VertexAIModelRoute.PARTNER_MODELS
# Check for gemma models
if "gemma/" in model:
return VertexAIModelRoute.GEMMA
# Check for model garden openai models
if "openai" in model:
return VertexAIModelRoute.MODEL_GARDEN
# Check for gemini models
if "gemini" in model:
return VertexAIModelRoute.GEMINI
# Default to non-gemini (legacy vertex models like chat-bison, text-bison, etc.)
return VertexAIModelRoute.NON_GEMINI
def get_supports_system_message(
model: str, custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"]
) -> bool:

18
litellm/llms/vertex_ai/context_caching/transformation.py
View file

@ -5,7 +5,7 @@ Why separate file? Make it easy to see how transformation works
"""
import re
from typing import List, Optional, Tuple
from typing import List, Optional, Tuple, Literal
from litellm.types.llms.openai import AllMessageValues
from litellm.types.llms.vertex_ai import CachedContentRequestBody
@ -155,13 +155,18 @@ def separate_cached_messages(
def transform_openai_messages_to_gemini_context_caching(
model: str, messages: List[AllMessageValues], cache_key: str
model: str,
messages: List[AllMessageValues],
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
cache_key: str,
vertex_project: Optional[str],
vertex_location: Optional[str],
) -> CachedContentRequestBody:
# Extract TTL from cached messages BEFORE system message transformation
ttl = extract_ttl_from_cached_messages(messages)
supports_system_message = get_supports_system_message(
model=model, custom_llm_provider="gemini"
model=model, custom_llm_provider=custom_llm_provider
)
transformed_system_messages, new_messages = _transform_system_message(
@ -170,9 +175,14 @@ def transform_openai_messages_to_gemini_context_caching(
transformed_messages = _gemini_convert_messages_with_history(messages=new_messages)
model_name = "models/{}".format(model)
if custom_llm_provider == "vertex_ai" or custom_llm_provider == "vertex_ai_beta":
model_name = f"projects/{vertex_project}/locations/{vertex_location}/publishers/google/{model_name}"
data = CachedContentRequestBody(
contents=transformed_messages,
model="models/{}".format(model),
model=model_name,
displayName=cache_key,
)

73
litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
View file

@ -41,8 +41,11 @@ class ContextCachingEndpoints(VertexBase):
def _get_token_and_url_context_caching(
self,
gemini_api_key: Optional[str],
custom_llm_provider: Literal["gemini"],
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
api_base: Optional[str],
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str],
) -> Tuple[Optional[str], str]:
"""
Internal function. Returns the token and url for the call.
@ -58,9 +61,15 @@ class ContextCachingEndpoints(VertexBase):
url = "https://generativelanguage.googleapis.com/v1beta/{}?key={}".format(
endpoint, gemini_api_key
)
elif custom_llm_provider == "vertex_ai":
auth_header = vertex_auth_header
endpoint = "cachedContents"
url = f"https://{vertex_location}-aiplatform.googleapis.com/v1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
else:
raise NotImplementedError
auth_header = vertex_auth_header
endpoint = "cachedContents"
url = f"https://{vertex_location}-aiplatform.googleapis.com/v1beta1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
return self._check_custom_proxy(
api_base=api_base,
@ -80,6 +89,10 @@ class ContextCachingEndpoints(VertexBase):
api_key: str,
api_base: Optional[str],
logging_obj: Logging,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str],
) -> Optional[str]:
"""
Checks if content already cached.
@ -94,8 +107,11 @@ class ContextCachingEndpoints(VertexBase):
_, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
custom_llm_provider="gemini",
custom_llm_provider=custom_llm_provider,
api_base=api_base,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header
)
try:
## LOGGING
@ -145,6 +161,10 @@ class ContextCachingEndpoints(VertexBase):
api_key: str,
api_base: Optional[str],
logging_obj: Logging,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str]
) -> Optional[str]:
"""
Checks if content already cached.
@ -159,8 +179,11 @@ class ContextCachingEndpoints(VertexBase):
_, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
custom_llm_provider="gemini",
custom_llm_provider=custom_llm_provider,
api_base=api_base,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header
)
try:
## LOGGING
@ -212,6 +235,10 @@ class ContextCachingEndpoints(VertexBase):
client: Optional[HTTPHandler],
timeout: Optional[Union[float, httpx.Timeout]],
logging_obj: Logging,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str],
extra_headers: Optional[dict] = None,
cached_content: Optional[str] = None,
) -> Tuple[List[AllMessageValues], dict, Optional[str]]:
@ -240,8 +267,11 @@ class ContextCachingEndpoints(VertexBase):
## AUTHORIZATION ##
token, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
custom_llm_provider="gemini",
custom_llm_provider=custom_llm_provider,
api_base=api_base,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header
)
headers = {
@ -273,6 +303,10 @@ class ContextCachingEndpoints(VertexBase):
api_key=api_key,
api_base=api_base,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header
)
if google_cache_name:
return non_cached_messages, optional_params, google_cache_name
@ -280,7 +314,12 @@ class ContextCachingEndpoints(VertexBase):
## TRANSFORM REQUEST
cached_content_request_body = (
transform_openai_messages_to_gemini_context_caching(
model=model, messages=cached_messages, cache_key=generated_cache_key
model=model,
messages=cached_messages,
cache_key=generated_cache_key,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
)
)
@ -328,6 +367,10 @@ class ContextCachingEndpoints(VertexBase):
client: Optional[AsyncHTTPHandler],
timeout: Optional[Union[float, httpx.Timeout]],
logging_obj: Logging,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str],
extra_headers: Optional[dict] = None,
cached_content: Optional[str] = None,
) -> Tuple[List[AllMessageValues], dict, Optional[str]]:
@ -356,8 +399,11 @@ class ContextCachingEndpoints(VertexBase):
## AUTHORIZATION ##
token, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
custom_llm_provider="gemini",
custom_llm_provider=custom_llm_provider,
api_base=api_base,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header
)
headers = {
@ -386,6 +432,10 @@ class ContextCachingEndpoints(VertexBase):
api_key=api_key,
api_base=api_base,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header
)
if google_cache_name:
@ -394,7 +444,12 @@ class ContextCachingEndpoints(VertexBase):
## TRANSFORM REQUEST
cached_content_request_body = (
transform_openai_messages_to_gemini_context_caching(
model=model, messages=cached_messages, cache_key=generated_cache_key
model=model,
messages=cached_messages,
cache_key=generated_cache_key,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
)
)

1
litellm/llms/vertex_ai/cost_calculator.py
View file

@ -44,6 +44,7 @@ def cost_router(
or "mistral" in model
or "jamba" in model
or "codestral" in model
or "gemma" in model
):
return "cost_per_token"
elif custom_llm_provider == "vertex_ai" and (

93
litellm/llms/vertex_ai/gemini/transformation.py
View file

@ -514,34 +514,35 @@ def sync_transform_request_body(
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
litellm_params: dict,
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str],
) -> RequestBody:
from ..context_caching.vertex_ai_context_caching import ContextCachingEndpoints
context_caching_endpoints = ContextCachingEndpoints()
if gemini_api_key is not None:
(
messages,
optional_params,
cached_content,
) = context_caching_endpoints.check_and_create_cache(
messages=messages,
optional_params=optional_params,
api_key=gemini_api_key,
api_base=api_base,
model=model,
client=client,
timeout=timeout,
extra_headers=extra_headers,
cached_content=optional_params.pop("cached_content", None),
logging_obj=logging_obj,
)
else: # [TODO] implement context caching for gemini as well
cached_content = None
if "cached_content" in optional_params:
cached_content = optional_params.pop("cached_content")
elif "cachedContent" in optional_params:
cached_content = optional_params.pop("cachedContent")
(
messages,
optional_params,
cached_content,
) = context_caching_endpoints.check_and_create_cache(
messages=messages,
optional_params=optional_params,
api_key=gemini_api_key or "dummy",
api_base=api_base,
model=model,
client=client,
timeout=timeout,
extra_headers=extra_headers,
cached_content=optional_params.pop("cached_content", None),
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header,
)
return _transform_request_body(
messages=messages,
@ -565,34 +566,34 @@ async def async_transform_request_body(
logging_obj: litellm.litellm_core_utils.litellm_logging.Logging, # type: ignore
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
litellm_params: dict,
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_auth_header: Optional[str],
) -> RequestBody:
from ..context_caching.vertex_ai_context_caching import ContextCachingEndpoints
context_caching_endpoints = ContextCachingEndpoints()
if gemini_api_key is not None:
(
messages,
optional_params,
cached_content,
) = await context_caching_endpoints.async_check_and_create_cache(
messages=messages,
optional_params=optional_params,
api_key=gemini_api_key,
api_base=api_base,
model=model,
client=client,
timeout=timeout,
extra_headers=extra_headers,
cached_content=optional_params.pop("cached_content", None),
logging_obj=logging_obj,
)
else: # [TODO] implement context caching for gemini as well
cached_content = None
if "cached_content" in optional_params:
cached_content = optional_params.pop("cached_content")
elif "cachedContent" in optional_params:
cached_content = optional_params.pop("cachedContent")
(
messages,
optional_params,
cached_content,
) = await context_caching_endpoints.async_check_and_create_cache(
messages=messages,
optional_params=optional_params,
api_key=gemini_api_key or "dummy",
api_base=api_base,
model=model,
client=client,
timeout=timeout,
extra_headers=extra_headers,
cached_content=optional_params.pop("cached_content", None),
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header,
)
return _transform_request_body(
messages=messages,

21
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -1792,7 +1792,6 @@ class VertexLLM(VertexBase):
gemini_api_key: Optional[str] = None,
extra_headers: Optional[dict] = None,
) -> CustomStreamWrapper:
request_body = await async_transform_request_body(**data) # type: ignore
should_use_v1beta1_features = self.is_using_v1beta1_features(
optional_params=optional_params
@ -1826,6 +1825,13 @@ class VertexLLM(VertexBase):
litellm_params=litellm_params,
)
request_body = await async_transform_request_body(
**data,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=auth_header) # type: ignore
## LOGGING
logging_obj.pre_call(
input=messages,
@ -1913,7 +1919,12 @@ class VertexLLM(VertexBase):
litellm_params=litellm_params,
)
request_body = await async_transform_request_body(**data) # type: ignore
request_body = await async_transform_request_body(
**data,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=auth_header) # type: ignore
_async_client_params = {}
if timeout:
_async_client_params["timeout"] = timeout
@ -2088,7 +2099,11 @@ class VertexLLM(VertexBase):
)
## TRANSFORMATION ##
data = sync_transform_request_body(**transform_request_params)
data = sync_transform_request_body(
**transform_request_params,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=auth_header)
## LOGGING
logging_obj.pre_call(

2
litellm/llms/vertex_ai/vertex_gemma_models/__init__.py Normal file
View file

@ -0,0 +1,2 @@
"""Vertex AI Gemma-AI Models Handler"""

145
litellm/llms/vertex_ai/vertex_gemma_models/main.py Normal file
View file

@ -0,0 +1,145 @@
"""
API Handler for calling Vertex AI Gemma Models
These models use a custom prediction endpoint format that wraps messages in 'instances'
with @requestFormat: "chatCompletions" and returns responses wrapped in 'predictions'.
Usage:
response = litellm.completion(
model="vertex_ai/gemma/gemma-3-12b-it-1222199011122",
messages=[{"role": "user", "content": "What is machine learning?"}],
vertex_project="your-project-id",
vertex_location="us-central1",
)
Sent to this route when `model` is in the format `vertex_ai/gemma/{MODEL_NAME}`
The API expects a custom endpoint URL format:
https://{ENDPOINT_NUMBER}.{location}-{REGION_NUMBER}.prediction.vertexai.goog/v1/projects/{PROJECT_ID}/locations/{location}/endpoints/{ENDPOINT_ID}:predict
"""
from typing import Callable, Optional, Union
import httpx # type: ignore
from litellm.utils import ModelResponse
from ..common_utils import VertexAIError
from ..vertex_llm_base import VertexBase
class VertexAIGemmaModels(VertexBase):
def __init__(self) -> None:
pass
def completion(
self,
model: str,
messages: list,
model_response: ModelResponse,
print_verbose: Callable,
encoding,
logging_obj,
api_base: Optional[str],
optional_params: dict,
custom_prompt_dict: dict,
headers: Optional[dict],
timeout: Union[float, httpx.Timeout],
litellm_params: dict,
vertex_project=None,
vertex_location=None,
vertex_credentials=None,
logger_fn=None,
acompletion: bool = False,
client=None,
):
"""
Handles calling Vertex AI Gemma Models
Sent to this route when `model` is in the format `vertex_ai/gemma/{MODEL_NAME}`
"""
try:
import vertexai
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexLLM,
)
from litellm.llms.vertex_ai.vertex_gemma_models.transformation import (
VertexGemmaConfig,
)
except Exception as e:
raise VertexAIError(
status_code=400,
message=f"""vertexai import failed please run `pip install -U "google-cloud-aiplatform>=1.38"`. Got error: {e}""",
)
if not (
hasattr(vertexai, "preview") or hasattr(vertexai.preview, "language_models")
):
raise VertexAIError(
status_code=400,
message="""Upgrade vertex ai. Run `pip install "google-cloud-aiplatform>=1.38"`""",
)
try:
model = model.replace("gemma/", "")
vertex_httpx_logic = VertexLLM()
access_token, project_id = vertex_httpx_logic._ensure_access_token(
credentials=vertex_credentials,
project_id=vertex_project,
custom_llm_provider="vertex_ai",
)
gemma_transformation = VertexGemmaConfig()
## CONSTRUCT API BASE
stream: bool = optional_params.get("stream", False) or False
optional_params["stream"] = stream
# If api_base is not provided, it should be set as an environment variable
# or passed explicitly because the endpoint URL is unique per deployment
if api_base is None:
raise VertexAIError(
status_code=400,
message="api_base is required for Vertex AI Gemma models. Please provide the full endpoint URL.",
)
# Check if we need to append :predict
if not api_base.endswith(":predict"):
_, api_base = self._check_custom_proxy(
api_base=api_base,
custom_llm_provider="vertex_ai",
gemini_api_key=None,
endpoint="predict",
stream=stream,
auth_header=None,
url=api_base,
)
# If api_base already ends with :predict, use it as-is
# Use the custom transformation handler for gemma models
return gemma_transformation.completion(
model=model,
messages=messages,
api_base=api_base,
api_key=access_token,
custom_prompt_dict=custom_prompt_dict,
model_response=model_response,
print_verbose=print_verbose,
logging_obj=logging_obj,
optional_params=optional_params,
acompletion=acompletion,
litellm_params=litellm_params,
logger_fn=logger_fn,
client=client,
timeout=timeout,
encoding=encoding,
custom_llm_provider="vertex_ai",
)
except Exception as e:
if hasattr(e, "status_code"):
raise e
raise VertexAIError(status_code=500, message=str(e))

354
litellm/llms/vertex_ai/vertex_gemma_models/transformation.py Normal file
View file

@ -0,0 +1,354 @@
"""
Transformation logic for Vertex AI Gemma Models
Handles the custom request/response format:
- Request: Wraps messages in 'instances' with @requestFormat: "chatCompletions"
- Response: Extracts data from 'predictions' wrapper
The actual message transformation reuses OpenAIGPTConfig since Gemma uses OpenAI-compatible format.
"""
from typing import Any, Callable, Dict, List, Optional, Union, cast
import httpx
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
class VertexGemmaConfig(OpenAIGPTConfig):
"""
Configuration and transformation class for Vertex AI Gemma models
Extends OpenAIGPTConfig to wrap/unwrap the instances/predictions format
used by Vertex AI's Gemma deployment endpoint.
"""
def __init__(self) -> None:
super().__init__()
def should_fake_stream(
self,
model: Optional[str],
stream: Optional[bool],
custom_llm_provider: Optional[str] = None,
) -> bool:
"""
Vertex AI Gemma models do not support streaming.
Return True to enable fake streaming on the client side.
"""
return True
def _handle_fake_stream_response(
self,
model_response: ModelResponse,
stream: bool,
) -> Union[ModelResponse, Any]:
"""
Helper method to return fake stream iterator if streaming is requested.
Args:
model_response: The completed model response
stream: Whether streaming was requested
Returns:
MockResponseIterator if stream=True, otherwise the model_response
"""
if stream:
from litellm.llms.base_llm.base_model_iterator import MockResponseIterator
return MockResponseIterator(model_response=model_response)
return model_response
def transform_request(
self,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform request to Vertex Gemma format.
Uses parent class to create OpenAI-compatible request, then wraps it
in the Vertex Gemma instances format.
"""
# Get the base OpenAI request from parent class
openai_request = super().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
# Remove params not needed/supported by Vertex Gemma
openai_request.pop("model", None)
openai_request.pop("stream", None) # Streaming not supported, will be faked client-side
openai_request.pop("stream_options", None) # Stream options not supported
# Wrap in Vertex Gemma format
return {
"instances": [
{
"@requestFormat": "chatCompletions",
**openai_request,
}
]
}
def _unwrap_predictions_response(
self,
response_json: Dict[str, Any],
) -> Dict[str, Any]:
"""
Unwrap the Vertex Gemma predictions format to OpenAI format.
Vertex Gemma wraps the OpenAI-compatible response in a 'predictions' field.
This method extracts it so the parent class can process it normally.
"""
if "predictions" not in response_json:
raise BaseLLMException(
status_code=422,
message="Invalid response format: missing 'predictions' field",
)
return response_json["predictions"]
def completion(
self,
model: str,
messages: list,
api_base: str,
api_key: str,
custom_prompt_dict: dict,
model_response: ModelResponse,
print_verbose: Callable,
logging_obj: Any,
optional_params: dict,
acompletion: bool,
litellm_params: dict,
logger_fn: Optional[Callable] = None,
client: Optional[httpx.Client] = None,
timeout: Optional[Union[float, httpx.Timeout]] = None,
encoding=None,
custom_llm_provider: str = "vertex_ai",
):
"""
Make completion request to Vertex Gemma endpoint.
Supports both sync and async requests with fake streaming.
"""
if acompletion:
return self._async_completion(
model=model,
messages=messages,
api_base=api_base,
api_key=api_key,
model_response=model_response,
print_verbose=print_verbose,
logging_obj=logging_obj,
optional_params=optional_params,
litellm_params=litellm_params,
timeout=timeout,
encoding=encoding,
)
else:
return self._sync_completion(
model=model,
messages=messages,
api_base=api_base,
api_key=api_key,
model_response=model_response,
print_verbose=print_verbose,
logging_obj=logging_obj,
optional_params=optional_params,
litellm_params=litellm_params,
timeout=timeout,
encoding=encoding,
)
def _sync_completion(
self,
model: str,
messages: list,
api_base: str,
api_key: str,
model_response: ModelResponse,
print_verbose: Callable,
logging_obj: Any,
optional_params: dict,
litellm_params: dict,
timeout: Optional[Union[float, httpx.Timeout]],
encoding: Any,
):
"""Synchronous completion request"""
from litellm.llms.custom_httpx.http_handler import HTTPHandler
from litellm.utils import convert_to_model_response_object
# Check if streaming is requested (will be faked)
stream = optional_params.get("stream", False)
# Transform the request using parent class methods
request_data = self.transform_request(
model=model,
messages=messages,
optional_params=optional_params.copy(),
litellm_params=litellm_params,
headers={},
)
# Set up headers
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
# Log the request
logging_obj.pre_call(
input=messages,
api_key=api_key,
additional_args={
"complete_input_dict": request_data,
"api_base": api_base,
},
)
# Make the HTTP request
http_handler = HTTPHandler(concurrent_limit=1)
response = http_handler.post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
if response.status_code != 200:
raise BaseLLMException(
status_code=response.status_code,
message=f"Request failed: {response.text}",
)
response_json = response.json()
# Unwrap predictions to get OpenAI-compatible response
openai_response = self._unwrap_predictions_response(response_json)
# Use litellm's standard response converter
model_response = cast(
ModelResponse,
convert_to_model_response_object(
response_object=openai_response,
model_response_object=model_response,
_response_headers={},
),
)
# Ensure model is set correctly
model_response.model = model
# Log the response
logging_obj.post_call(
input=messages,
api_key=api_key,
original_response=response_json,
additional_args={"complete_input_dict": request_data},
)
# Return fake stream iterator if streaming was requested
return self._handle_fake_stream_response(model_response=model_response, stream=stream)
async def _async_completion(
self,
model: str,
messages: list,
api_base: str,
api_key: str,
model_response: ModelResponse,
print_verbose: Callable,
logging_obj: Any,
optional_params: dict,
litellm_params: dict,
timeout: Optional[Union[float, httpx.Timeout]],
encoding: Any,
):
"""Asynchronous completion request"""
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.types.utils import LlmProviders
from litellm.utils import convert_to_model_response_object
# Check if streaming is requested (will be faked)
stream = optional_params.get("stream", False)
# Transform the request using parent class async methods
request_data = await self.async_transform_request(
model=model,
messages=messages,
optional_params=optional_params.copy(),
litellm_params=litellm_params,
headers={},
)
# Set up headers
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
# Log the request
logging_obj.pre_call(
input=messages,
api_key=api_key,
additional_args={
"complete_input_dict": request_data,
"api_base": api_base,
},
)
# Make the HTTP request
http_handler = get_async_httpx_client(
llm_provider=LlmProviders.VERTEX_AI,
)
response = await http_handler.post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
if response.status_code != 200:
raise BaseLLMException(
status_code=response.status_code,
message=f"Request failed: {response.text}",
)
response_json = response.json()
# Unwrap predictions to get OpenAI-compatible response
openai_response = self._unwrap_predictions_response(response_json)
# Use litellm's standard response converter
model_response = cast(
ModelResponse,
convert_to_model_response_object(
response_object=openai_response,
model_response_object=model_response,
_response_headers={},
),
)
# Ensure model is set correctly
model_response.model = model
# Log the response
logging_obj.post_call(
input=messages,
api_key=api_key,
original_response=response_json,
additional_args={"complete_input_dict": request_data},
)
# Return fake stream iterator if streaming was requested
return self._handle_fake_stream_response(model_response=model_response, stream=stream)

100
litellm/llms/watsonx/chat/transformation.py
View file

@ -4,10 +4,14 @@ Translation from OpenAI's `/chat/completions` endpoint to IBM WatsonX's `/text/c
Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-chat
"""
from typing import List, Optional, Tuple, Union
from typing import Dict, List, Optional, Tuple, Union
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.watsonx import WatsonXAIEndpoint, WatsonXAPIParams
from litellm.types.llms.watsonx import (
WatsonXAIEndpoint,
WatsonXAPIParams,
WatsonXModelPattern,
)
from ....utils import _remove_additional_properties, _remove_strict_from_schema
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
@ -120,3 +124,95 @@ class IBMWatsonXChatConfig(IBMWatsonXMixin, OpenAIGPTConfig):
None if model.startswith("deployment/") else api_params["project_id"]
)
return payload
@staticmethod
def _apply_prompt_template_core(model: str, messages: List[Dict[str, str]], hf_template_fn) -> Optional[str]:
"""Core logic for applying prompt templates"""
from litellm.litellm_core_utils.prompt_templates.factory import (
custom_prompt,
ibm_granite_pt,
mistral_instruct_pt,
)
if WatsonXModelPattern.GRANITE_CHAT.value in model:
return ibm_granite_pt(messages=messages)
elif WatsonXModelPattern.IBM_MISTRAL.value in model:
return mistral_instruct_pt(messages=messages)
elif WatsonXModelPattern.GPT_OSS.value in model:
hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
try:
return hf_template_fn(model=hf_model, messages=messages)
except Exception:
pass
elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
return custom_prompt(
role_dict={
"system": {"pre_message": "<|start_header_id|>system<|end_header_id|>\n", "post_message": "<|eot_id|>"},
"user": {"pre_message": "<|start_header_id|>user<|end_header_id|>\n", "post_message": "<|eot_id|>"},
"assistant": {"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n", "post_message": "<|eot_id|>"},
},
messages=messages,
initial_prompt_value="<|begin_of_text|>",
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
)
return None
@staticmethod
async def aapply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
"""Apply prompt template (async version)"""
import litellm
from litellm.litellm_core_utils.prompt_templates.factory import (
ahf_chat_template,
custom_prompt,
hf_chat_template,
ibm_granite_pt,
mistral_instruct_pt,
)
if WatsonXModelPattern.GRANITE_CHAT.value in model:
return ibm_granite_pt(messages=messages)
elif WatsonXModelPattern.IBM_MISTRAL.value in model:
return mistral_instruct_pt(messages=messages)
elif WatsonXModelPattern.GPT_OSS.value in model:
hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
try:
# Use sync if cached, async if not
if hf_model in litellm.known_tokenizer_config:
return hf_chat_template(model=hf_model, messages=messages)
else:
return await ahf_chat_template(model=hf_model, messages=messages)
except Exception:
pass
elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
return custom_prompt(
role_dict={
"system": {
"pre_message": "<|start_header_id|>system<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"user": {
"pre_message": "<|start_header_id|>user<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"assistant": {
"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
},
messages=messages,
initial_prompt_value="<|begin_of_text|>",
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
)
return None
@staticmethod
def apply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
"""Apply prompt template (sync version)"""
from litellm.litellm_core_utils.prompt_templates.factory import (
hf_chat_template,
)
return IBMWatsonXChatConfig._apply_prompt_template_core(
model=model, messages=messages, hf_template_fn=hf_chat_template
)

90
litellm/llms/watsonx/common_utils.py
View file

@ -131,34 +131,102 @@ def _get_api_params(
)
def convert_watsonx_messages_to_prompt(
async def _aconvert_watsonx_messages_core(
model: str,
messages: List[AllMessageValues],
provider: str,
custom_prompt_dict: Dict,
apply_template_fn,
) -> str:
"""Async core logic for converting watsonx messages to prompt"""
from litellm.types.llms.watsonx import WatsonXModelPattern
# handle anthropic prompts and amazon titan prompts
if model in custom_prompt_dict:
# check if the model has a registered custom prompt
model_prompt_dict = custom_prompt_dict[model]
prompt = ptf.custom_prompt(
return ptf.custom_prompt(
messages=messages,
role_dict=model_prompt_dict.get(
"role_dict", model_prompt_dict.get("roles")
),
role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
eos_token=model_prompt_dict.get("eos_token", ""),
)
return prompt
elif provider == "ibm-mistralai":
prompt = ptf.mistral_instruct_pt(messages=messages)
elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
return ptf.mistral_instruct_pt(messages=messages)
else:
prompt: str = ptf.prompt_factory( # type: ignore
# Try applying specific template first
result = await apply_template_fn(model=model, messages=messages)
if result:
return result
# Fallback to default
return ptf.prompt_factory(
model=model, messages=messages, custom_llm_provider="watsonx"
) # type: ignore
def _convert_watsonx_messages_core(
model: str,
messages: List[AllMessageValues],
provider: str,
custom_prompt_dict: Dict,
apply_template_fn,
) -> str:
"""Sync core logic for converting watsonx messages to prompt"""
from litellm.types.llms.watsonx import WatsonXModelPattern
# handle anthropic prompts and amazon titan prompts
if model in custom_prompt_dict:
model_prompt_dict = custom_prompt_dict[model]
return ptf.custom_prompt(
messages=messages,
role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
eos_token=model_prompt_dict.get("eos_token", ""),
)
return prompt
elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
return ptf.mistral_instruct_pt(messages=messages)
else:
# Try applying specific template first
result = apply_template_fn(model=model, messages=messages)
if result:
return result
# Fallback to default
return ptf.prompt_factory(
model=model, messages=messages, custom_llm_provider="watsonx"
) # type: ignore
async def aconvert_watsonx_messages_to_prompt(
model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
) -> str:
"""Async version of convert_watsonx_messages_to_prompt"""
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
return await _aconvert_watsonx_messages_core(
model=model,
messages=messages,
provider=provider,
custom_prompt_dict=custom_prompt_dict,
apply_template_fn=IBMWatsonXChatConfig.aapply_prompt_template,
)
def convert_watsonx_messages_to_prompt(
model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
) -> str:
"""Sync version of convert_watsonx_messages_to_prompt"""
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
return _convert_watsonx_messages_core(
model=model,
messages=messages,
provider=provider,
custom_prompt_dict=custom_prompt_dict,
apply_template_fn=IBMWatsonXChatConfig.apply_prompt_template,
)
# Mixin class for shared IBM Watson X functionality

44
litellm/llms/watsonx/completion/transformation.py
View file

@ -228,39 +228,35 @@ class IBMWatsonXAIConfig(IBMWatsonXMixin, BaseConfig):
"us-south",
]
def transform_request(
self,
model: str,
messages: List[AllMessageValues],
optional_params: Dict,
litellm_params: Dict,
headers: Dict,
) -> Dict:
provider = model.split("/")[0]
prompt = convert_watsonx_messages_to_prompt(
model=model,
messages=messages,
provider=provider,
custom_prompt_dict={},
)
def _build_request_payload(self, model: str, prompt: str, optional_params: Dict) -> Dict:
"""Shared logic to build request payload"""
extra_body_params = optional_params.pop("extra_body", {})
optional_params.update(extra_body_params)
watsonx_api_params = _get_api_params(params=optional_params)
watsonx_auth_payload = self._prepare_payload(
model=model,
api_params=watsonx_api_params,
)
# init the payload to the text generation call
payload = {
watsonx_auth_payload = self._prepare_payload(model=model, api_params=watsonx_api_params)
return {
"input": prompt,
"moderations": optional_params.pop("moderations", {}),
"parameters": optional_params,
**watsonx_auth_payload,
}
return payload
async def atransform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
"""Async version of transform_request"""
from litellm.llms.watsonx.common_utils import (
aconvert_watsonx_messages_to_prompt,
)
provider = model.split("/")[0]
prompt = await aconvert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
def transform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
"""Sync version of transform_request"""
provider = model.split("/")[0]
prompt = convert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
def transform_response(
self,

51
litellm/main.py
View file

@ -85,6 +85,10 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.llms.base_llm import BaseConfig, BaseImageGenerationConfig
from litellm.llms.bedrock.common_utils import BedrockModelInfo
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.vertex_ai.common_utils import (
VertexAIModelRoute,
get_vertex_ai_model_route,
)
from litellm.realtime_api.main import _realtime_health_check
from litellm.secret_managers.main import get_secret_bool, get_secret_str
from litellm.types.router import GenericLiteLLMParams
@ -150,7 +154,6 @@ from .llms.bedrock.chat import BedrockConverseLLM, BedrockLLM
from .llms.bedrock.embed.embedding import BedrockEmbedding
from .llms.bedrock.image.image_handler import BedrockImageGeneration
from .llms.bytez.chat.transformation import BytezChatConfig
from .llms.lemonade.chat.transformation import LemonadeChatConfig
from .llms.codestral.completion.handler import CodestralTextCompletion
from .llms.cohere.embed import handler as cohere_embed
from .llms.custom_httpx.aiohttp_handler import BaseLLMAIOHTTPHandler
@ -162,6 +165,7 @@ from .llms.gemini.common_utils import get_api_key_from_env
from .llms.groq.chat.handler import GroqChatCompletion
from .llms.heroku.chat.transformation import HerokuChatConfig
from .llms.huggingface.embedding.handler import HuggingFaceEmbedding
from .llms.lemonade.chat.transformation import LemonadeChatConfig
from .llms.nlp_cloud.chat.handler import completion as nlp_cloud_chat_completion
from .llms.oci.chat.transformation import OCIChatConfig
from .llms.ollama.completion import handler as ollama
@ -192,6 +196,7 @@ from .llms.vertex_ai.multimodal_embeddings.embedding_handler import (
from .llms.vertex_ai.text_to_speech.text_to_speech_handler import VertexTextToSpeechAPI
from .llms.vertex_ai.vertex_ai_partner_models.main import VertexAIPartnerModels
from .llms.vertex_ai.vertex_embeddings.embedding_handler import VertexEmbedding
from .llms.vertex_ai.vertex_gemma_models.main import VertexAIGemmaModels
from .llms.vertex_ai.vertex_model_garden.main import VertexAIModelGardenModels
from .llms.vllm.completion import handler as vllm_handler
from .llms.watsonx.chat.handler import WatsonXChatHandler
@ -255,6 +260,7 @@ vertex_multimodal_embedding = VertexMultimodalEmbedding()
vertex_image_generation = VertexImageGeneration()
google_batch_embeddings = GoogleBatchEmbeddings()
vertex_partner_models_chat_completion = VertexAIPartnerModels()
vertex_gemma_chat_completion = VertexAIGemmaModels()
vertex_model_garden_chat_completion = VertexAIModelGardenModels()
vertex_text_to_speech = VertexTextToSpeechAPI()
sagemaker_llm = SagemakerLLM()
@ -2872,10 +2878,10 @@ def completion( # type: ignore # noqa: PLR0915
custom_llm_provider=custom_llm_provider, # type: ignore
client=client,
api_base=api_base,
extra_headers=extra_headers,
extra_headers=headers,
)
elif custom_llm_provider == "vertex_ai":
elif custom_llm_provider == "vertex_ai":
vertex_ai_project = (
optional_params.pop("vertex_project", None)
or optional_params.pop("vertex_ai_project", None)
@ -2897,7 +2903,9 @@ def completion( # type: ignore # noqa: PLR0915
api_base = api_base or litellm.api_base or get_secret("VERTEXAI_API_BASE")
new_params = safe_deep_copy(optional_params or {})
if vertex_partner_models_chat_completion.is_vertex_partner_model(model):
model_route = get_vertex_ai_model_route(model=model, litellm_params=litellm_params)
if model_route == VertexAIModelRoute.PARTNER_MODELS:
model_response = vertex_partner_models_chat_completion.completion(
model=model,
messages=messages,
@ -2918,10 +2926,7 @@ def completion( # type: ignore # noqa: PLR0915
timeout=timeout,
client=client,
)
elif "gemini" in model or (
litellm_params.get("base_model") is not None
and "gemini" in litellm_params["base_model"]
):
elif model_route == VertexAIModelRoute.GEMINI:
model_response = vertex_chat_completion.completion( # type: ignore
model=model,
messages=messages,
@ -2941,9 +2946,31 @@ def completion( # type: ignore # noqa: PLR0915
custom_llm_provider=custom_llm_provider, # type: ignore
client=client,
api_base=api_base,
extra_headers=extra_headers,
extra_headers=headers,
)
elif "openai" in model:
elif model_route == VertexAIModelRoute.GEMMA:
# Vertex Gemma Models with custom prediction endpoint
model_response = vertex_gemma_chat_completion.completion(
model=model,
messages=messages,
model_response=model_response,
print_verbose=print_verbose,
optional_params=new_params,
litellm_params=litellm_params, # type: ignore
logger_fn=logger_fn,
encoding=encoding,
api_base=api_base,
vertex_location=vertex_ai_location,
vertex_project=vertex_ai_project,
vertex_credentials=vertex_credentials,
logging_obj=logging,
acompletion=acompletion,
headers=headers,
custom_prompt_dict=custom_prompt_dict,
timeout=timeout,
client=client,
)
elif model_route == VertexAIModelRoute.MODEL_GARDEN:
# Vertex Model Garden - OpenAI compatible models
model_response = vertex_model_garden_chat_completion.completion(
model=model,
@ -2965,7 +2992,7 @@ def completion( # type: ignore # noqa: PLR0915
timeout=timeout,
client=client,
)
else:
else: # VertexAIModelRoute.NON_GEMINI
model_response = vertex_ai_non_gemini.completion(
model=model,
messages=messages,
@ -3969,6 +3996,7 @@ def embedding( # noqa: PLR0915
"""
azure = kwargs.get("azure", None)
client = kwargs.pop("client", None)
shared_session = kwargs.get("shared_session", None)
max_retries = kwargs.get("max_retries", None)
litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj") # type: ignore
mock_response: Optional[List[float]] = kwargs.get("mock_response", None) # type: ignore
@ -4158,6 +4186,7 @@ def embedding( # noqa: PLR0915
client=client,
aembedding=aembedding,
max_retries=max_retries,
shared_session=shared_session,
)
elif custom_llm_provider == "databricks":
api_base = api_base or litellm.api_base or get_secret("DATABRICKS_API_BASE") # type: ignore

712
litellm/model_prices_and_context_window_backup.json
View file

@ -866,6 +866,36 @@
"mode": "audio_transcription",
"output_cost_per_second": 0.0
},
"au.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"input_cost_per_token_above_200k_tokens": 6.6e-06,
"output_cost_per_token_above_200k_tokens": 2.475e-05,
"cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 200000,
"mode": "chat",
"output_cost_per_token": 1.65e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 346
},
"azure/ada": {
"input_cost_per_token": 1e-07,
"litellm_provider": "azure",
@ -3164,6 +3194,42 @@
"supports_function_calling": true,
"supports_vision": true
},
"azure_ai/Phi-4-mini-reasoning": {
"input_cost_per_token": 8e-08,
"litellm_provider": "azure_ai",
"max_input_tokens": 131072,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "chat",
"output_cost_per_token": 3.2e-07,
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
"supports_function_calling": true
},
"azure_ai/Phi-4-reasoning": {
"input_cost_per_token": 1.25e-07,
"litellm_provider": "azure_ai",
"max_input_tokens": 32768,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "chat",
"output_cost_per_token": 5e-07,
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_reasoning": true
},
"azure_ai/MAI-DS-R1": {
"input_cost_per_token": 1.35e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 128000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 5.4e-06,
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
"supports_reasoning": true,
"supports_tool_choice": true
},
"azure_ai/cohere-rerank-v3-english": {
"input_cost_per_query": 0.002,
"input_cost_per_token": 0.0,
@ -5294,6 +5360,16 @@
"output_cost_per_token": 0.0,
"supports_embedding_image_input": true
},
"cohere.embed-v4:0": {
"input_cost_per_token": 1.2e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 128000,
"max_tokens": 128000,
"mode": "embedding",
"output_cost_per_token": 0.0,
"output_vector_size": 1536,
"supports_embedding_image_input": true
},
"cohere.rerank-v3-5:0": {
"input_cost_per_query": 0.002,
"input_cost_per_token": 0.0,
@ -7838,19 +7914,19 @@
"tool_use_system_prompt_tokens": 159
},
"eu.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"input_cost_per_token_above_200k_tokens": 6.6e-06,
"output_cost_per_token_above_200k_tokens": 2.475e-05,
"cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 200000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token": 1.65e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
@ -11820,6 +11896,66 @@
"video"
]
},
"global.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 200000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 346
},
"global.anthropic.claude-sonnet-4-20250514-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"gpt-3.5-turbo": {
"input_cost_per_token": 0.5e-06,
"litellm_provider": "openai",
@ -12872,6 +13008,72 @@
"supports_tool_choice": true,
"supports_vision": true
},
"gpt-5-pro": {
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 272000,
"max_tokens": 272000,
"mode": "responses",
"output_cost_per_token": 1.2e-04,
"output_cost_per_token_batches": 6e-05,
"supported_endpoints": [
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": false,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"gpt-5-pro-2025-10-06": {
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 272000,
"max_tokens": 272000,
"mode": "responses",
"output_cost_per_token": 1.2e-04,
"output_cost_per_token_batches": 6e-05,
"supported_endpoints": [
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": false,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"gpt-5-codex": {
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
@ -13189,6 +13391,19 @@
"/v1/images/generations"
]
},
"gpt-image-1-mini": {
"cache_read_input_image_token_cost": 2.5e-07,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_image_token": 2.5e-06,
"input_cost_per_token": 2e-06,
"litellm_provider": "openai",
"mode": "chat",
"output_cost_per_image_token": 8e-06,
"supported_endpoints": [
"/v1/images/generations",
"/v1/images/edits"
]
},
"gpt-realtime": {
"cache_creation_input_audio_token_cost": 4e-07,
"cache_read_input_token_cost": 4e-07,
@ -13221,6 +13436,37 @@
"supports_system_messages": true,
"supports_tool_choice": true
},
"gpt-realtime-mini": {
"cache_creation_input_audio_token_cost": 3e-07,
"cache_read_input_audio_token_cost": 3e-07,
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "openai",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/realtime"
],
"supported_modalities": [
"text",
"image",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"gpt-realtime-2025-08-28": {
"cache_creation_input_audio_token_cost": 4e-07,
"cache_read_input_token_cost": 4e-07,
@ -14219,19 +14465,19 @@
"output_cost_per_token": 1.8e-08
},
"jp.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"input_cost_per_token_above_200k_tokens": 6.6e-06,
"output_cost_per_token_above_200k_tokens": 2.475e-05,
"cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 200000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token": 1.65e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
@ -14623,6 +14869,54 @@
"/v1/images/generations"
]
},
"low/1024-x-1024/gpt-image-1-mini": {
"input_cost_per_image": 0.005,
"litellm_provider": "openai",
"mode": "image_generation",
"supported_endpoints": [
"/v1/images/generations"
]
},
"low/1024-x-1536/gpt-image-1-mini": {
"input_cost_per_image": 0.006,
"litellm_provider": "openai",
"mode": "image_generation",
"supported_endpoints": [
"/v1/images/generations"
]
},
"low/1536-x-1024/gpt-image-1-mini": {
"input_cost_per_image": 0.006,
"litellm_provider": "openai",
"mode": "image_generation",
"supported_endpoints": [
"/v1/images/generations"
]
},
"medium/1024-x-1024/gpt-image-1-mini": {
"input_cost_per_image": 0.011,
"litellm_provider": "openai",
"mode": "image_generation",
"supported_endpoints": [
"/v1/images/generations"
]
},
"medium/1024-x-1536/gpt-image-1-mini": {
"input_cost_per_image": 0.015,
"litellm_provider": "openai",
"mode": "image_generation",
"supported_endpoints": [
"/v1/images/generations"
]
},
"medium/1536-x-1024/gpt-image-1-mini": {
"input_cost_per_image": 0.015,
"litellm_provider": "openai",
"mode": "image_generation",
"supported_endpoints": [
"/v1/images/generations"
]
},
"medlm-large": {
"input_cost_per_character": 5e-06,
"litellm_provider": "vertex_ai-language-models",
@ -16369,6 +16663,42 @@
"supports_function_calling": true,
"supports_response_schema": false
},
"oci/cohere.command-latest": {
"input_cost_per_token": 1.56e-06,
"litellm_provider": "oci",
"max_input_tokens": 128000,
"max_output_tokens": 4000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.56e-06,
"source": "https://www.oracle.com/cloud/ai/generative-ai/pricing/",
"supports_function_calling": true,
"supports_response_schema": false
},
"oci/cohere.command-a-03-2025": {
"input_cost_per_token": 1.56e-06,
"litellm_provider": "oci",
"max_input_tokens": 256000,
"max_output_tokens": 4000,
"max_tokens": 256000,
"mode": "chat",
"output_cost_per_token": 1.56e-06,
"source": "https://www.oracle.com/cloud/ai/generative-ai/pricing/",
"supports_function_calling": true,
"supports_response_schema": false
},
"oci/cohere.command-plus-latest": {
"input_cost_per_token": 1.56e-06,
"litellm_provider": "oci",
"max_input_tokens": 128000,
"max_output_tokens": 4000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.56e-06,
"source": "https://www.oracle.com/cloud/ai/generative-ai/pricing/",
"supports_function_calling": true,
"supports_response_schema": false
},
"ollama/codegeex4": {
"input_cost_per_token": 0.0,
"litellm_provider": "ollama",
@ -19640,6 +19970,39 @@
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"together_ai/moonshotai/Kimi-K2-Instruct-0905": {
"input_cost_per_token": 1e-06,
"litellm_provider": "together_ai",
"max_input_tokens": 262144,
"mode": "chat",
"output_cost_per_token": 3e-06,
"source": "https://www.together.ai/models/kimi-k2-0905",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"together_ai/Qwen/Qwen3-Next-80B-A3B-Instruct": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "together_ai",
"max_input_tokens": 262144,
"mode": "chat",
"output_cost_per_token": 1.5e-06,
"source": "https://www.together.ai/models/qwen3-next-80b-a3b-instruct",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"together_ai/Qwen/Qwen3-Next-80B-A3B-Thinking": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "together_ai",
"max_input_tokens": 262144,
"mode": "chat",
"output_cost_per_token": 1.5e-06,
"source": "https://www.together.ai/models/qwen3-next-80b-a3b-thinking",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"tts-1": {
"input_cost_per_character": 1.5e-05,
"litellm_provider": "openai",
@ -19848,19 +20211,19 @@
"tool_use_system_prompt_tokens": 159
},
"us.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"cache_creation_input_token_cost": 4.125e-06,
"cache_read_input_token_cost": 33e-07,
"input_cost_per_token": 33e-06,
"input_cost_per_token_above_200k_tokens": 66e-06,
"output_cost_per_token_above_200k_tokens": 2.475e-05,
"cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
"cache_read_input_token_cost_above_200k_tokens": 66e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 200000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token": 1.65e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
@ -22173,6 +22536,307 @@
"supports_tool_choice": true,
"supports_vision": false
},
"watsonx/bigscience/mt0-xxl-13b": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/core42/jais-13b-chat": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/google/flan-t5-xl-3b": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.0001,
"output_cost_per_token": 0.00025,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-13b-chat-v2": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-13b-instruct-v2": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-3-3-8b-instruct": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.00025,
"output_cost_per_token": 0.001,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/ibm/granite-4-h-small": {
"max_tokens": 20480,
"max_input_tokens": 20480,
"max_output_tokens": 20480,
"input_cost_per_token": 0.000625,
"output_cost_per_token": 0.0025,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/ibm/granite-guardian-3-2-2b": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.00015,
"output_cost_per_token": 0.0006,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-guardian-3-3-8b": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.00025,
"output_cost_per_token": 0.001,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-ttm-1024-96-r2": {
"max_tokens": 512,
"max_input_tokens": 512,
"max_output_tokens": 512,
"input_cost_per_token": 0.000625,
"output_cost_per_token": 0.000625,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-ttm-1536-96-r2": {
"max_tokens": 512,
"max_input_tokens": 512,
"max_output_tokens": 512,
"input_cost_per_token": 0.000625,
"output_cost_per_token": 0.000625,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-ttm-512-96-r2": {
"max_tokens": 512,
"max_input_tokens": 512,
"max_output_tokens": 512,
"input_cost_per_token": 0.000625,
"output_cost_per_token": 0.000625,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/ibm/granite-vision-3-2-2b": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.00015,
"output_cost_per_token": 0.0006,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": true
},
"watsonx/meta-llama/llama-3-2-11b-vision-instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.00025,
"output_cost_per_token": 0.001,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": true
},
"watsonx/meta-llama/llama-3-2-1b-instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.0001,
"output_cost_per_token": 0.0002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/meta-llama/llama-3-2-3b-instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.00015,
"output_cost_per_token": 0.0006,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/meta-llama/llama-3-2-90b-vision-instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.002,
"output_cost_per_token": 0.008,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": true
},
"watsonx/meta-llama/llama-3-3-70b-instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.002,
"output_cost_per_token": 0.006,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/meta-llama/llama-4-maverick-17b": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/meta-llama/llama-guard-3-11b-vision": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.00025,
"output_cost_per_token": 0.001,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": true
},
"watsonx/mistralai/mistral-medium-2505": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.00225,
"output_cost_per_token": 0.00675,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/mistralai/mistral-small-2503": {
"max_tokens": 32000,
"max_input_tokens": 32000,
"max_output_tokens": 32000,
"input_cost_per_token": 0.0002,
"output_cost_per_token": 0.0006,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_vision": false
},
"watsonx/mistralai/pixtral-12b-2409": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 0.00015,
"output_cost_per_token": 0.00015,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": true
},
"watsonx/openai/gpt-oss-120b": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.004,
"output_cost_per_token": 0.016,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"watsonx/sdaia/allam-1-13b-instruct": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.002,
"litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
"supports_vision": false
},
"whisper-1": {
"input_cost_per_second": 0.0001,
"litellm_provider": "openai",

10
litellm/passthrough/main.py
View file

@ -242,12 +242,14 @@ def llm_passthrough_route(
request_query_params=request_query_params,
litellm_params=litellm_params_dict,
)
# need to encode the id of application-inference-profile for bedrock
# [TODO: Refactor to bedrockpassthroughconfig] need to encode the id of application-inference-profile for bedrock
if custom_llm_provider == "bedrock" and "application-inference-profile" in endpoint:
encoded_url_str = CommonUtils.encode_bedrock_runtime_modelid_arn(str(updated_url))
encoded_url_str = CommonUtils.encode_bedrock_runtime_modelid_arn(
str(updated_url)
)
updated_url = httpx.URL(encoded_url_str)
# Add or update query parameters
provider_api_key = provider_config.get_api_key(api_key)

208
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -109,10 +109,21 @@ class MCPRequestHandler:
request.body = mock_body # type: ignore
if ".well-known" in str(request.url): # public routes
validated_user_api_key_auth = UserAPIKeyAuth()
# elif litellm_api_key == "":
# from fastapi import HTTPException
# raise HTTPException(
# status_code=401,
# detail="LiteLLM API key is missing. Please add it or use OAuth authentication.",
# headers={
# "WWW-Authenticate": f'Bearer resource_metadata=f"{request.base_url}/.well-known/oauth-protected-resource"',
# },
# )
else:
validated_user_api_key_auth = await user_api_key_auth(
api_key=litellm_api_key, request=request
)
return (
validated_user_api_key_auth,
mcp_auth_header,
@ -333,6 +344,172 @@ class MCPRequestHandler:
verbose_logger.warning(f"Failed to get allowed MCP servers: {str(e)}")
return []
@staticmethod
async def _get_key_object_permission(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
):
"""Helper to get key object_permission from cache or DB."""
from litellm.proxy.auth.auth_checks import get_object_permission
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if not user_api_key_auth:
return None
# Already loaded
if user_api_key_auth.object_permission:
return user_api_key_auth.object_permission
# Need to fetch from DB
if user_api_key_auth.object_permission_id and prisma_client:
return await get_object_permission(
object_permission_id=user_api_key_auth.object_permission_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
return None
@staticmethod
async def _get_team_object_permission(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
):
"""Helper to get team object_permission from cache or DB."""
from litellm.proxy.auth.auth_checks import (
get_object_permission,
get_team_object,
)
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if not user_api_key_auth or not user_api_key_auth.team_id or not prisma_client:
return None
# First get the team object (which may have object_permission already loaded)
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
team_id=user_api_key_auth.team_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
if not team_obj:
return None
# Already loaded
if team_obj.object_permission:
return team_obj.object_permission
# Need to fetch from DB using object_permission_id
if team_obj.object_permission_id:
return await get_object_permission(
object_permission_id=team_obj.object_permission_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
return None
@staticmethod
async def get_allowed_tools_for_server(
server_id: str,
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> Optional[List[str]]:
"""
Get list of allowed tool names for a specific server based on key/team permissions.
Follows same inheritance logic as get_allowed_mcp_servers.
Args:
server_id: Server ID to check permissions for
user_api_key_auth: User auth
Returns:
List[str] if restrictions exist, None if no restrictions (allow all)
"""
if not user_api_key_auth:
return None
try:
# Get key and team object permissions
key_obj_perm = await MCPRequestHandler._get_key_object_permission(
user_api_key_auth
)
team_obj_perm = await MCPRequestHandler._get_team_object_permission(
user_api_key_auth
)
# Extract tool permissions for this server
key_tools = (
key_obj_perm.mcp_tool_permissions.get(server_id)
if key_obj_perm and key_obj_perm.mcp_tool_permissions
else None
)
team_tools = (
team_obj_perm.mcp_tool_permissions.get(server_id)
if team_obj_perm and team_obj_perm.mcp_tool_permissions
else None
)
# Apply same inheritance logic as get_allowed_mcp_servers
if team_tools:
if key_tools:
# Both have restrictions → intersection
return list(set(team_tools) & set(key_tools))
else:
# Only team has restrictions → inherit from team
return team_tools
else:
# No team restrictions → use key restrictions
return key_tools
except Exception as e:
verbose_logger.warning(f"Failed to get allowed tools for server: {str(e)}")
return None
@staticmethod
async def is_tool_allowed_for_server(
tool_name: str,
server_id: str,
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> bool:
"""
Check if a specific tool is allowed for a server based on key/team permissions.
Args:
tool_name: Name of the tool to check
server_id: Server ID
user_api_key_auth: User auth
Returns:
True if allowed, False if blocked
"""
allowed_tools = await MCPRequestHandler.get_allowed_tools_for_server(
server_id=server_id,
user_api_key_auth=user_api_key_auth,
)
# None means no restrictions (allow all)
if allowed_tools is None:
return True
# Empty list means no tools allowed
if not allowed_tools:
return False
# Check if tool is in allowed list
return tool_name in allowed_tools
@staticmethod
def is_tool_allowed(
allowed_mcp_servers: List[str],
@ -403,41 +580,24 @@ class MCPRequestHandler:
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> List[str]:
"""
The `object_permission` for a team is not stored on the user_api_key_auth object
Get allowed MCP servers for a team.
first we check if the team has a object_permission_id attached
- if it does then we look up the object_permission for the team
Uses the helper _get_team_object_permission which:
1. First checks if object_permission is already loaded on the team
2. If not, fetches from DB using object_permission_id if it exists
"""
from litellm.proxy.auth.auth_checks import get_team_object
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if user_api_key_auth is None:
return []
if user_api_key_auth.team_id is None:
return []
if prisma_client is None:
verbose_logger.debug("prisma_client is None")
return []
try:
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
team_id=user_api_key_auth.team_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
# Use the helper method that properly handles fetching from DB if needed
object_permissions = await MCPRequestHandler._get_team_object_permission(
user_api_key_auth
)
if team_obj is None:
verbose_logger.debug("team_obj is None")
return []
object_permissions = team_obj.object_permission
if object_permissions is None:
return []

532
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -195,6 +195,7 @@ class MCPServerManager:
name=name_for_prefix,
alias=alias,
server_name=server_name,
spec_path=server_config.get("spec_path", None),
url=server_config.get("url", None) or "",
command=server_config.get("command", None) or "",
args=server_config.get("args", None) or [],
@ -215,15 +216,167 @@ class MCPServerManager:
extra_headers=server_config.get("extra_headers", None),
allowed_tools=server_config.get("allowed_tools", None),
disallowed_tools=server_config.get("disallowed_tools", None),
allowed_params=server_config.get("allowed_params", None),
access_groups=server_config.get("access_groups", None),
)
self.config_mcp_servers[server_id] = new_server
# Check if this is an OpenAPI-based server
spec_path = server_config.get("spec_path", None)
if spec_path:
verbose_logger.info(
f"Loading OpenAPI spec from {spec_path} for server {server_name}"
)
self._register_openapi_tools(
spec_path=spec_path,
server=new_server,
base_url=server_config.get("url", ""),
)
verbose_logger.debug(
f"Loaded MCP Servers: {json.dumps(self.config_mcp_servers, indent=4, default=str)}"
)
self.initialize_tool_name_to_mcp_server_name_mapping()
def _register_openapi_tools(self, spec_path: str, server: MCPServer, base_url: str):
"""
Register tools from an OpenAPI specification for a given server.
This creates "virtual" MCP tools from OpenAPI endpoints that are:
1. Registered in the global tool registry with server prefix
2. Mapped to the server for routing
3. Executed via the local tool handler
Args:
spec_path: Path to the OpenAPI specification file
server: The MCPServer instance to register tools for
base_url: Base URL for API calls
"""
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
build_input_schema,
create_tool_function,
)
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
get_base_url as get_openapi_base_url,
)
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
load_openapi_spec,
)
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
try:
# Load OpenAPI spec
spec = load_openapi_spec(spec_path)
# Use base_url from config if provided, otherwise extract from spec
if not base_url:
base_url = get_openapi_base_url(spec)
verbose_logger.info(
f"Registering OpenAPI tools for server {server.name} with base URL: {base_url}"
)
# Get server prefix for tool naming
server_prefix = get_server_prefix(server)
# Build headers from server configuration
headers = {}
# Add authentication headers if configured
if server.authentication_token:
from litellm.types.mcp import MCPAuth
if server.auth_type == MCPAuth.bearer_token:
headers["Authorization"] = f"Bearer {server.authentication_token}"
elif server.auth_type == MCPAuth.api_key:
headers["Authorization"] = f"ApiKey {server.authentication_token}"
elif server.auth_type == MCPAuth.basic:
headers["Authorization"] = f"Basic {server.authentication_token}"
# Add any extra headers from server config
# Note: extra_headers is a List[str] of header names to forward, not a dict
# For OpenAPI tools, we'll just use the authentication headers
# If extra_headers were needed, they would be processed separately
verbose_logger.debug(
f"Using headers for OpenAPI tools (excluding sensitive values): "
f"{list(headers.keys())}"
)
# Extract and register tools from OpenAPI paths
paths = spec.get("paths", {})
registered_count = 0
verbose_logger.debug(f"Processing {len(paths)} paths from OpenAPI spec")
for path, path_item in paths.items():
for method in ["get", "post", "put", "delete", "patch"]:
if method not in path_item:
continue
operation = path_item[method]
# Generate tool name (without prefix initially)
operation_id = operation.get(
"operationId", f"{method}_{path.replace('/', '_')}"
)
base_tool_name = operation_id.replace(" ", "_").lower()
# Add server prefix to tool name
prefixed_tool_name = add_server_prefix_to_tool_name(
base_tool_name, server_prefix
)
# Get description
description = operation.get(
"summary",
operation.get("description", f"{method.upper()} {path}"),
)
# Build input schema using imported function
input_schema = build_input_schema(operation)
# Create tool function with headers using imported function
tool_func = create_tool_function(
path, method, operation, base_url, headers=headers
)
tool_func.__name__ = prefixed_tool_name
tool_func.__doc__ = description
# Register tool with prefixed name in global registry
global_mcp_tool_registry.register_tool(
name=prefixed_tool_name,
description=description,
input_schema=input_schema,
handler=tool_func,
)
# Update tool name to server name mapping (for both prefixed and base names)
self.tool_name_to_mcp_server_name_mapping[
base_tool_name
] = server_prefix
self.tool_name_to_mcp_server_name_mapping[
prefixed_tool_name
] = server_prefix
registered_count += 1
verbose_logger.debug(
f"Registered OpenAPI tool: {prefixed_tool_name} for server {server.name}"
)
verbose_logger.info(
f"Successfully registered {registered_count} OpenAPI tools for server {server.name}"
)
except Exception as e:
verbose_logger.error(
f"Failed to register OpenAPI tools for server {server.name}: {str(e)}"
)
raise e
def remove_server(self, mcp_server: LiteLLM_MCPServerTable):
"""
Remove a server from the registry
@ -469,6 +622,10 @@ class MCPServerManager:
Returns:
List[MCPTool]: List of tools available on the server with prefixed names
"""
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
verbose_logger.debug(f"Connecting to url: {server.url}")
verbose_logger.info(f"_get_tools_from_server for {server.name}...")
@ -481,7 +638,14 @@ class MCPServerManager:
extra_headers=extra_headers,
)
tools = await self._fetch_tools_with_timeout(client, server.name)
## HANDLE OPENAPI TOOLS
if server.spec_path:
_tools = global_mcp_tool_registry.list_tools(tool_prefix=server.name)
tools = global_mcp_tool_registry.convert_tools_to_mcp_sdk_tool_type(
_tools
)
else:
tools = await self._fetch_tools_with_timeout(client, server.name)
prefixed_or_original_tools = self._create_prefixed_tools(
tools, server, add_prefix=add_prefix
@ -597,11 +761,168 @@ class MCPServerManager:
Check if the tool is allowed or banned for the given server
"""
if server.allowed_tools:
return tool_name in server.allowed_tools
return (
tool_name in server.allowed_tools
or f"{server.name}-{tool_name}" in server.allowed_tools
)
if server.disallowed_tools:
return tool_name not in server.disallowed_tools
return (
tool_name not in server.disallowed_tools
and f"{server.name}-{tool_name}" not in server.disallowed_tools
)
return True
def validate_allowed_params(
self, tool_name: str, arguments: Dict[str, Any], server: MCPServer
) -> None:
"""
Filter arguments to only include allowed parameters for the given tool.
Args:
tool_name: Name of the tool (with or without prefix)
arguments: Dictionary of arguments to filter
server: MCPServer configuration
Returns:
Filtered dictionary containing only allowed parameters
Raises:
HTTPException: If allowed_params is configured for this tool but arguments contain disallowed params
"""
from litellm.proxy._experimental.mcp_server.utils import (
get_server_name_prefix_tool_mcp,
)
# If no allowed_params configured, return all arguments
if not server.allowed_params:
return
# Get the unprefixed tool name to match against config
unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(tool_name)
# Check both prefixed and unprefixed tool names
allowed_params_list = server.allowed_params.get(
tool_name
) or server.allowed_params.get(unprefixed_tool_name)
# If this tool doesn't have allowed_params specified, allow all params
if allowed_params_list is None:
return None
# Filter arguments to only include allowed parameters
disallowed_params = [
param for param in arguments.keys() if param not in allowed_params_list
]
if disallowed_params:
raise HTTPException(
status_code=403,
detail={
"error": f"Parameters {disallowed_params} are not allowed for tool {tool_name}. "
f"Allowed parameters: {allowed_params_list}. "
f"Contact proxy admin to allow these parameters."
},
)
async def check_tool_permission_for_key_team(
self,
tool_name: str,
server: MCPServer,
user_api_key_auth: Optional[UserAPIKeyAuth],
) -> None:
"""
Check if a tool is allowed based on key/team object_permission.mcp_tool_permissions.
Uses MCPRequestHandler.is_tool_allowed_for_server for consistent inheritance logic.
Raises HTTPException if tool is not allowed.
Args:
tool_name: Name of the tool to check
server: MCPServer object
user_api_key_auth: User authentication
Raises:
HTTPException: If tool is not allowed for this key/team
"""
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
MCPRequestHandler,
)
if not user_api_key_auth:
return
# Check if tool is allowed
is_allowed = await MCPRequestHandler.is_tool_allowed_for_server(
tool_name=tool_name,
server_id=server.server_id,
user_api_key_auth=user_api_key_auth,
)
if not is_allowed:
raise HTTPException(
status_code=403,
detail={
"error": f"Tool '{tool_name}' is not allowed for your key/team on server '{server.name}'. Contact proxy admin for access."
},
)
async def _call_openapi_tool_handler(
self,
server: MCPServer,
tool_name: str,
arguments: Dict[str, Any],
) -> CallToolResult:
"""
Call an OpenAPI tool handler directly.
For OpenAPI servers, instead of using MCP protocol, we call the tool handler
that was registered during OpenAPI spec parsing. This handler makes direct
HTTP requests to the API.
Args:
tool_name: The full tool name (with prefix) to call
arguments: Tool arguments to pass to the handler
Returns:
CallToolResult with the response from the API
"""
from mcp.types import TextContent
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
# Get the tool from the registry
tool = global_mcp_tool_registry.get_tool(f"{server.name}-{tool_name}")
if tool is None:
# Tool not found in registry
error_msg = f"OpenAPI tool {tool_name} not found in registry"
verbose_logger.error(error_msg)
return CallToolResult(
content=[TextContent(type="text", text=error_msg)],
isError=True,
)
try:
# Call the tool handler with the arguments
# The handler is an async function that makes the HTTP request
handler_result = await tool.handler(**arguments)
# Convert the handler result (string response) to CallToolResult format
result = CallToolResult(
content=[TextContent(type="text", text=str(handler_result))],
isError=False,
)
return result
except Exception as e:
error_msg = f"Error calling OpenAPI tool {tool_name}: {str(e)}"
verbose_logger.error(error_msg)
return CallToolResult(
content=[TextContent(type="text", text=error_msg)],
isError=True,
)
async def pre_call_tool_check(
self,
name: str,
@ -611,7 +932,6 @@ class MCPServerManager:
proxy_logging_obj: ProxyLogging,
server: MCPServer,
):
## check if the tool is allowed or banned for the given server
if not self.check_allowed_or_banned_tools(name, server):
raise HTTPException(
@ -621,6 +941,20 @@ class MCPServerManager:
},
)
## check tool-level permissions from object_permission
await self.check_tool_permission_for_key_team(
tool_name=name,
server=server,
user_api_key_auth=user_api_key_auth,
)
## filter parameters based on allowed_params configuration
self.validate_allowed_params(
tool_name=name,
arguments=arguments,
server=server,
)
pre_hook_kwargs = {
"name": name,
"arguments": arguments,
@ -684,6 +1018,46 @@ class MCPServerManager:
verbose_logger.error(f"Guardrail blocked MCP tool call pre call: {str(e)}")
raise e
def _create_during_hook_task(
self,
name: str,
arguments: Dict[str, Any],
server_name_from_prefix: Optional[str],
user_api_key_auth: Optional[UserAPIKeyAuth],
proxy_logging_obj: ProxyLogging,
start_time: datetime.datetime,
):
"""Create and return a during hook task for MCP tool calls."""
from litellm.types.llms.base import HiddenParams
from litellm.types.mcp import MCPDuringCallRequestObject
request_obj = MCPDuringCallRequestObject(
tool_name=name,
arguments=arguments,
server_name=server_name_from_prefix,
start_time=start_time.timestamp() if start_time else None,
hidden_params=HiddenParams(),
)
during_hook_kwargs = {
"name": name,
"arguments": arguments,
"server_name": server_name_from_prefix,
"user_api_key_auth": user_api_key_auth,
}
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
request_obj, during_hook_kwargs
)
return asyncio.create_task(
proxy_logging_obj.during_call_hook(
user_api_key_dict=user_api_key_auth,
data=synthetic_llm_data,
call_type="mcp_call", # type: ignore
)
)
async def call_tool(
self,
name: str,
@ -747,95 +1121,92 @@ class MCPServerManager:
server=mcp_server,
)
# Get server-specific auth header if available
server_auth_header: Optional[Union[Dict[str, str], str]] = None
if mcp_server_auth_headers and mcp_server.alias:
server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
elif mcp_server_auth_headers and mcp_server.server_name:
server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
# Prepare tasks for during hooks
tasks = []
if proxy_logging_obj:
during_hook_task = self._create_during_hook_task(
name=name,
arguments=arguments,
server_name_from_prefix=server_name_from_prefix,
user_api_key_auth=user_api_key_auth,
proxy_logging_obj=proxy_logging_obj,
start_time=start_time,
)
tasks.append(during_hook_task)
# Fall back to deprecated mcp_auth_header if no server-specific header found
if server_auth_header is None:
server_auth_header = mcp_auth_header
# For OpenAPI servers, call the tool handler directly instead of via MCP client
if mcp_server.spec_path:
verbose_logger.debug(
f"Calling OpenAPI tool {name} directly via HTTP handler"
)
tasks.append(
asyncio.create_task(
self._call_openapi_tool_handler(mcp_server, name, arguments)
)
)
else:
# For regular MCP servers, use the MCP client
# Get server-specific auth header if available
server_auth_header: Optional[Union[Dict[str, str], str]] = None
if mcp_server_auth_headers and mcp_server.alias:
server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
elif mcp_server_auth_headers and mcp_server.server_name:
server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
# oauth2 headers
extra_headers: Optional[Dict[str, str]] = None
if mcp_server.auth_type == MCPAuth.oauth2:
extra_headers = oauth2_headers
# Fall back to deprecated mcp_auth_header if no server-specific header found
if server_auth_header is None:
server_auth_header = mcp_auth_header
if mcp_server.extra_headers and raw_headers:
if extra_headers is None:
extra_headers = {}
for header in mcp_server.extra_headers:
if header in raw_headers:
extra_headers[header] = raw_headers[header]
# oauth2 headers
extra_headers: Optional[Dict[str, str]] = None
if mcp_server.auth_type == MCPAuth.oauth2:
extra_headers = oauth2_headers
client = self._create_mcp_client(
server=mcp_server,
mcp_auth_header=server_auth_header,
extra_headers=extra_headers,
)
if mcp_server.extra_headers and raw_headers:
if extra_headers is None:
extra_headers = {}
for header in mcp_server.extra_headers:
if header in raw_headers:
extra_headers[header] = raw_headers[header]
client = self._create_mcp_client(
server=mcp_server,
mcp_auth_header=server_auth_header,
extra_headers=extra_headers,
)
async with client:
# Use the original tool name (without prefix) for the actual call
call_tool_params = MCPCallToolRequestParams(
name=original_tool_name,
arguments=arguments,
)
tasks = []
if proxy_logging_obj:
# Create synthetic LLM data for during hook processing
from litellm.types.llms.base import HiddenParams
from litellm.types.mcp import MCPDuringCallRequestObject
request_obj = MCPDuringCallRequestObject(
tool_name=name,
arguments=arguments,
server_name=server_name_from_prefix,
start_time=start_time.timestamp() if start_time else None,
hidden_params=HiddenParams(),
)
async def _call_tool_via_client(client, params):
async with client:
return await client.call_tool(params)
during_hook_kwargs = {
"name": name,
"arguments": arguments,
"server_name": server_name_from_prefix,
"user_api_key_auth": user_api_key_auth,
}
tasks.append(
asyncio.create_task(_call_tool_via_client(client, call_tool_params))
)
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
request_obj, during_hook_kwargs
)
try:
mcp_responses = await asyncio.gather(*tasks)
during_hook_task = asyncio.create_task(
proxy_logging_obj.during_call_hook(
user_api_key_dict=user_api_key_auth,
data=synthetic_llm_data,
call_type="mcp_call", # type: ignore
)
)
tasks.append(during_hook_task)
# If proxy_logging_obj is None, the tool call result is at index 0
# If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
result_index = 1 if proxy_logging_obj else 0
result = mcp_responses[result_index]
tasks.append(asyncio.create_task(client.call_tool(call_tool_params)))
try:
mcp_responses = await asyncio.gather(*tasks)
# If proxy_logging_obj is None, the tool call result is at index 0
# If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
result_index = 1 if proxy_logging_obj else 0
result = mcp_responses[result_index]
return cast(CallToolResult, result)
except (
BlockedPiiEntityError,
GuardrailRaisedException,
HTTPException,
) as e:
# Re-raise guardrail exceptions to properly fail the MCP call
verbose_logger.error(
f"Guardrail blocked MCP tool call during result check: {str(e)}"
)
raise e
return cast(CallToolResult, result)
except (
BlockedPiiEntityError,
GuardrailRaisedException,
HTTPException,
) as e:
# Re-raise guardrail exceptions to properly fail the MCP call
verbose_logger.error(
f"Guardrail blocked MCP tool call during result check: {str(e)}"
)
raise e
#########################################################
# End of Methods that call the upstream MCP servers
@ -1009,6 +1380,7 @@ class MCPServerManager:
if not server:
return {
"server_id": server_id,
"server_name": None,
"status": "unknown",
"error": "Server not found",
"last_health_check": datetime.now().isoformat(),
@ -1023,6 +1395,7 @@ class MCPServerManager:
return {
"server_id": server_id,
"server_name": server.name,
"status": "healthy",
"tools_count": len(tools),
"last_health_check": datetime.now().isoformat(),
@ -1035,6 +1408,7 @@ class MCPServerManager:
return {
"server_id": server_id,
"server_name": server.name,
"status": "unhealthy",
"last_health_check": datetime.now().isoformat(),
"response_time_ms": round(response_time, 2),

236
litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py Normal file
View file

@ -0,0 +1,236 @@
"""
This module is used to generate MCP tools from OpenAPI specs.
"""
import json
from typing import Any, Dict, Optional
import httpx
from litellm._logging import verbose_logger
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
# Store the base URL and headers globally
BASE_URL = ""
HEADERS: Dict[str, str] = {}
def load_openapi_spec(filepath: str) -> Dict[str, Any]:
"""Load OpenAPI specification from JSON file."""
with open(filepath, "r") as f:
return json.load(f)
def get_base_url(spec: Dict[str, Any]) -> str:
"""Extract base URL from OpenAPI spec."""
# OpenAPI 3.x
if "servers" in spec and spec["servers"]:
return spec["servers"][0]["url"]
# OpenAPI 2.x (Swagger)
elif "host" in spec:
scheme = spec.get("schemes", ["https"])[0]
base_path = spec.get("basePath", "")
return f"{scheme}://{spec['host']}{base_path}"
return ""
def extract_parameters(operation: Dict[str, Any]) -> tuple:
"""Extract parameter names from OpenAPI operation."""
path_params = []
query_params = []
body_params = []
# OpenAPI 3.x and 2.x parameters
if "parameters" in operation:
for param in operation["parameters"]:
param_name = param["name"]
if param.get("in") == "path":
path_params.append(param_name)
elif param.get("in") == "query":
query_params.append(param_name)
elif param.get("in") == "body":
body_params.append(param_name)
# OpenAPI 3.x requestBody
if "requestBody" in operation:
body_params.append("body")
return path_params, query_params, body_params
def build_input_schema(operation: Dict[str, Any]) -> Dict[str, Any]:
"""Build MCP input schema from OpenAPI operation."""
properties = {}
required = []
# Process parameters
if "parameters" in operation:
for param in operation["parameters"]:
param_name = param["name"]
param_schema = param.get("schema", {})
param_type = param_schema.get("type", "string")
properties[param_name] = {
"type": param_type,
"description": param.get("description", ""),
}
if param.get("required", False):
required.append(param_name)
# Process requestBody (OpenAPI 3.x)
if "requestBody" in operation:
request_body = operation["requestBody"]
content = request_body.get("content", {})
# Try to get JSON schema
if "application/json" in content:
schema = content["application/json"].get("schema", {})
properties["body"] = {
"type": "object",
"description": request_body.get("description", "Request body"),
"properties": schema.get("properties", {}),
}
if request_body.get("required", False):
required.append("body")
return {
"type": "object",
"properties": properties,
"required": required if required else [],
}
def create_tool_function(
path: str,
method: str,
operation: Dict[str, Any],
base_url: str,
headers: Optional[Dict[str, str]] = None,
):
"""Create a tool function for an OpenAPI operation.
Args:
path: API endpoint path
method: HTTP method (get, post, put, delete, patch)
operation: OpenAPI operation object
base_url: Base URL for the API
headers: Optional headers to include in requests (e.g., authentication)
"""
if headers is None:
headers = {}
path_params, query_params, body_params = extract_parameters(operation)
all_params = path_params + query_params + body_params
# Build function signature dynamically
if all_params:
params_str = ", ".join(f"{p}: str = ''" for p in all_params)
else:
params_str = ""
# Create the function code as a string
func_code = f'''
async def tool_function({params_str}) -> str:
"""Dynamically generated tool function."""
url = base_url + path
# Replace path parameters
path_param_names = {path_params}
for param_name in path_param_names:
param_value = locals().get(param_name, "")
if param_value:
url = url.replace("{{" + param_name + "}}", str(param_value))
# Build query params
query_param_names = {query_params}
params = {{}}
for param_name in query_param_names:
param_value = locals().get(param_name, "")
if param_value:
params[param_name] = param_value
# Build request body
body_param_names = {body_params}
json_body = None
if body_param_names:
body_value = locals().get("body", {{}})
if isinstance(body_value, dict):
json_body = body_value
elif body_value:
# If it's a string, try to parse as JSON
import json as json_module
try:
json_body = json_module.loads(body_value) if isinstance(body_value, str) else {{"data": body_value}}
except:
json_body = {{"data": body_value}}
# Make HTTP request
async with httpx.AsyncClient() as client:
if "{method.lower()}" == "get":
response = await client.get(url, params=params, headers=headers)
elif "{method.lower()}" == "post":
response = await client.post(url, params=params, json=json_body, headers=headers)
elif "{method.lower()}" == "put":
response = await client.put(url, params=params, json=json_body, headers=headers)
elif "{method.lower()}" == "delete":
response = await client.delete(url, params=params, headers=headers)
elif "{method.lower()}" == "patch":
response = await client.patch(url, params=params, json=json_body, headers=headers)
else:
return "Unsupported HTTP method: {method}"
return response.text
'''
# Execute the function code to create the actual function
local_vars = {
"httpx": httpx,
"headers": headers,
"base_url": base_url,
"path": path,
"method": method,
}
exec(func_code, local_vars)
return local_vars["tool_function"]
def register_tools_from_openapi(spec: Dict[str, Any], base_url: str):
"""Register MCP tools from OpenAPI specification."""
paths = spec.get("paths", {})
for path, path_item in paths.items():
for method in ["get", "post", "put", "delete", "patch"]:
if method in path_item:
operation = path_item[method]
# Generate tool name
operation_id = operation.get(
"operationId", f"{method}_{path.replace('/', '_')}"
)
tool_name = operation_id.replace(" ", "_").lower()
# Get description
description = operation.get(
"summary", operation.get("description", f"{method.upper()} {path}")
)
# Build input schema
input_schema = build_input_schema(operation)
# Create tool function
tool_func = create_tool_function(path, method, operation, base_url)
tool_func.__name__ = tool_name
tool_func.__doc__ = description
# Register tool with local registry
global_mcp_tool_registry.register_tool(
name=tool_name,
description=description,
input_schema=input_schema,
handler=tool_func,
)
verbose_logger.debug(f"Registered tool: {tool_name}")

162
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -364,25 +364,25 @@ if MCP_AVAILABLE:
def _tool_name_matches(tool_name: str, filter_list: List[str]) -> bool:
"""
Check if a tool name matches any name in the filter list.
Checks both the full tool name and unprefixed version (without server prefix).
This allows users to configure simple tool names regardless of prefixing.
Args:
tool_name: The tool name to check (may be prefixed like "server-tool_name")
filter_list: List of tool names to match against
Returns:
True if the tool name (prefixed or unprefixed) is in the filter list
"""
from litellm.proxy._experimental.mcp_server.utils import (
get_server_name_prefix_tool_mcp,
)
# Check if the full name is in the list
if tool_name in filter_list:
return True
# Check if the unprefixed name is in the list
unprefixed_name, _ = get_server_name_prefix_tool_mcp(tool_name)
return unprefixed_name in filter_list
@ -393,34 +393,36 @@ if MCP_AVAILABLE:
) -> List[MCPTool]:
"""
Filter tools by allowed/disallowed tools configuration.
If allowed_tools is set, only tools in that list are returned.
If disallowed_tools is set, tools in that list are excluded.
Tool names are matched with and without server prefixes for flexibility.
Args:
tools: List of tools to filter
mcp_server: Server configuration with allowed_tools/disallowed_tools
Returns:
Filtered list of tools
"""
tools_to_return = tools
# Filter by allowed_tools (whitelist)
if mcp_server.allowed_tools:
tools_to_return = [
tool for tool in tools
tool
for tool in tools
if _tool_name_matches(tool.name, mcp_server.allowed_tools)
]
# Filter by disallowed_tools (blacklist)
if mcp_server.disallowed_tools:
tools_to_return = [
tool for tool in tools_to_return
tool
for tool in tools_to_return
if not _tool_name_matches(tool.name, mcp_server.disallowed_tools)
]
return tools_to_return
async def _get_tools_from_mcp_servers(
@ -497,10 +499,17 @@ if MCP_AVAILABLE:
extra_headers=extra_headers,
add_prefix=add_prefix,
)
filtered_tools = filter_tools_by_allowed_tools(tools, server)
filtered_tools = await filter_tools_by_key_team_permissions(
tools=filtered_tools,
server_id=server_id,
user_api_key_auth=user_api_key_auth,
)
all_tools.extend(filtered_tools)
verbose_logger.debug(
f"Successfully fetched {len(tools)} tools from server {server.name}, {len(filtered_tools)} after filtering"
)
@ -513,8 +522,41 @@ if MCP_AVAILABLE:
verbose_logger.info(
f"Successfully fetched {len(all_tools)} tools total from all MCP servers"
)
return all_tools
async def filter_tools_by_key_team_permissions(
tools: List[MCPTool],
server_id: str,
user_api_key_auth: Optional[UserAPIKeyAuth],
) -> List[MCPTool]:
"""
Filter tools based on key/team mcp_tool_permissions.
Note: Tool names in the DB are stored without server prefixes,
but tool names from MCP servers are prefixed. We need to strip
the prefix before comparing.
"""
# Filter by key/team tool-level permissions
allowed_tool_names = await MCPRequestHandler.get_allowed_tools_for_server(
server_id=server_id,
user_api_key_auth=user_api_key_auth,
)
if allowed_tool_names is not None:
# Strip prefix from tool names before comparing
# Tools are stored in DB without prefix, but come from MCP server with prefix
filtered_tools = []
for t in tools:
# Get tool name without server prefix
unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(t.name)
if unprefixed_tool_name in allowed_tool_names:
filtered_tools.append(t)
else:
# No restrictions, return all tools
filtered_tools = tools
return filtered_tools
async def _list_mcp_tools(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
mcp_auth_header: Optional[str] = None,
@ -557,30 +599,7 @@ if MCP_AVAILABLE:
)
# Continue with empty managed tools list instead of failing completely
# Get tools from local registry
local_tools = []
try:
local_tools_raw = global_mcp_tool_registry.list_tools()
# Convert local tools to MCPTool format
for tool in local_tools_raw:
# Convert from litellm.types.mcp_server.tool_registry.MCPTool to mcp.types.Tool
mcp_tool = MCPTool(
name=tool.name,
description=tool.description,
inputSchema=tool.input_schema,
)
local_tools.append(mcp_tool)
except Exception as e:
verbose_logger.exception(
f"Error getting tools from local registry: {str(e)}"
)
# Continue with empty local tools list instead of failing completely
# Combine all tools
all_tools = managed_tools + local_tools
return all_tools
return managed_tools
@client
async def call_mcp_tool(
@ -641,33 +660,42 @@ if MCP_AVAILABLE:
standard_logging_mcp_tool_call
)
litellm_logging_obj.model = f"MCP: {name}"
# Try managed server tool first (pass the full prefixed name)
# Primary and recommended way to use MCP servers
# Check if tool exists in local registry first (for OpenAPI-based tools)
# These tools are registered with their prefixed names
#########################################################
mcp_server: Optional[MCPServer] = (
global_mcp_server_manager._get_mcp_server_from_tool_name(name)
)
if mcp_server:
standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
mcp_server.mcp_info or {}
).get("mcp_server_cost_info")
response = await _handle_managed_mcp_tool(
name=name, # Pass the full name (potentially prefixed)
arguments=arguments,
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
local_tool = global_mcp_tool_registry.get_tool(name)
if local_tool:
verbose_logger.debug(f"Executing local registry tool: {name}")
response = await _handle_local_mcp_tool(name, arguments)
# Fall back to local tool registry (use original name)
#########################################################
# Deprecated: Local MCP Server Tool
# Try managed MCP server tool (pass the full prefixed name)
# Primary and recommended way to use external MCP servers
#########################################################
else:
response = await _handle_local_mcp_tool(original_tool_name, arguments)
mcp_server: Optional[MCPServer] = (
global_mcp_server_manager._get_mcp_server_from_tool_name(name)
)
if mcp_server:
standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
mcp_server.mcp_info or {}
).get("mcp_server_cost_info")
response = await _handle_managed_mcp_tool(
name=name, # Pass the full name (potentially prefixed)
arguments=arguments,
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
# Fall back to local tool registry with original name (legacy support)
#########################################################
# Deprecated: Local MCP Server Tool
#########################################################
else:
response = await _handle_local_mcp_tool(original_tool_name, arguments)
#########################################################
# Post MCP Tool Call Hook
@ -739,14 +767,21 @@ if MCP_AVAILABLE:
Handle tool execution for local registry tools
Note: Local tools don't use prefixes, so we use the original name
"""
import inspect
tool = global_mcp_tool_registry.get_tool(name)
if not tool:
raise HTTPException(status_code=404, detail=f"Tool '{name}' not found")
try:
result = tool.handler(**arguments)
# Check if handler is async or sync
if inspect.iscoroutinefunction(tool.handler):
result = await tool.handler(**arguments)
else:
result = tool.handler(**arguments)
return [TextContent(text=str(result), type="text")]
except Exception as e:
verbose_logger.exception(f"Error executing local tool {name}: {str(e)}")
return [TextContent(text=f"Error: {str(e)}", type="text")]
def _get_mcp_servers_in_path(path: str) -> Optional[List[str]]:
@ -867,6 +902,7 @@ if MCP_AVAILABLE:
await session_manager.handle_request(scope, receive, send)
except Exception as e:
raise e
verbose_logger.exception(f"Error handling MCP request: {e}")
# Instead of re-raising, try to send a graceful error response
try:

22
litellm/proxy/_experimental/mcp_server/tool_registry.py
View file

@ -1,6 +1,8 @@
import json
from typing import Any, Callable, Dict, List, Optional
from mcp.types import Tool as MCPToolSDKTool
from litellm._logging import verbose_logger
from litellm.proxy.types_utils.utils import get_instance_fn
from litellm.types.mcp_server.tool_registry import MCPTool
@ -39,12 +41,30 @@ class MCPToolRegistry:
"""
return self.tools.get(name)
def list_tools(self) -> List[MCPTool]:
def list_tools(self, tool_prefix: Optional[str] = None) -> List[MCPTool]:
"""
List all registered tools
"""
if tool_prefix:
return [
tool
for tool in self.tools.values()
if tool.name.startswith(tool_prefix)
]
return list(self.tools.values())
def convert_tools_to_mcp_sdk_tool_type(
self, tools: List[MCPTool]
) -> List[MCPToolSDKTool]:
return [
MCPToolSDKTool(
name=tool.name,
description=tool.description,
inputSchema=tool.input_schema,
)
for tool in tools
]
def load_tools_from_config(
self, mcp_tools_config: Optional[Dict[str, Any]] = None
) -> None:

1
litellm/proxy/_experimental/out/_next/static/WkpkdsewrdPMuTzVGS_5j/_buildManifest.js
View file

@ -1 +0,0 @@
self.__BUILD_MANIFEST={__rewrites:{afterFiles:[],beforeFiles:[],fallback:[]},"/_error":["static/chunks/pages/_error-28b803cb2479b966.js"],sortedPages:["/_app","/_error"]},self.__BUILD_MANIFEST_CB&&self.__BUILD_MANIFEST_CB();

1
litellm/proxy/_experimental/out/_next/static/chunks/1039-1031e149b66e294d.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1052-80a465c977e6b146.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1160-3efb81c958413447.js Normal file
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/117-6a9f6a591033d4a0.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1223-de5e7e4f043a5233.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1264-2979d95e0b56a75c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1307-6b882871684ce94c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/13b76428-e1bf383848c17260.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/13b76428-ebdf3012af0e4489.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1487-ada9ecf7dd9bca97.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1491-80fe911f18f4631c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1526-8e2976ed10cb2fa0.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1529-e0933e3af843b646.js Normal file
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more