scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 85

Merge latest changes from origin/main into fix/redact-aws-credentials

Browse source 
Resolved import conflict for CustomPricingLiteLLMParams which was moved
from litellm.types.router to litellm.types.utils in main branch.
This commit is contained in:
Tim Elfrink 2025-10-09 08:12:22 +02:00
commit 02f0b1e83a
292 changed files with 5615 additions and 560 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

310
docs/my-website/docs/mcp.md
View file

@ -246,8 +246,203 @@ litellm_settings:
</TabItem>
</Tabs>
## MCP Tool Filtering
## Converting OpenAPI Specs to MCP Servers
LiteLLM can automatically convert OpenAPI specifications into MCP servers, allowing you to expose any REST API as MCP tools. This is useful when you have existing APIs with OpenAPI/Swagger documentation and want to make them available as MCP tools.
### Benefits
- **Rapid Integration**: Convert existing APIs to MCP tools without writing custom MCP server code
- **Automatic Tool Generation**: LiteLLM automatically generates MCP tools from your OpenAPI spec
- **Unified Interface**: Use the same MCP interface for both native MCP servers and OpenAPI-based APIs
- **Easy Testing**: Test and iterate on API integrations quickly
### Configuration
Add your OpenAPI-based MCP server to your `config.yaml`:
```yaml title="config.yaml - OpenAPI to MCP" showLineNumbers
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-xxxxxxx
mcp_servers:
# OpenAPI Spec Example - Petstore API
petstore_mcp:
url: "https://petstore.swagger.io/v2"
spec_path: "/path/to/openapi.json"
auth_type: "none"
# OpenAPI Spec with API Key Authentication
my_api_mcp:
url: "http://0.0.0.0:8090"
spec_path: "/path/to/openapi.json"
auth_type: "api_key"
auth_value: "your-api-key-here"
# OpenAPI Spec with Bearer Token
secured_api_mcp:
url: "https://api.example.com"
spec_path: "/path/to/openapi.json"
auth_type: "bearer_token"
auth_value: "your-bearer-token"
```
### Configuration Parameters
| Parameter | Required | Description |
|-----------|----------|-------------|
| `url` | Yes | The base URL of your API endpoint |
| `spec_path` | Yes | Path or URL to your OpenAPI specification file (JSON or YAML) |
| `auth_type` | No | Authentication type: `none`, `api_key`, `bearer_token`, `basic`, `authorization` |
| `auth_value` | No | Authentication value (required if `auth_type` is set) |
| `description` | No | Optional description for the MCP server |
| `allowed_tools` | No | List of specific tools to allow (see [MCP Tool Filtering](#mcp-tool-filtering)) |
| `disallowed_tools` | No | List of specific tools to block (see [MCP Tool Filtering](#mcp-tool-filtering)) |
### Usage Example
Once configured, you can use the OpenAPI-based MCP server just like any other MCP server:
<Tabs>
<TabItem value="fastmcp" label="Python FastMCP">
```python title="Using OpenAPI-based MCP Server" showLineNumbers
from fastmcp import Client
import asyncio
# Standard MCP configuration
config = {
"mcpServers": {
"petstore": {
"url": "http://localhost:4000/petstore_mcp/mcp",
"headers": {
"x-litellm-api-key": "Bearer sk-1234"
}
}
}
}
# Create a client that connects to the server
client = Client(config)
async def main():
async with client:
# List available tools generated from OpenAPI spec
tools = await client.list_tools()
print(f"Available tools: {[tool.name for tool in tools]}")
# Example: Get a pet by ID (from Petstore API)
response = await client.call_tool(
name="getpetbyid",
arguments={"petId": "1"}
)
print(f"Response:\n{response}\n")
# Example: Find pets by status
response = await client.call_tool(
name="findpetsbystatus",
arguments={"status": "available"}
)
print(f"Response:\n{response}\n")
if __name__ == "__main__":
asyncio.run(main())
```
</TabItem>
<TabItem value="cursor" label="Cursor IDE">
```json title="Cursor MCP Configuration for OpenAPI Server" showLineNumbers
{
"mcpServers": {
"Petstore": {
"url": "http://localhost:4000/petstore_mcp/mcp",
"headers": {
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
}
}
}
}
```
</TabItem>
<TabItem value="openai" label="OpenAI Responses API">
```bash title="Using OpenAPI MCP Server with OpenAI" showLineNumbers
curl --location 'https://api.openai.com/v1/responses' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $OPENAI_API_KEY" \
--data '{
"model": "gpt-4o",
"tools": [
{
"type": "mcp",
"server_label": "petstore",
"server_url": "http://localhost:4000/petstore_mcp/mcp",
"require_approval": "never",
"headers": {
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY"
}
}
],
"input": "Find all available pets in the petstore",
"tool_choice": "required"
}'
```
</TabItem>
</Tabs>
### How It Works
1. **Spec Loading**: LiteLLM loads your OpenAPI specification from the provided `spec_path`
2. **Tool Generation**: Each API endpoint in the spec becomes an MCP tool
3. **Parameter Mapping**: OpenAPI parameters are automatically mapped to MCP tool parameters
4. **Request Handling**: When a tool is called, LiteLLM converts the MCP request to the appropriate HTTP request
5. **Response Translation**: API responses are converted back to MCP format
### OpenAPI Spec Requirements
Your OpenAPI specification should follow standard OpenAPI/Swagger conventions:
- **Supported versions**: OpenAPI 3.0.x, OpenAPI 3.1.x, Swagger 2.0
- **Required fields**: `paths`, `info` sections should be properly defined
- **Operation IDs**: Each operation should have a unique `operationId` (this becomes the tool name)
- **Parameters**: Request parameters should be properly documented with types and descriptions
### Example OpenAPI Spec Structure
```yaml title="sample-openapi.yaml" showLineNumbers
openapi: 3.0.0
info:
title: My API
version: 1.0.0
paths:
/pets/{petId}:
get:
operationId: getPetById
summary: Get a pet by ID
parameters:
- name: petId
in: path
required: true
schema:
type: integer
responses:
'200':
description: Successful response
content:
application/json:
schema:
type: object
```
## Allow/Disallow MCP Tools
Control which tools are available from your MCP servers. You can either allow only specific tools or block dangerous ones.
<Tabs>
@ -306,6 +501,119 @@ mcp_servers:
- If you specify both `allowed_tools` and `disallowed_tools`, the allowed list takes priority
- Tool names are case-sensitive
---
## Allow/Disallow MCP Tool Parameters
Control which parameters are allowed for specific MCP tools using the `allowed_params` configuration. This provides fine-grained control over tool usage by restricting the parameters that can be passed to each tool.
### Configuration
`allowed_params` is a dictionary that maps tool names to lists of allowed parameter names. When configured, only the specified parameters will be accepted for that tool - any other parameters will be rejected with a 403 error.
```yaml title="config.yaml with allowed_params" showLineNumbers
mcp_servers:
deepwiki_mcp:
url: https://mcp.deepwiki.com/mcp
transport: "http"
auth_type: "none"
allowed_params:
# Tool name: list of allowed parameters
read_wiki_contents: ["status"]
my_api_mcp:
url: "https://my-api-server.com"
auth_type: "api_key"
auth_value: "my-key"
allowed_params:
# Using unprefixed tool name
getpetbyid: ["status"]
# Using prefixed tool name (both formats work)
my_api_mcp-findpetsbystatus: ["status", "limit"]
# Another tool with multiple allowed params
create_issue: ["title", "body", "labels"]
```
### How It Works
1. **Tool-specific filtering**: Each tool can have its own list of allowed parameters
2. **Flexible naming**: Tool names can be specified with or without the server prefix (e.g., both `"getpetbyid"` and `"my_api_mcp-getpetbyid"` work)
3. **Whitelist approach**: Only parameters in the allowed list are permitted
4. **Unlisted tools**: If `allowed_params` is not set, all parameters are allowed
5. **Error handling**: Requests with disallowed parameters receive a 403 error with details about which parameters are allowed
### Example Request Behavior
With the configuration above, here's how requests would be handled:
**✅ Allowed Request:**
```json
{
"tool": "read_wiki_contents",
"arguments": {
"status": "active"
}
}
```
**❌ Rejected Request:**
```json
{
"tool": "read_wiki_contents",
"arguments": {
"status": "active",
"limit": 10 // This parameter is not allowed
}
}
```
**Error Response:**
```json
{
"error": "Parameters ['limit'] are not allowed for tool read_wiki_contents. Allowed parameters: ['status']. Contact proxy admin to allow these parameters."
}
```
### Use Cases
- **Security**: Prevent users from accessing sensitive parameters or dangerous operations
- **Cost control**: Restrict expensive parameters (e.g., limiting result counts)
- **Compliance**: Enforce parameter usage policies for regulatory requirements
- **Staged rollouts**: Gradually enable parameters as tools are tested
- **Multi-tenant isolation**: Different parameter access for different user groups
### Combining with Tool Filtering
`allowed_params` works alongside `allowed_tools` and `disallowed_tools` for complete control:
```yaml title="Combined filtering example" showLineNumbers
mcp_servers:
github_mcp:
url: "https://api.githubcopilot.com/mcp"
auth_type: oauth2
authorization_url: https://github.com/login/oauth/authorize
token_url: https://github.com/login/oauth/access_token
client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
scopes: ["public_repo", "user:email"]
# Only allow specific tools
allowed_tools: ["create_issue", "list_issues", "search_issues"]
# Block dangerous operations
disallowed_tools: ["delete_repo"]
# Restrict parameters per tool
allowed_params:
create_issue: ["title", "body", "labels"]
list_issues: ["state", "sort", "perPage"]
search_issues: ["query", "sort", "order", "perPage"]
```
This configuration ensures that:
1. Only the three listed tools are available
2. The `delete_repo` tool is explicitly blocked
3. Each tool can only use its specified parameters
---
## MCP Server Access Control
LiteLLM Proxy provides two methods for controlling access to specific MCP servers:

17
docs/my-website/docs/proxy/admin_ui_sso.md
View file

@ -81,6 +81,23 @@ MICROSOFT_TENANT="5a39737
http://localhost:4000/sso/callback
```
**Using App Roles for User Permissions**
You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token and assign the corresponding role to the user.
Supported roles:
- `proxy_admin` - Admin over the platform
- `proxy_admin_viewer` - Can login, view all keys, view all spend (read-only)
- `internal_user` - Normal user. Can login, view spend and depending on team-member permissions - view/create/delete their own keys.
To set up app roles:
1. Navigate to your App Registration on https://portal.azure.com/
2. Go to "App roles" and create a new app role
3. Use one of the supported role names above (e.g., `proxy_admin`)
4. Assign users to these roles in your Enterprise Application
5. When users sign in via SSO, LiteLLM will automatically assign them the corresponding role
</TabItem>
<TabItem value="Generic" label="Generic SSO Provider">

2
docs/my-website/docs/proxy/caching.md
View file

@ -278,6 +278,8 @@ Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable cac
REDIS_HOST = "" # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
REDIS_PORT = "" # REDIS_PORT='18841'
REDIS_PASSWORD = "" # REDIS_PASSWORD='liteLlmIsAmazing'
REDIS_USERNAME = "" # REDIS_USERNAME='my-redis-username' [OPTIONAL] if your redis server requires a username
REDIS_SSL = "True" # REDIS_SSL='True' to enable SSL by default is False
```
**Additional kwargs**

48
docs/my-website/docs/tutorials/msft_sso.md
View file

@ -140,6 +140,54 @@ litellm_settings:
<Image img={require('../../img/msft_default_settings.png')} style={{ width: '900px', height: 'auto' }} />
## 4. Using Entra ID App Roles for User Permissions
You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token during SSO sign-in and assign the corresponding role to the user.
### 4.1 Supported Roles
LiteLLM supports the following app roles (case-insensitive):
- `proxy_admin` - Admin over the entire LiteLLM platform
- `proxy_admin_viewer` - Read-only admin access (can view all keys and spend)
- `org_admin` - Admin over a specific organization (can create teams and users within their org)
- `internal_user` - Standard user (can create/view/delete their own keys and view their own spend)
### 4.2 Create App Roles in Entra ID
1. Navigate to your App Registration on https://portal.azure.com/
2. Go to **App roles** > **Create app role**
3. Configure the app role:
- **Display name**: Proxy Admin (or your preferred display name)
- **Value**: `proxy_admin` (use one of the supported role values above)
- **Description**: Administrator access to LiteLLM proxy
- **Allowed member types**: Users/Groups
4. Click **Apply** to save the role
### 4.3 Assign Users to App Roles
1. Navigate to **Enterprise Applications** on https://portal.azure.com/
2. Select your LiteLLM application
3. Go to **Users and groups** > **Add user/group**
4. Select the user and assign them to one of the app roles you created
### 4.4 Test the Role Assignment
1. Sign in to LiteLLM UI via SSO as a user with an assigned app role
2. LiteLLM will automatically extract the app role from the JWT token
3. The user will be assigned the corresponding LiteLLM role in the database
4. The user's permissions will reflect their assigned role
**How it works:**
- When a user signs in via Microsoft SSO, LiteLLM extracts the `roles` claim from the JWT `id_token`
- If any of the roles match a valid LiteLLM role (case-insensitive), that role is assigned to the user
- If multiple roles are present, LiteLLM uses the first valid role it finds
- This role assignment persists in the LiteLLM database and determines the user's access level
## Video Walkthrough
This walks through setting up sso auto-add for **Microsoft Entra ID**

3
litellm/__init__.py
View file

@ -1190,6 +1190,9 @@ from .llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
from .llms.azure.responses.o_series_transformation import (
AzureOpenAIOSeriesResponsesAPIConfig,
)
from .llms.litellm_proxy.responses.transformation import (
LiteLLMProxyResponsesAPIConfig,
)
from .llms.openai.chat.o_series_transformation import (
OpenAIOSeriesConfig as OpenAIO1Config, # maintain backwards compatibility
OpenAIOSeriesConfig,

19
litellm/_redis.py
View file

@ -177,14 +177,21 @@ def get_redis_url_from_environment():
raise ValueError(
"Either 'REDIS_URL' or both 'REDIS_HOST' and 'REDIS_PORT' must be specified for Redis."
)
if "REDIS_PASSWORD" in os.environ:
redis_password = f":{os.environ['REDIS_PASSWORD']}@"
if "REDIS_SSL" in os.environ and os.environ["REDIS_SSL"].lower() == "true":
redis_protocol = "rediss"
else:
redis_password = ""
redis_protocol = "redis"
# Build authentication part of URL
auth_part = ""
if "REDIS_USERNAME" in os.environ and "REDIS_PASSWORD" in os.environ:
auth_part = f"{os.environ['REDIS_USERNAME']}:{os.environ['REDIS_PASSWORD']}@"
elif "REDIS_PASSWORD" in os.environ:
auth_part = f"{os.environ['REDIS_PASSWORD']}@"
return (
f"redis://{redis_password}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
f"{redis_protocol}://{auth_part}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
)

16
litellm/caching/caching_handler.py
View file

@ -14,10 +14,10 @@ It utilizes the (RedisCache, s3Cache, RedisSemanticCache, QdrantSemanticCache, I
In each method it will call the appropriate method from caching.py
"""
import time
import asyncio
import datetime
import inspect
import time
from typing import (
TYPE_CHECKING,
Any,
@ -62,12 +62,10 @@ else:
LiteLLMLoggingObj = Any
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.litellm_core_utils.core_helpers import (
_get_parent_otel_span_from_kwargs,
_get_parent_otel_span_from_kwargs,
)
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
class CachingHandlerResponse(BaseModel):
@ -214,9 +212,7 @@ class LLMCachingHandler:
end_time=end_time,
cache_hit=cache_hit,
)
cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
**kwargs
)
cache_key = litellm.cache.get_cache_key(**kwargs)
if (
isinstance(cached_result, BaseModel)
or isinstance(cached_result, CustomStreamWrapper)
@ -330,9 +326,7 @@ class LLMCachingHandler:
end_time=end_time,
cache_hit=cache_hit
)
cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
**kwargs
)
cache_key = litellm.cache.get_cache_key(**kwargs)
if (
isinstance(cached_result, BaseModel)
or isinstance(cached_result, CustomStreamWrapper)

8
litellm/files/main.py
View file

@ -268,6 +268,7 @@ def create_file(
raise e
@client
async def afile_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -308,6 +309,7 @@ async def afile_retrieve(
raise e
@client
def file_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -422,6 +424,7 @@ def file_retrieve(
# Delete file
@client
async def afile_delete(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -462,6 +465,7 @@ async def afile_delete(
raise e
@client
def file_delete(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@ -577,6 +581,7 @@ def file_delete(
# List files
@client
async def afile_list(
custom_llm_provider: Literal["openai", "azure"] = "openai",
purpose: Optional[str] = None,
@ -617,6 +622,7 @@ async def afile_list(
raise e
@client
def file_list(
custom_llm_provider: Literal["openai", "azure"] = "openai",
purpose: Optional[str] = None,
@ -729,6 +735,7 @@ def file_list(
raise e
@client
async def afile_content(
file_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
@ -771,6 +778,7 @@ async def afile_content(
raise e
@client
def file_content(
file_id: str,
model: Optional[str] = None,

2
litellm/litellm_core_utils/litellm_logging.py
View file

@ -81,12 +81,12 @@ from litellm.types.llms.openai import (
)
from litellm.types.mcp import MCPPostCallResponseObject
from litellm.types.rerank import RerankResponse
from litellm.types.router import CustomPricingLiteLLMParams
from litellm.types.utils import (
CachingDetails,
CallTypes,
CostBreakdown,
CostResponseTypes,
CustomPricingLiteLLMParams,
DynamicPromptManagementParamLiteral,
EmbeddingResponse,
GuardrailStatus,

290
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -2,7 +2,6 @@ import copy
import json
import mimetypes
import re
from litellm._uuid import uuid
import xml.etree.ElementTree as ET
from enum import Enum
from typing import Any, List, Optional, Tuple, cast, overload
@ -13,6 +12,7 @@ import litellm
import litellm.types
import litellm.types.llms
from litellm import verbose_logger
from litellm._uuid import uuid
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_async_httpx_client
from litellm.types.files import get_file_extension_from_mime_type
from litellm.types.llms.anthropic import *
@ -232,7 +232,6 @@ def ollama_pt(
## MERGE CONSECUTIVE ASSISTANT CONTENT ##
while msg_i < len(messages) and messages[msg_i]["role"] == "assistant":
assistant_content_str += convert_content_list_to_str(messages[msg_i])
msg_i += 1
tool_calls = messages[msg_i].get("tool_calls")
ollama_tool_calls = []
@ -258,7 +257,7 @@ def ollama_pt(
f"Tool Calls: {json.dumps(ollama_tool_calls, indent=2)}"
)
msg_i += 1
msg_i += 1
if assistant_content_str:
prompt += f"### Assistant:\n{assistant_content_str}\n\n"
@ -365,62 +364,20 @@ def phind_codellama_pt(messages):
return prompt
def hf_chat_template( # noqa: PLR0915
model: str, messages: list, chat_template: Optional[Any] = None
):
# Define Jinja2 environment
env = ImmutableSandboxedEnvironment()
def raise_exception(message):
raise Exception(f"Error message - {message}")
# Create a template object from the template text
env.globals["raise_exception"] = raise_exception
## get the tokenizer config from huggingface
bos_token = ""
eos_token = ""
if chat_template is None:
def _get_tokenizer_config(hf_model_name):
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
# Make a GET request to fetch the JSON data
client = HTTPHandler(concurrent_limit=1)
response = client.get(url)
except Exception as e:
raise e
if response.status_code == 200:
# Parse the JSON data
tokenizer_config = json.loads(response.content)
return {"status": "success", "tokenizer": tokenizer_config}
else:
return {"status": "failure"}
if model in litellm.known_tokenizer_config:
tokenizer_config = litellm.known_tokenizer_config[model]
else:
tokenizer_config = _get_tokenizer_config(model)
litellm.known_tokenizer_config.update({model: tokenizer_config})
if (
tokenizer_config["status"] == "failure"
or "chat_template" not in tokenizer_config["tokenizer"]
):
raise Exception("No chat template found")
## read the bos token, eos token and chat template from the json
tokenizer_config = tokenizer_config["tokenizer"] # type: ignore
bos_token = tokenizer_config["bos_token"] # type: ignore
if bos_token is not None and not isinstance(bos_token, str):
if isinstance(bos_token, dict):
bos_token = bos_token.get("content", None)
eos_token = tokenizer_config["eos_token"] # type: ignore
if eos_token is not None and not isinstance(eos_token, str):
if isinstance(eos_token, dict):
eos_token = eos_token.get("content", None)
chat_template = tokenizer_config["chat_template"] # type: ignore
def _render_chat_template(env, chat_template: str, bos_token: str, eos_token: str, messages: list) -> str:
"""
Shared template rendering logic for both sync and async hf_chat_template
Args:
env: Jinja2 environment
chat_template: Chat template string
bos_token: Beginning of sequence token
eos_token: End of sequence token
messages: Messages to render
Returns:
Rendered template string
"""
try:
template = env.from_string(chat_template) # type: ignore
except Exception as e:
@ -435,7 +392,6 @@ def hf_chat_template( # noqa: PLR0915
bos_token="<bos>",
)
return True
# This will be raised if Jinja attempts to render the system message and it can't
except Exception:
return False
@ -469,7 +425,7 @@ def hf_chat_template( # noqa: PLR0915
)
except Exception as e:
if "Conversation roles must alternate user/assistant" in str(e):
# reformat messages to ensure user/assistant are alternating, if there's either 2 consecutive 'user' messages or 2 consecutive 'assistant' message, add a blank 'user' or 'assistant' message to ensure compatibility
# reformat messages to ensure user/assistant are alternating
new_messages = []
for i in range(len(reformatted_messages) - 1):
new_messages.append(reformatted_messages[i])
@ -495,6 +451,188 @@ def hf_chat_template( # noqa: PLR0915
) # don't use verbose_logger.exception, if exception is raised
async def _afetch_and_extract_template(
model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
) -> Tuple[str, str, str]:
"""
Async version: Fetch template and tokens from HuggingFace.
Returns: (chat_template, bos_token, eos_token)
"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_extract_token_value,
)
bos_token = ""
eos_token = ""
if chat_template is None:
# Fetch or retrieve cached tokenizer config
if model in litellm.known_tokenizer_config:
tokenizer_config = litellm.known_tokenizer_config[model]
else:
tokenizer_config = await get_config_fn(hf_model_name=model)
litellm.known_tokenizer_config.update({model: tokenizer_config})
# Try to get chat template from tokenizer_config.json first
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
and "chat_template" in tokenizer_config["tokenizer"]
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
chat_template = tokenizer_data["chat_template"]
else:
# Fallback: Try to fetch chat template from separate .jinja file
template_result = await get_template_fn(hf_model_name=model)
if template_result.get("status") == "success":
chat_template = template_result["chat_template"]
# Still try to get tokens from tokenizer_config if available
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
else:
raise Exception("No chat template found")
return chat_template, bos_token, eos_token # type: ignore
def _fetch_and_extract_template(
model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
) -> Tuple[str, str, str]:
"""
Sync version: Fetch template and tokens from HuggingFace.
Returns: (chat_template, bos_token, eos_token)
"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_extract_token_value,
)
bos_token = ""
eos_token = ""
if chat_template is None:
# Fetch or retrieve cached tokenizer config
if model in litellm.known_tokenizer_config:
tokenizer_config = litellm.known_tokenizer_config[model]
else:
tokenizer_config = get_config_fn(hf_model_name=model)
litellm.known_tokenizer_config.update({model: tokenizer_config})
# Try to get chat template from tokenizer_config.json first
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
and "chat_template" in tokenizer_config["tokenizer"]
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
chat_template = tokenizer_data["chat_template"]
else:
# Fallback: Try to fetch chat template from separate .jinja file
template_result = get_template_fn(hf_model_name=model)
if template_result.get("status") == "success":
chat_template = template_result["chat_template"]
# Still try to get tokens from tokenizer_config if available
if (
tokenizer_config.get("status") == "success"
and "tokenizer" in tokenizer_config
and isinstance(tokenizer_config["tokenizer"], dict)
):
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
bos_token = _extract_token_value(
token_value=tokenizer_data.get("bos_token")
)
eos_token = _extract_token_value(
token_value=tokenizer_data.get("eos_token")
)
else:
raise Exception("No chat template found")
return chat_template, bos_token, eos_token # type: ignore
async def ahf_chat_template(
model: str, messages: list, chat_template: Optional[Any] = None
):
"""HuggingFace chat template (async version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_aget_chat_template_file,
_aget_tokenizer_config,
strftime_now,
)
env = ImmutableSandboxedEnvironment()
env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
env.globals["strftime_now"] = strftime_now
template, bos_token, eos_token = await _afetch_and_extract_template(
model=model,
chat_template=chat_template,
get_config_fn=_aget_tokenizer_config,
get_template_fn=_aget_chat_template_file,
)
return _render_chat_template(
env=env,
chat_template=template,
bos_token=bos_token,
eos_token=eos_token,
messages=messages,
)
def hf_chat_template(
model: str, messages: list, chat_template: Optional[Any] = None
):
"""HuggingFace chat template (sync version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_get_chat_template_file,
_get_tokenizer_config,
strftime_now,
)
env = ImmutableSandboxedEnvironment()
env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
env.globals["strftime_now"] = strftime_now
template, bos_token, eos_token = _fetch_and_extract_template(
model=model,
chat_template=chat_template,
get_config_fn=_get_tokenizer_config,
get_template_fn=_get_chat_template_file,
)
return _render_chat_template(
env=env,
chat_template=template,
bos_token=bos_token,
eos_token=eos_token,
messages=messages,
)
def deepseek_r1_pt(messages):
return hf_chat_template(
model="deepseek-r1/deepseek-r1-7b-instruct", messages=messages
@ -4032,33 +4170,9 @@ def prompt_factory(
elif custom_llm_provider == "azure_text":
return azure_text_pt(messages=messages)
elif custom_llm_provider == "watsonx":
if "granite" in model and "chat" in model:
# granite-13b-chat-v1 and granite-13b-chat-v2 use a specific prompt template
return ibm_granite_pt(messages=messages)
elif "ibm-mistral" in model and "instruct" in model:
# models like ibm-mistral/mixtral-8x7b-instruct-v01-q use the mistral instruct prompt template
return mistral_instruct_pt(messages=messages)
elif "meta-llama/llama-3" in model and "instruct" in model:
# https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/
return custom_prompt(
role_dict={
"system": {
"pre_message": "<|start_header_id|>system<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"user": {
"pre_message": "<|start_header_id|>user<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"assistant": {
"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
},
messages=messages,
initial_prompt_value="<|begin_of_text|>",
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
)
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
return IBMWatsonXChatConfig.apply_prompt_template(model=model, messages=messages)
try:
if "meta-llama/llama-2" in model and "chat" in model:
return llama_2_chat_pt(messages=messages)

139
litellm/litellm_core_utils/prompt_templates/huggingface_template_handler.py Normal file
View file

@ -0,0 +1,139 @@
import json
from datetime import datetime
from typing import Any, Dict, Union
from litellm.llms.custom_httpx.http_handler import (
_get_httpx_client,
get_async_httpx_client,
)
from litellm.types.llms.custom_http import httpxSpecialProvider
def strftime_now(fmt: str) -> str:
"""
Custom function for templates that need current date/time formatting (e.g., gpt-oss)
Args:
fmt: Format string for datetime.now().strftime()
Returns:
Formatted string
"""
return datetime.now().strftime(fmt)
def _get_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch tokenizer_config.json from HuggingFace (sync)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'tokenizer' keys
"""
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
client = _get_httpx_client()
response = client.get(url=url)
except Exception as e:
raise e
if response.status_code == 200:
tokenizer_config = json.loads(response.content)
return {"status": "success", "tokenizer": tokenizer_config}
else:
return {"status": "failure"}
async def _aget_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch tokenizer_config.json from HuggingFace (async)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'tokenizer' keys
"""
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.PromptFactory,
)
response = await client.get(url=url)
except Exception as e:
raise e
if response.status_code == 200:
tokenizer_config = json.loads(response.content)
return {"status": "success", "tokenizer": tokenizer_config}
else:
return {"status": "failure"}
def _get_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch chat template from separate .jinja file (sync)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'chat_template' keys
"""
template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
client = _get_httpx_client()
for filename in template_filenames:
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
response = client.get(url=url)
if response.status_code == 200:
return {"status": "success", "chat_template": response.content.decode("utf-8")}
except Exception:
continue
return {"status": "failure"}
async def _aget_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
"""
Fetch chat template from separate .jinja file (async)
Args:
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
Returns:
Dict with 'status' and optionally 'chat_template' keys
"""
template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.PromptFactory,
)
for filename in template_filenames:
try:
url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
response = await client.get(url=url)
if response.status_code == 200:
return {"status": "success", "chat_template": response.content.decode("utf-8")}
except Exception:
continue
return {"status": "failure"}
def _extract_token_value(token_value: Union[None, str, Dict[str, Any]]) -> str:
"""
Extract token string from various formats (string, dict, etc.)
Args:
token_value: Token value in various formats (None, str, or dict with 'content' key)
Returns:
Extracted token string
"""
if token_value is None or isinstance(token_value, str):
return token_value or ""
if isinstance(token_value, dict):
return token_value.get("content", "")
return ""

8
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
View file

@ -270,7 +270,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
processed_chunk.get("delta", {}).get("stop_reason")
is not None
):
self.holding_stop_reason_chunk = processed_chunk
else:
self.chunk_queue.append(processed_chunk)
@ -380,4 +379,11 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
self.current_content_block_start = content_block_start
return True
# For parallel tool calls, we'll necessarily have a new content block
# if we get a function name since it signals a new tool call
if block_type == "tool_use" and content_block_start.get("name"):
self.current_content_block_type = block_type
self.current_content_block_start = content_block_start
return True
return False

9
litellm/llms/azure/common_utils.py
View file

@ -365,6 +365,11 @@ def get_azure_ad_token(
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
except ValueError:
verbose_logger.debug("Azure AD Token Provider could not be used.")
except Exception as e:
verbose_logger.error(
f"Error calling Azure AD token provider: {str(e)}. Follow docs - https://docs.litellm.ai/docs/providers/azure/#azure-ad-token-refresh---defaultazurecredential"
)
raise e
#########################################################
# If litellm.enable_azure_ad_token_refresh is True and no other token provider is available,
@ -561,7 +566,9 @@ class BaseAzureLLM(BaseOpenAILLM):
"Using Azure AD token provider based on Service Principal with Secret workflow for Azure Auth"
)
try:
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
azure_ad_token_provider = get_azure_ad_token_provider(
azure_scope=scope,
)
except ValueError:
verbose_logger.debug("Azure AD Token Provider could not be used.")
if api_version is None:

48
litellm/llms/litellm_proxy/responses/transformation.py Normal file
View file

@ -0,0 +1,48 @@
"""
Responses API transformation for LiteLLM Proxy provider.
LiteLLM Proxy supports the OpenAI Responses API natively when the underlying model supports it.
This config enables pass-through behavior to the proxy's /v1/responses endpoint.
"""
from typing import Optional
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import LlmProviders
class LiteLLMProxyResponsesAPIConfig(OpenAIResponsesAPIConfig):
"""
Configuration for LiteLLM Proxy Responses API support.
Extends OpenAI's config since the proxy follows OpenAI's API spec,
but uses LITELLM_PROXY_API_BASE for the base URL.
"""
@property
def custom_llm_provider(self) -> LlmProviders:
return LlmProviders.LITELLM_PROXY
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the endpoint for LiteLLM Proxy responses API.
Uses LITELLM_PROXY_API_BASE environment variable if api_base is not provided.
"""
api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
if api_base is None:
raise ValueError(
"api_base not set for LiteLLM Proxy responses API. "
"Set via api_base parameter or LITELLM_PROXY_API_BASE environment variable"
)
# Remove trailing slashes
api_base = api_base.rstrip("/")
return f"{api_base}/responses"

8
litellm/llms/openai/chat/gpt_transformation.py
View file

@ -397,13 +397,13 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
)
from litellm.types.llms.openai import ChatCompletionToolParam
for message in messages:
message = cast(
for i, message in enumerate(messages):
messages[i] = cast(
AllMessageValues, filter_value_from_dict(message, "cache_control") # type: ignore
)
if tools is not None:
for tool in tools:
tool = cast(
for i, tool in enumerate(tools):
tools[i] = cast(
ChatCompletionToolParam,
filter_value_from_dict(tool, "cache_control"), # type: ignore
)

66
litellm/llms/openrouter/chat/transformation.py
View file

@ -6,6 +6,7 @@ Calls done in OpenAI/openai.py as OpenRouter is openai-compatible.
Docs: https://openrouter.ai/docs/parameters
"""
from enum import Enum
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union
import httpx
@ -20,6 +21,12 @@ from ...openai.chat.gpt_transformation import OpenAIGPTConfig
from ..common_utils import OpenRouterException
class CacheControlSupportedModels(str, Enum):
"""Models that support cache_control in content blocks."""
CLAUDE = "claude"
GEMINI = "gemini"
class OpenrouterConfig(OpenAIGPTConfig):
def map_openai_params(
self,
@ -48,19 +55,73 @@ class OpenrouterConfig(OpenAIGPTConfig):
)
return mapped_openai_params
def _supports_cache_control_in_content(self, model: str) -> bool:
"""
Check if the model supports cache_control in content blocks.
Returns:
bool: True if model supports cache_control (Claude or Gemini models)
"""
model_lower = model.lower()
return any(
supported_model.value in model_lower
for supported_model in CacheControlSupportedModels
)
def remove_cache_control_flag_from_messages_and_tools(
self,
model: str,
messages: List[AllMessageValues],
tools: Optional[List["ChatCompletionToolParam"]] = None,
) -> Tuple[List[AllMessageValues], Optional[List["ChatCompletionToolParam"]]]:
if "claude" in model.lower(): # don't remove 'cache_control' flag
if self._supports_cache_control_in_content(model):
return messages, tools
else:
return super().remove_cache_control_flag_from_messages_and_tools(
model, messages, tools
)
def _move_cache_control_to_content(
self, messages: List[AllMessageValues]
) -> List[AllMessageValues]:
"""
Move cache_control from message level to content blocks.
OpenRouter requires cache_control to be inside content blocks, not at message level.
When cache_control is at message level, it's added to ALL content blocks
to cache the entire message content.
"""
transformed_messages = []
for message in messages:
message_copy = dict(message)
cache_control = message_copy.pop("cache_control", None)
if cache_control is not None:
content = message_copy.get("content")
if isinstance(content, list):
# Content is already a list, add cache_control to all blocks
if len(content) > 0:
content_copy = []
for block in content:
block_copy = dict(block)
block_copy["cache_control"] = cache_control
content_copy.append(block_copy)
message_copy["content"] = content_copy
else:
# Content is a string, convert to structured format
message_copy["content"] = [
{
"type": "text",
"text": content,
"cache_control": cache_control,
}
]
transformed_messages.append(message_copy)
return transformed_messages
def transform_request(
self,
model: str,
@ -75,6 +136,9 @@ class OpenrouterConfig(OpenAIGPTConfig):
Returns:
dict: The transformed request. Sent as the body of the API call.
"""
if self._supports_cache_control_in_content(model):
messages = self._move_cache_control_to_content(messages)
extra_body = optional_params.pop("extra_body", {})
response = super().transform_request(
model, messages, optional_params, litellm_params, headers

100
litellm/llms/watsonx/chat/transformation.py
View file

@ -4,10 +4,14 @@ Translation from OpenAI's `/chat/completions` endpoint to IBM WatsonX's `/text/c
Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-chat
"""
from typing import List, Optional, Tuple, Union
from typing import Dict, List, Optional, Tuple, Union
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.watsonx import WatsonXAIEndpoint, WatsonXAPIParams
from litellm.types.llms.watsonx import (
WatsonXAIEndpoint,
WatsonXAPIParams,
WatsonXModelPattern,
)
from ....utils import _remove_additional_properties, _remove_strict_from_schema
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
@ -120,3 +124,95 @@ class IBMWatsonXChatConfig(IBMWatsonXMixin, OpenAIGPTConfig):
None if model.startswith("deployment/") else api_params["project_id"]
)
return payload
@staticmethod
def _apply_prompt_template_core(model: str, messages: List[Dict[str, str]], hf_template_fn) -> Optional[str]:
"""Core logic for applying prompt templates"""
from litellm.litellm_core_utils.prompt_templates.factory import (
custom_prompt,
ibm_granite_pt,
mistral_instruct_pt,
)
if WatsonXModelPattern.GRANITE_CHAT.value in model:
return ibm_granite_pt(messages=messages)
elif WatsonXModelPattern.IBM_MISTRAL.value in model:
return mistral_instruct_pt(messages=messages)
elif WatsonXModelPattern.GPT_OSS.value in model:
hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
try:
return hf_template_fn(model=hf_model, messages=messages)
except Exception:
pass
elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
return custom_prompt(
role_dict={
"system": {"pre_message": "<|start_header_id|>system<|end_header_id|>\n", "post_message": "<|eot_id|>"},
"user": {"pre_message": "<|start_header_id|>user<|end_header_id|>\n", "post_message": "<|eot_id|>"},
"assistant": {"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n", "post_message": "<|eot_id|>"},
},
messages=messages,
initial_prompt_value="<|begin_of_text|>",
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
)
return None
@staticmethod
async def aapply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
"""Apply prompt template (async version)"""
import litellm
from litellm.litellm_core_utils.prompt_templates.factory import (
ahf_chat_template,
custom_prompt,
hf_chat_template,
ibm_granite_pt,
mistral_instruct_pt,
)
if WatsonXModelPattern.GRANITE_CHAT.value in model:
return ibm_granite_pt(messages=messages)
elif WatsonXModelPattern.IBM_MISTRAL.value in model:
return mistral_instruct_pt(messages=messages)
elif WatsonXModelPattern.GPT_OSS.value in model:
hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
try:
# Use sync if cached, async if not
if hf_model in litellm.known_tokenizer_config:
return hf_chat_template(model=hf_model, messages=messages)
else:
return await ahf_chat_template(model=hf_model, messages=messages)
except Exception:
pass
elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
return custom_prompt(
role_dict={
"system": {
"pre_message": "<|start_header_id|>system<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"user": {
"pre_message": "<|start_header_id|>user<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
"assistant": {
"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
"post_message": "<|eot_id|>",
},
},
messages=messages,
initial_prompt_value="<|begin_of_text|>",
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
)
return None
@staticmethod
def apply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
"""Apply prompt template (sync version)"""
from litellm.litellm_core_utils.prompt_templates.factory import (
hf_chat_template,
)
return IBMWatsonXChatConfig._apply_prompt_template_core(
model=model, messages=messages, hf_template_fn=hf_chat_template
)

90
litellm/llms/watsonx/common_utils.py
View file

@ -131,34 +131,102 @@ def _get_api_params(
)
def convert_watsonx_messages_to_prompt(
async def _aconvert_watsonx_messages_core(
model: str,
messages: List[AllMessageValues],
provider: str,
custom_prompt_dict: Dict,
apply_template_fn,
) -> str:
"""Async core logic for converting watsonx messages to prompt"""
from litellm.types.llms.watsonx import WatsonXModelPattern
# handle anthropic prompts and amazon titan prompts
if model in custom_prompt_dict:
# check if the model has a registered custom prompt
model_prompt_dict = custom_prompt_dict[model]
prompt = ptf.custom_prompt(
return ptf.custom_prompt(
messages=messages,
role_dict=model_prompt_dict.get(
"role_dict", model_prompt_dict.get("roles")
),
role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
eos_token=model_prompt_dict.get("eos_token", ""),
)
return prompt
elif provider == "ibm-mistralai":
prompt = ptf.mistral_instruct_pt(messages=messages)
elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
return ptf.mistral_instruct_pt(messages=messages)
else:
prompt: str = ptf.prompt_factory( # type: ignore
# Try applying specific template first
result = await apply_template_fn(model=model, messages=messages)
if result:
return result
# Fallback to default
return ptf.prompt_factory(
model=model, messages=messages, custom_llm_provider="watsonx"
) # type: ignore
def _convert_watsonx_messages_core(
model: str,
messages: List[AllMessageValues],
provider: str,
custom_prompt_dict: Dict,
apply_template_fn,
) -> str:
"""Sync core logic for converting watsonx messages to prompt"""
from litellm.types.llms.watsonx import WatsonXModelPattern
# handle anthropic prompts and amazon titan prompts
if model in custom_prompt_dict:
model_prompt_dict = custom_prompt_dict[model]
return ptf.custom_prompt(
messages=messages,
role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
eos_token=model_prompt_dict.get("eos_token", ""),
)
return prompt
elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
return ptf.mistral_instruct_pt(messages=messages)
else:
# Try applying specific template first
result = apply_template_fn(model=model, messages=messages)
if result:
return result
# Fallback to default
return ptf.prompt_factory(
model=model, messages=messages, custom_llm_provider="watsonx"
) # type: ignore
async def aconvert_watsonx_messages_to_prompt(
model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
) -> str:
"""Async version of convert_watsonx_messages_to_prompt"""
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
return await _aconvert_watsonx_messages_core(
model=model,
messages=messages,
provider=provider,
custom_prompt_dict=custom_prompt_dict,
apply_template_fn=IBMWatsonXChatConfig.aapply_prompt_template,
)
def convert_watsonx_messages_to_prompt(
model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
) -> str:
"""Sync version of convert_watsonx_messages_to_prompt"""
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
return _convert_watsonx_messages_core(
model=model,
messages=messages,
provider=provider,
custom_prompt_dict=custom_prompt_dict,
apply_template_fn=IBMWatsonXChatConfig.apply_prompt_template,
)
# Mixin class for shared IBM Watson X functionality

44
litellm/llms/watsonx/completion/transformation.py
View file

@ -228,39 +228,35 @@ class IBMWatsonXAIConfig(IBMWatsonXMixin, BaseConfig):
"us-south",
]
def transform_request(
self,
model: str,
messages: List[AllMessageValues],
optional_params: Dict,
litellm_params: Dict,
headers: Dict,
) -> Dict:
provider = model.split("/")[0]
prompt = convert_watsonx_messages_to_prompt(
model=model,
messages=messages,
provider=provider,
custom_prompt_dict={},
)
def _build_request_payload(self, model: str, prompt: str, optional_params: Dict) -> Dict:
"""Shared logic to build request payload"""
extra_body_params = optional_params.pop("extra_body", {})
optional_params.update(extra_body_params)
watsonx_api_params = _get_api_params(params=optional_params)
watsonx_auth_payload = self._prepare_payload(
model=model,
api_params=watsonx_api_params,
)
# init the payload to the text generation call
payload = {
watsonx_auth_payload = self._prepare_payload(model=model, api_params=watsonx_api_params)
return {
"input": prompt,
"moderations": optional_params.pop("moderations", {}),
"parameters": optional_params,
**watsonx_auth_payload,
}
return payload
async def atransform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
"""Async version of transform_request"""
from litellm.llms.watsonx.common_utils import (
aconvert_watsonx_messages_to_prompt,
)
provider = model.split("/")[0]
prompt = await aconvert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
def transform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
"""Sync version of transform_request"""
provider = model.split("/")[0]
prompt = convert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
def transform_response(
self,

33
litellm/model_prices_and_context_window_backup.json
View file

@ -12975,6 +12975,39 @@
"supports_vision": true,
"supports_web_search": true
},
"gpt-5-pro-2025-10-06": {
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_batches": 7.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 272000,
"max_tokens": 272000,
"mode": "responses",
"output_cost_per_token": 1.2e-04,
"output_cost_per_token_batches": 6e-05,
"supported_endpoints": [
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_native_streaming": false,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_web_search": true
},
"gpt-5-codex": {
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,

82
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -109,10 +109,21 @@ class MCPRequestHandler:
request.body = mock_body # type: ignore
if ".well-known" in str(request.url): # public routes
validated_user_api_key_auth = UserAPIKeyAuth()
# elif litellm_api_key == "":
# from fastapi import HTTPException
# raise HTTPException(
# status_code=401,
# detail="LiteLLM API key is missing. Please add it or use OAuth authentication.",
# headers={
# "WWW-Authenticate": f'Bearer resource_metadata=f"{request.base_url}/.well-known/oauth-protected-resource"',
# },
# )
else:
validated_user_api_key_auth = await user_api_key_auth(
api_key=litellm_api_key, request=request
)
return (
validated_user_api_key_auth,
mcp_auth_header,
@ -344,14 +355,14 @@ class MCPRequestHandler:
proxy_logging_obj,
user_api_key_cache,
)
if not user_api_key_auth:
return None
# Already loaded
if user_api_key_auth.object_permission:
return user_api_key_auth.object_permission
# Need to fetch from DB
if user_api_key_auth.object_permission_id and prisma_client:
return await get_object_permission(
@ -361,7 +372,7 @@ class MCPRequestHandler:
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
return None
@staticmethod
@ -369,16 +380,19 @@ class MCPRequestHandler:
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
):
"""Helper to get team object_permission from cache or DB."""
from litellm.proxy.auth.auth_checks import get_object_permission, get_team_object
from litellm.proxy.auth.auth_checks import (
get_object_permission,
get_team_object,
)
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if not user_api_key_auth or not user_api_key_auth.team_id or not prisma_client:
return None
# First get the team object (which may have object_permission already loaded)
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
team_id=user_api_key_auth.team_id,
@ -387,14 +401,14 @@ class MCPRequestHandler:
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
if not team_obj:
return None
# Already loaded
if team_obj.object_permission:
return team_obj.object_permission
# Need to fetch from DB using object_permission_id
if team_obj.object_permission_id:
return await get_object_permission(
@ -404,7 +418,7 @@ class MCPRequestHandler:
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
return None
@staticmethod
@ -415,26 +429,38 @@ class MCPRequestHandler:
"""
Get list of allowed tool names for a specific server based on key/team permissions.
Follows same inheritance logic as get_allowed_mcp_servers.
Args:
server_id: Server ID to check permissions for
user_api_key_auth: User auth
Returns:
List[str] if restrictions exist, None if no restrictions (allow all)
"""
if not user_api_key_auth:
return None
try:
# Get key and team object permissions
key_obj_perm = await MCPRequestHandler._get_key_object_permission(user_api_key_auth)
team_obj_perm = await MCPRequestHandler._get_team_object_permission(user_api_key_auth)
key_obj_perm = await MCPRequestHandler._get_key_object_permission(
user_api_key_auth
)
team_obj_perm = await MCPRequestHandler._get_team_object_permission(
user_api_key_auth
)
# Extract tool permissions for this server
key_tools = key_obj_perm.mcp_tool_permissions.get(server_id) if key_obj_perm and key_obj_perm.mcp_tool_permissions else None
team_tools = team_obj_perm.mcp_tool_permissions.get(server_id) if team_obj_perm and team_obj_perm.mcp_tool_permissions else None
key_tools = (
key_obj_perm.mcp_tool_permissions.get(server_id)
if key_obj_perm and key_obj_perm.mcp_tool_permissions
else None
)
team_tools = (
team_obj_perm.mcp_tool_permissions.get(server_id)
if team_obj_perm and team_obj_perm.mcp_tool_permissions
else None
)
# Apply same inheritance logic as get_allowed_mcp_servers
if team_tools:
if key_tools:
@ -446,7 +472,7 @@ class MCPRequestHandler:
else:
# No team restrictions → use key restrictions
return key_tools
except Exception as e:
verbose_logger.warning(f"Failed to get allowed tools for server: {str(e)}")
return None
@ -459,12 +485,12 @@ class MCPRequestHandler:
) -> bool:
"""
Check if a specific tool is allowed for a server based on key/team permissions.
Args:
tool_name: Name of the tool to check
server_id: Server ID
user_api_key_auth: User auth
Returns:
True if allowed, False if blocked
"""
@ -472,15 +498,15 @@ class MCPRequestHandler:
server_id=server_id,
user_api_key_auth=user_api_key_auth,
)
# None means no restrictions (allow all)
if allowed_tools is None:
return True
# Empty list means no tools allowed
if not allowed_tools:
return False
# Check if tool is in allowed list
return tool_name in allowed_tools
@ -555,7 +581,7 @@ class MCPRequestHandler:
) -> List[str]:
"""
Get allowed MCP servers for a team.
Uses the helper _get_team_object_permission which:
1. First checks if object_permission is already loaded on the team
2. If not, fetches from DB using object_permission_id if it exists
@ -571,7 +597,7 @@ class MCPRequestHandler:
object_permissions = await MCPRequestHandler._get_team_object_permission(
user_api_key_auth
)
if object_permissions is None:
return []

477
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -195,6 +195,7 @@ class MCPServerManager:
name=name_for_prefix,
alias=alias,
server_name=server_name,
spec_path=server_config.get("spec_path", None),
url=server_config.get("url", None) or "",
command=server_config.get("command", None) or "",
args=server_config.get("args", None) or [],
@ -215,15 +216,167 @@ class MCPServerManager:
extra_headers=server_config.get("extra_headers", None),
allowed_tools=server_config.get("allowed_tools", None),
disallowed_tools=server_config.get("disallowed_tools", None),
allowed_params=server_config.get("allowed_params", None),
access_groups=server_config.get("access_groups", None),
)
self.config_mcp_servers[server_id] = new_server
# Check if this is an OpenAPI-based server
spec_path = server_config.get("spec_path", None)
if spec_path:
verbose_logger.info(
f"Loading OpenAPI spec from {spec_path} for server {server_name}"
)
self._register_openapi_tools(
spec_path=spec_path,
server=new_server,
base_url=server_config.get("url", ""),
)
verbose_logger.debug(
f"Loaded MCP Servers: {json.dumps(self.config_mcp_servers, indent=4, default=str)}"
)
self.initialize_tool_name_to_mcp_server_name_mapping()
def _register_openapi_tools(self, spec_path: str, server: MCPServer, base_url: str):
"""
Register tools from an OpenAPI specification for a given server.
This creates "virtual" MCP tools from OpenAPI endpoints that are:
1. Registered in the global tool registry with server prefix
2. Mapped to the server for routing
3. Executed via the local tool handler
Args:
spec_path: Path to the OpenAPI specification file
server: The MCPServer instance to register tools for
base_url: Base URL for API calls
"""
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
build_input_schema,
create_tool_function,
)
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
get_base_url as get_openapi_base_url,
)
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
load_openapi_spec,
)
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
try:
# Load OpenAPI spec
spec = load_openapi_spec(spec_path)
# Use base_url from config if provided, otherwise extract from spec
if not base_url:
base_url = get_openapi_base_url(spec)
verbose_logger.info(
f"Registering OpenAPI tools for server {server.name} with base URL: {base_url}"
)
# Get server prefix for tool naming
server_prefix = get_server_prefix(server)
# Build headers from server configuration
headers = {}
# Add authentication headers if configured
if server.authentication_token:
from litellm.types.mcp import MCPAuth
if server.auth_type == MCPAuth.bearer_token:
headers["Authorization"] = f"Bearer {server.authentication_token}"
elif server.auth_type == MCPAuth.api_key:
headers["Authorization"] = f"ApiKey {server.authentication_token}"
elif server.auth_type == MCPAuth.basic:
headers["Authorization"] = f"Basic {server.authentication_token}"
# Add any extra headers from server config
# Note: extra_headers is a List[str] of header names to forward, not a dict
# For OpenAPI tools, we'll just use the authentication headers
# If extra_headers were needed, they would be processed separately
verbose_logger.debug(
f"Using headers for OpenAPI tools (excluding sensitive values): "
f"{list(headers.keys())}"
)
# Extract and register tools from OpenAPI paths
paths = spec.get("paths", {})
registered_count = 0
verbose_logger.debug(f"Processing {len(paths)} paths from OpenAPI spec")
for path, path_item in paths.items():
for method in ["get", "post", "put", "delete", "patch"]:
if method not in path_item:
continue
operation = path_item[method]
# Generate tool name (without prefix initially)
operation_id = operation.get(
"operationId", f"{method}_{path.replace('/', '_')}"
)
base_tool_name = operation_id.replace(" ", "_").lower()
# Add server prefix to tool name
prefixed_tool_name = add_server_prefix_to_tool_name(
base_tool_name, server_prefix
)
# Get description
description = operation.get(
"summary",
operation.get("description", f"{method.upper()} {path}"),
)
# Build input schema using imported function
input_schema = build_input_schema(operation)
# Create tool function with headers using imported function
tool_func = create_tool_function(
path, method, operation, base_url, headers=headers
)
tool_func.__name__ = prefixed_tool_name
tool_func.__doc__ = description
# Register tool with prefixed name in global registry
global_mcp_tool_registry.register_tool(
name=prefixed_tool_name,
description=description,
input_schema=input_schema,
handler=tool_func,
)
# Update tool name to server name mapping (for both prefixed and base names)
self.tool_name_to_mcp_server_name_mapping[base_tool_name] = (
server_prefix
)
self.tool_name_to_mcp_server_name_mapping[prefixed_tool_name] = (
server_prefix
)
registered_count += 1
verbose_logger.debug(
f"Registered OpenAPI tool: {prefixed_tool_name} for server {server.name}"
)
verbose_logger.info(
f"Successfully registered {registered_count} OpenAPI tools for server {server.name}"
)
except Exception as e:
verbose_logger.error(
f"Failed to register OpenAPI tools for server {server.name}: {str(e)}"
)
raise e
def remove_server(self, mcp_server: LiteLLM_MCPServerTable):
"""
Remove a server from the registry
@ -469,6 +622,10 @@ class MCPServerManager:
Returns:
List[MCPTool]: List of tools available on the server with prefixed names
"""
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
verbose_logger.debug(f"Connecting to url: {server.url}")
verbose_logger.info(f"_get_tools_from_server for {server.name}...")
@ -481,7 +638,14 @@ class MCPServerManager:
extra_headers=extra_headers,
)
tools = await self._fetch_tools_with_timeout(client, server.name)
## HANDLE OPENAPI TOOLS
if server.spec_path:
_tools = global_mcp_tool_registry.list_tools(tool_prefix=server.name)
tools = global_mcp_tool_registry.convert_tools_to_mcp_sdk_tool_type(
_tools
)
else:
tools = await self._fetch_tools_with_timeout(client, server.name)
prefixed_or_original_tools = self._create_prefixed_tools(
tools, server, add_prefix=add_prefix
@ -597,11 +761,69 @@ class MCPServerManager:
Check if the tool is allowed or banned for the given server
"""
if server.allowed_tools:
return tool_name in server.allowed_tools
return (
tool_name in server.allowed_tools
or f"{server.name}-{tool_name}" in server.allowed_tools
)
if server.disallowed_tools:
return tool_name not in server.disallowed_tools
return (
tool_name not in server.disallowed_tools
and f"{server.name}-{tool_name}" not in server.disallowed_tools
)
return True
def validate_allowed_params(
self, tool_name: str, arguments: Dict[str, Any], server: MCPServer
) -> None:
"""
Filter arguments to only include allowed parameters for the given tool.
Args:
tool_name: Name of the tool (with or without prefix)
arguments: Dictionary of arguments to filter
server: MCPServer configuration
Returns:
Filtered dictionary containing only allowed parameters
Raises:
HTTPException: If allowed_params is configured for this tool but arguments contain disallowed params
"""
from litellm.proxy._experimental.mcp_server.utils import (
get_server_name_prefix_tool_mcp,
)
# If no allowed_params configured, return all arguments
if not server.allowed_params:
return
# Get the unprefixed tool name to match against config
unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(tool_name)
# Check both prefixed and unprefixed tool names
allowed_params_list = server.allowed_params.get(
tool_name
) or server.allowed_params.get(unprefixed_tool_name)
# If this tool doesn't have allowed_params specified, allow all params
if allowed_params_list is None:
return None
# Filter arguments to only include allowed parameters
disallowed_params = [
param for param in arguments.keys() if param not in allowed_params_list
]
if disallowed_params:
raise HTTPException(
status_code=403,
detail={
"error": f"Parameters {disallowed_params} are not allowed for tool {tool_name}. "
f"Allowed parameters: {allowed_params_list}. "
f"Contact proxy admin to allow these parameters."
},
)
async def check_tool_permission_for_key_team(
self,
tool_name: str,
@ -621,18 +843,20 @@ class MCPServerManager:
Raises:
HTTPException: If tool is not allowed for this key/team
"""
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import MCPRequestHandler
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
MCPRequestHandler,
)
if not user_api_key_auth:
return
# Check if tool is allowed
is_allowed = await MCPRequestHandler.is_tool_allowed_for_server(
tool_name=tool_name,
server_id=server.server_id,
user_api_key_auth=user_api_key_auth,
)
if not is_allowed:
raise HTTPException(
status_code=403,
@ -641,6 +865,64 @@ class MCPServerManager:
},
)
async def _call_openapi_tool_handler(
self,
server: MCPServer,
tool_name: str,
arguments: Dict[str, Any],
) -> CallToolResult:
"""
Call an OpenAPI tool handler directly.
For OpenAPI servers, instead of using MCP protocol, we call the tool handler
that was registered during OpenAPI spec parsing. This handler makes direct
HTTP requests to the API.
Args:
tool_name: The full tool name (with prefix) to call
arguments: Tool arguments to pass to the handler
Returns:
CallToolResult with the response from the API
"""
from mcp.types import TextContent
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
# Get the tool from the registry
tool = global_mcp_tool_registry.get_tool(f"{server.name}-{tool_name}")
if tool is None:
# Tool not found in registry
error_msg = f"OpenAPI tool {tool_name} not found in registry"
verbose_logger.error(error_msg)
return CallToolResult(
content=[TextContent(type="text", text=error_msg)],
isError=True,
)
try:
# Call the tool handler with the arguments
# The handler is an async function that makes the HTTP request
handler_result = await tool.handler(**arguments)
# Convert the handler result (string response) to CallToolResult format
result = CallToolResult(
content=[TextContent(type="text", text=str(handler_result))],
isError=False,
)
return result
except Exception as e:
error_msg = f"Error calling OpenAPI tool {tool_name}: {str(e)}"
verbose_logger.error(error_msg)
return CallToolResult(
content=[TextContent(type="text", text=error_msg)],
isError=True,
)
async def pre_call_tool_check(
self,
name: str,
@ -667,6 +949,13 @@ class MCPServerManager:
user_api_key_auth=user_api_key_auth,
)
## filter parameters based on allowed_params configuration
self.validate_allowed_params(
tool_name=name,
arguments=arguments,
server=server,
)
pre_hook_kwargs = {
"name": name,
"arguments": arguments,
@ -793,95 +1082,109 @@ class MCPServerManager:
server=mcp_server,
)
# Get server-specific auth header if available
server_auth_header: Optional[Union[Dict[str, str], str]] = None
if mcp_server_auth_headers and mcp_server.alias:
server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
elif mcp_server_auth_headers and mcp_server.server_name:
server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
# Prepare tasks for during hooks
tasks = []
if proxy_logging_obj:
# Create synthetic LLM data for during hook processing
from litellm.types.llms.base import HiddenParams
from litellm.types.mcp import MCPDuringCallRequestObject
# Fall back to deprecated mcp_auth_header if no server-specific header found
if server_auth_header is None:
server_auth_header = mcp_auth_header
# oauth2 headers
extra_headers: Optional[Dict[str, str]] = None
if mcp_server.auth_type == MCPAuth.oauth2:
extra_headers = oauth2_headers
if mcp_server.extra_headers and raw_headers:
if extra_headers is None:
extra_headers = {}
for header in mcp_server.extra_headers:
if header in raw_headers:
extra_headers[header] = raw_headers[header]
client = self._create_mcp_client(
server=mcp_server,
mcp_auth_header=server_auth_header,
extra_headers=extra_headers,
)
async with client:
# Use the original tool name (without prefix) for the actual call
call_tool_params = MCPCallToolRequestParams(
name=original_tool_name,
request_obj = MCPDuringCallRequestObject(
tool_name=name,
arguments=arguments,
server_name=server_name_from_prefix,
start_time=start_time.timestamp() if start_time else None,
hidden_params=HiddenParams(),
)
tasks = []
if proxy_logging_obj:
# Create synthetic LLM data for during hook processing
from litellm.types.llms.base import HiddenParams
from litellm.types.mcp import MCPDuringCallRequestObject
request_obj = MCPDuringCallRequestObject(
tool_name=name,
during_hook_kwargs = {
"name": name,
"arguments": arguments,
"server_name": server_name_from_prefix,
"user_api_key_auth": user_api_key_auth,
}
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
request_obj, during_hook_kwargs
)
during_hook_task = asyncio.create_task(
proxy_logging_obj.during_call_hook(
user_api_key_dict=user_api_key_auth,
data=synthetic_llm_data,
call_type="mcp_call", # type: ignore
)
)
tasks.append(during_hook_task)
# For OpenAPI servers, call the tool handler directly instead of via MCP client
if mcp_server.spec_path:
verbose_logger.debug(
f"Calling OpenAPI tool {name} directly via HTTP handler"
)
tasks.append(
asyncio.create_task(
self._call_openapi_tool_handler(mcp_server, name, arguments)
)
)
else:
# For regular MCP servers, use the MCP client
# Get server-specific auth header if available
server_auth_header: Optional[Union[Dict[str, str], str]] = None
if mcp_server_auth_headers and mcp_server.alias:
server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
elif mcp_server_auth_headers and mcp_server.server_name:
server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
# Fall back to deprecated mcp_auth_header if no server-specific header found
if server_auth_header is None:
server_auth_header = mcp_auth_header
# oauth2 headers
extra_headers: Optional[Dict[str, str]] = None
if mcp_server.auth_type == MCPAuth.oauth2:
extra_headers = oauth2_headers
if mcp_server.extra_headers and raw_headers:
if extra_headers is None:
extra_headers = {}
for header in mcp_server.extra_headers:
if header in raw_headers:
extra_headers[header] = raw_headers[header]
client = self._create_mcp_client(
server=mcp_server,
mcp_auth_header=server_auth_header,
extra_headers=extra_headers,
)
async with client:
# Use the original tool name (without prefix) for the actual call
call_tool_params = MCPCallToolRequestParams(
name=original_tool_name,
arguments=arguments,
server_name=server_name_from_prefix,
start_time=start_time.timestamp() if start_time else None,
hidden_params=HiddenParams(),
)
tasks.append(asyncio.create_task(client.call_tool(call_tool_params)))
during_hook_kwargs = {
"name": name,
"arguments": arguments,
"server_name": server_name_from_prefix,
"user_api_key_auth": user_api_key_auth,
}
try:
mcp_responses = await asyncio.gather(*tasks)
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
request_obj, during_hook_kwargs
)
# If proxy_logging_obj is None, the tool call result is at index 0
# If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
result_index = 1 if proxy_logging_obj else 0
result = mcp_responses[result_index]
during_hook_task = asyncio.create_task(
proxy_logging_obj.during_call_hook(
user_api_key_dict=user_api_key_auth,
data=synthetic_llm_data,
call_type="mcp_call", # type: ignore
)
)
tasks.append(during_hook_task)
tasks.append(asyncio.create_task(client.call_tool(call_tool_params)))
try:
mcp_responses = await asyncio.gather(*tasks)
# If proxy_logging_obj is None, the tool call result is at index 0
# If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
result_index = 1 if proxy_logging_obj else 0
result = mcp_responses[result_index]
return cast(CallToolResult, result)
except (
BlockedPiiEntityError,
GuardrailRaisedException,
HTTPException,
) as e:
# Re-raise guardrail exceptions to properly fail the MCP call
verbose_logger.error(
f"Guardrail blocked MCP tool call during result check: {str(e)}"
)
raise e
return cast(CallToolResult, result)
except (
BlockedPiiEntityError,
GuardrailRaisedException,
HTTPException,
) as e:
# Re-raise guardrail exceptions to properly fail the MCP call
verbose_logger.error(
f"Guardrail blocked MCP tool call during result check: {str(e)}"
)
raise e
#########################################################
# End of Methods that call the upstream MCP servers

236
litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py Normal file
View file

@ -0,0 +1,236 @@
"""
This module is used to generate MCP tools from OpenAPI specs.
"""
import json
from typing import Any, Dict, Optional
import httpx
from litellm._logging import verbose_logger
from litellm.proxy._experimental.mcp_server.tool_registry import (
global_mcp_tool_registry,
)
# Store the base URL and headers globally
BASE_URL = ""
HEADERS: Dict[str, str] = {}
def load_openapi_spec(filepath: str) -> Dict[str, Any]:
"""Load OpenAPI specification from JSON file."""
with open(filepath, "r") as f:
return json.load(f)
def get_base_url(spec: Dict[str, Any]) -> str:
"""Extract base URL from OpenAPI spec."""
# OpenAPI 3.x
if "servers" in spec and spec["servers"]:
return spec["servers"][0]["url"]
# OpenAPI 2.x (Swagger)
elif "host" in spec:
scheme = spec.get("schemes", ["https"])[0]
base_path = spec.get("basePath", "")
return f"{scheme}://{spec['host']}{base_path}"
return ""
def extract_parameters(operation: Dict[str, Any]) -> tuple:
"""Extract parameter names from OpenAPI operation."""
path_params = []
query_params = []
body_params = []
# OpenAPI 3.x and 2.x parameters
if "parameters" in operation:
for param in operation["parameters"]:
param_name = param["name"]
if param.get("in") == "path":
path_params.append(param_name)
elif param.get("in") == "query":
query_params.append(param_name)
elif param.get("in") == "body":
body_params.append(param_name)
# OpenAPI 3.x requestBody
if "requestBody" in operation:
body_params.append("body")
return path_params, query_params, body_params
def build_input_schema(operation: Dict[str, Any]) -> Dict[str, Any]:
"""Build MCP input schema from OpenAPI operation."""
properties = {}
required = []
# Process parameters
if "parameters" in operation:
for param in operation["parameters"]:
param_name = param["name"]
param_schema = param.get("schema", {})
param_type = param_schema.get("type", "string")
properties[param_name] = {
"type": param_type,
"description": param.get("description", ""),
}
if param.get("required", False):
required.append(param_name)
# Process requestBody (OpenAPI 3.x)
if "requestBody" in operation:
request_body = operation["requestBody"]
content = request_body.get("content", {})
# Try to get JSON schema
if "application/json" in content:
schema = content["application/json"].get("schema", {})
properties["body"] = {
"type": "object",
"description": request_body.get("description", "Request body"),
"properties": schema.get("properties", {}),
}
if request_body.get("required", False):
required.append("body")
return {
"type": "object",
"properties": properties,
"required": required if required else [],
}
def create_tool_function(
path: str,
method: str,
operation: Dict[str, Any],
base_url: str,
headers: Optional[Dict[str, str]] = None,
):
"""Create a tool function for an OpenAPI operation.
Args:
path: API endpoint path
method: HTTP method (get, post, put, delete, patch)
operation: OpenAPI operation object
base_url: Base URL for the API
headers: Optional headers to include in requests (e.g., authentication)
"""
if headers is None:
headers = {}
path_params, query_params, body_params = extract_parameters(operation)
all_params = path_params + query_params + body_params
# Build function signature dynamically
if all_params:
params_str = ", ".join(f"{p}: str = ''" for p in all_params)
else:
params_str = ""
# Create the function code as a string
func_code = f'''
async def tool_function({params_str}) -> str:
"""Dynamically generated tool function."""
url = base_url + path
# Replace path parameters
path_param_names = {path_params}
for param_name in path_param_names:
param_value = locals().get(param_name, "")
if param_value:
url = url.replace("{{" + param_name + "}}", str(param_value))
# Build query params
query_param_names = {query_params}
params = {{}}
for param_name in query_param_names:
param_value = locals().get(param_name, "")
if param_value:
params[param_name] = param_value
# Build request body
body_param_names = {body_params}
json_body = None
if body_param_names:
body_value = locals().get("body", {{}})
if isinstance(body_value, dict):
json_body = body_value
elif body_value:
# If it's a string, try to parse as JSON
import json as json_module
try:
json_body = json_module.loads(body_value) if isinstance(body_value, str) else {{"data": body_value}}
except:
json_body = {{"data": body_value}}
# Make HTTP request
async with httpx.AsyncClient() as client:
if "{method.lower()}" == "get":
response = await client.get(url, params=params, headers=headers)
elif "{method.lower()}" == "post":
response = await client.post(url, params=params, json=json_body, headers=headers)
elif "{method.lower()}" == "put":
response = await client.put(url, params=params, json=json_body, headers=headers)
elif "{method.lower()}" == "delete":
response = await client.delete(url, params=params, headers=headers)
elif "{method.lower()}" == "patch":
response = await client.patch(url, params=params, json=json_body, headers=headers)
else:
return "Unsupported HTTP method: {method}"
return response.text
'''
# Execute the function code to create the actual function
local_vars = {
"httpx": httpx,
"headers": headers,
"base_url": base_url,
"path": path,
"method": method,
}
exec(func_code, local_vars)
return local_vars["tool_function"]
def register_tools_from_openapi(spec: Dict[str, Any], base_url: str):
"""Register MCP tools from OpenAPI specification."""
paths = spec.get("paths", {})
for path, path_item in paths.items():
for method in ["get", "post", "put", "delete", "patch"]:
if method in path_item:
operation = path_item[method]
# Generate tool name
operation_id = operation.get(
"operationId", f"{method}_{path.replace('/', '_')}"
)
tool_name = operation_id.replace(" ", "_").lower()
# Get description
description = operation.get(
"summary", operation.get("description", f"{method.upper()} {path}")
)
# Build input schema
input_schema = build_input_schema(operation)
# Create tool function
tool_func = create_tool_function(path, method, operation, base_url)
tool_func.__name__ = tool_name
tool_func.__doc__ = description
# Register tool with local registry
global_mcp_tool_registry.register_tool(
name=tool_name,
description=description,
input_schema=input_schema,
handler=tool_func,
)
verbose_logger.debug(f"Registered tool: {tool_name}")

131
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -364,25 +364,25 @@ if MCP_AVAILABLE:
def _tool_name_matches(tool_name: str, filter_list: List[str]) -> bool:
"""
Check if a tool name matches any name in the filter list.
Checks both the full tool name and unprefixed version (without server prefix).
This allows users to configure simple tool names regardless of prefixing.
Args:
tool_name: The tool name to check (may be prefixed like "server-tool_name")
filter_list: List of tool names to match against
Returns:
True if the tool name (prefixed or unprefixed) is in the filter list
"""
from litellm.proxy._experimental.mcp_server.utils import (
get_server_name_prefix_tool_mcp,
)
# Check if the full name is in the list
if tool_name in filter_list:
return True
# Check if the unprefixed name is in the list
unprefixed_name, _ = get_server_name_prefix_tool_mcp(tool_name)
return unprefixed_name in filter_list
@ -393,34 +393,36 @@ if MCP_AVAILABLE:
) -> List[MCPTool]:
"""
Filter tools by allowed/disallowed tools configuration.
If allowed_tools is set, only tools in that list are returned.
If disallowed_tools is set, tools in that list are excluded.
Tool names are matched with and without server prefixes for flexibility.
Args:
tools: List of tools to filter
mcp_server: Server configuration with allowed_tools/disallowed_tools
Returns:
Filtered list of tools
"""
tools_to_return = tools
# Filter by allowed_tools (whitelist)
if mcp_server.allowed_tools:
tools_to_return = [
tool for tool in tools
tool
for tool in tools
if _tool_name_matches(tool.name, mcp_server.allowed_tools)
]
# Filter by disallowed_tools (blacklist)
if mcp_server.disallowed_tools:
tools_to_return = [
tool for tool in tools_to_return
tool
for tool in tools_to_return
if not _tool_name_matches(tool.name, mcp_server.disallowed_tools)
]
return tools_to_return
async def _get_tools_from_mcp_servers(
@ -497,17 +499,17 @@ if MCP_AVAILABLE:
extra_headers=extra_headers,
add_prefix=add_prefix,
)
filtered_tools = filter_tools_by_allowed_tools(tools, server)
filtered_tools = await filter_tools_by_key_team_permissions(
tools=filtered_tools,
server_id=server_id,
user_api_key_auth=user_api_key_auth,
)
all_tools.extend(filtered_tools)
verbose_logger.debug(
f"Successfully fetched {len(tools)} tools from server {server.name}, {len(filtered_tools)} after filtering"
)
@ -520,6 +522,7 @@ if MCP_AVAILABLE:
verbose_logger.info(
f"Successfully fetched {len(all_tools)} tools total from all MCP servers"
)
return all_tools
async def filter_tools_by_key_team_permissions(
@ -529,7 +532,7 @@ if MCP_AVAILABLE:
) -> List[MCPTool]:
"""
Filter tools based on key/team mcp_tool_permissions.
Note: Tool names in the DB are stored without server prefixes,
but tool names from MCP servers are prefixed. We need to strip
the prefix before comparing.
@ -551,7 +554,7 @@ if MCP_AVAILABLE:
else:
# No restrictions, return all tools
filtered_tools = tools
return filtered_tools
async def _list_mcp_tools(
@ -596,30 +599,7 @@ if MCP_AVAILABLE:
)
# Continue with empty managed tools list instead of failing completely
# Get tools from local registry
local_tools = []
try:
local_tools_raw = global_mcp_tool_registry.list_tools()
# Convert local tools to MCPTool format
for tool in local_tools_raw:
# Convert from litellm.types.mcp_server.tool_registry.MCPTool to mcp.types.Tool
mcp_tool = MCPTool(
name=tool.name,
description=tool.description,
inputSchema=tool.input_schema,
)
local_tools.append(mcp_tool)
except Exception as e:
verbose_logger.exception(
f"Error getting tools from local registry: {str(e)}"
)
# Continue with empty local tools list instead of failing completely
# Combine all tools
all_tools = managed_tools + local_tools
return all_tools
return managed_tools
@client
async def call_mcp_tool(
@ -680,33 +660,42 @@ if MCP_AVAILABLE:
standard_logging_mcp_tool_call
)
litellm_logging_obj.model = f"MCP: {name}"
# Try managed server tool first (pass the full prefixed name)
# Primary and recommended way to use MCP servers
# Check if tool exists in local registry first (for OpenAPI-based tools)
# These tools are registered with their prefixed names
#########################################################
mcp_server: Optional[MCPServer] = (
global_mcp_server_manager._get_mcp_server_from_tool_name(name)
)
if mcp_server:
standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
mcp_server.mcp_info or {}
).get("mcp_server_cost_info")
response = await _handle_managed_mcp_tool(
name=name, # Pass the full name (potentially prefixed)
arguments=arguments,
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
local_tool = global_mcp_tool_registry.get_tool(name)
if local_tool:
verbose_logger.debug(f"Executing local registry tool: {name}")
response = await _handle_local_mcp_tool(name, arguments)
# Fall back to local tool registry (use original name)
#########################################################
# Deprecated: Local MCP Server Tool
# Try managed MCP server tool (pass the full prefixed name)
# Primary and recommended way to use external MCP servers
#########################################################
else:
response = await _handle_local_mcp_tool(original_tool_name, arguments)
mcp_server: Optional[MCPServer] = (
global_mcp_server_manager._get_mcp_server_from_tool_name(name)
)
if mcp_server:
standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
mcp_server.mcp_info or {}
).get("mcp_server_cost_info")
response = await _handle_managed_mcp_tool(
name=name, # Pass the full name (potentially prefixed)
arguments=arguments,
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
# Fall back to local tool registry with original name (legacy support)
#########################################################
# Deprecated: Local MCP Server Tool
#########################################################
else:
response = await _handle_local_mcp_tool(original_tool_name, arguments)
#########################################################
# Post MCP Tool Call Hook
@ -778,14 +767,21 @@ if MCP_AVAILABLE:
Handle tool execution for local registry tools
Note: Local tools don't use prefixes, so we use the original name
"""
import inspect
tool = global_mcp_tool_registry.get_tool(name)
if not tool:
raise HTTPException(status_code=404, detail=f"Tool '{name}' not found")
try:
result = tool.handler(**arguments)
# Check if handler is async or sync
if inspect.iscoroutinefunction(tool.handler):
result = await tool.handler(**arguments)
else:
result = tool.handler(**arguments)
return [TextContent(text=str(result), type="text")]
except Exception as e:
verbose_logger.exception(f"Error executing local tool {name}: {str(e)}")
return [TextContent(text=f"Error: {str(e)}", type="text")]
def _get_mcp_servers_in_path(path: str) -> Optional[List[str]]:
@ -906,6 +902,7 @@ if MCP_AVAILABLE:
await session_manager.handle_request(scope, receive, send)
except Exception as e:
raise e
verbose_logger.exception(f"Error handling MCP request: {e}")
# Instead of re-raising, try to send a graceful error response
try:

22
litellm/proxy/_experimental/mcp_server/tool_registry.py
View file

@ -1,6 +1,8 @@
import json
from typing import Any, Callable, Dict, List, Optional
from mcp.types import Tool as MCPToolSDKTool
from litellm._logging import verbose_logger
from litellm.proxy.types_utils.utils import get_instance_fn
from litellm.types.mcp_server.tool_registry import MCPTool
@ -39,12 +41,30 @@ class MCPToolRegistry:
"""
return self.tools.get(name)
def list_tools(self) -> List[MCPTool]:
def list_tools(self, tool_prefix: Optional[str] = None) -> List[MCPTool]:
"""
List all registered tools
"""
if tool_prefix:
return [
tool
for tool in self.tools.values()
if tool.name.startswith(tool_prefix)
]
return list(self.tools.values())
def convert_tools_to_mcp_sdk_tool_type(
self, tools: List[MCPTool]
) -> List[MCPToolSDKTool]:
return [
MCPToolSDKTool(
name=tool.name,
description=tool.description,
inputSchema=tool.input_schema,
)
for tool in tools
]
def load_tools_from_config(
self, mcp_tools_config: Optional[Dict[str, Any]] = None
) -> None:

1
litellm/proxy/_experimental/out/_next/static/0NP4pGCTzzyECenXO-S3V/_buildManifest.js
View file

@ -1 +0,0 @@
self.__BUILD_MANIFEST={__rewrites:{afterFiles:[],beforeFiles:[],fallback:[]},"/_error":["static/chunks/pages/_error-28b803cb2479b966.js"],sortedPages:["/_app","/_error"]},self.__BUILD_MANIFEST_CB&&self.__BUILD_MANIFEST_CB();

1
litellm/proxy/_experimental/out/_next/static/chunks/1039-1031e149b66e294d.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1052-80a465c977e6b146.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1160-3efb81c958413447.js Normal file
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/117-6a9f6a591033d4a0.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1223-de5e7e4f043a5233.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1264-2979d95e0b56a75c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1307-6b882871684ce94c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/13b76428-e1bf383848c17260.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/13b76428-ebdf3012af0e4489.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1487-ada9ecf7dd9bca97.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1491-80fe911f18f4631c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1526-8e2976ed10cb2fa0.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1529-e0933e3af843b646.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/162-b79fa8ec9408cd55.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1633-0db2561cf68d80bc.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/17-782feb91c41095ca.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/1739-5149135b6e98ac40.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2004-946c4fd2a52c86d0.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2019-d4e12fe6a3ca4e25.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2052-68db39dea49a676f.js Normal file
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/2117-2da5ec8fad6a6c25.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2162-947823f263cd4e0b.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2202-563a3700a73a4a11.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/226-81daaf8cff08ccfe.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2284-4cbc9a7f33eb7c89.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2344-169e12738d6439ab.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2417-b28f330f82bf4a82.js Normal file
View file

File diff suppressed because one or more lines are too long

4
litellm/proxy/_experimental/out/_next/static/chunks/2525-13b137f40949dcf1.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2662-51eb8b1bec576f6d.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2842-3febfc5e0bb91eff.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/2926-b5dcad9f62f5e2b1.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/293-a7c1c04c415a418e.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3014691f-702e24806fe9cec4.js Normal file
View file

@ -0,0 +1 @@
"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[3665],{84566:function(e,t,s){s.d(t,{GH$:function(){return l}});var c=s(2265);let l=({color:e="currentColor",size:t=24,className:s,...l})=>c.createElement("svg",{viewBox:"0 0 24 24",xmlns:"http://www.w3.org/2000/svg",width:t,height:t,fill:e,...l,className:"remixicon "+(s||"")},c.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM11.0026 16L6.75999 11.7574L8.17421 10.3431L11.0026 13.1716L16.6595 7.51472L18.0737 8.92893L11.0026 16Z"}))}}]);

1
litellm/proxy/_experimental/out/_next/static/chunks/3014691f-b7b79b78e27792f3.js
View file

@ -1 +0,0 @@
"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[665],{84566:function(e,t,s){s.d(t,{GH$:function(){return l}});var c=s(2265);let l=({color:e="currentColor",size:t=24,className:s,...l})=>c.createElement("svg",{viewBox:"0 0 24 24",xmlns:"http://www.w3.org/2000/svg",width:t,height:t,fill:e,...l,className:"remixicon "+(s||"")},c.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM11.0026 16L6.75999 11.7574L8.17421 10.3431L11.0026 13.1716L16.6595 7.51472L18.0737 8.92893L11.0026 16Z"}))}}]);

12
litellm/proxy/_experimental/out/_next/static/chunks/313-0025fb08e386c4b8.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3202-10bb7b948f3a8eb2.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3250-3256164511237d25.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3298-8d1992a5407c2cae.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3603-b101c17ea3d68f19.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3669-cbf664b1e9c58f8a.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3709-b17db86e0ab325d2.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3752-53808701995a5f10.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3801-4bb3b8116c02dad5.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3866-e3419825a249263f.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4292-5c3ab7c3b62e1965.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4365-a008db87ace8b7cb.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4696-f31deddc1ce81a6b.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4851-fbdd7aec2937c09d.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/49-036448f42c39b97c.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4924-e47559a81a4aa31c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/50-d0da2dd7acce2eb9.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5079-a43d5cc0d7429256.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5096-4ef058eb8abce1d7.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5105-eb18802ec448789d.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5188-0e9d5e6db19ac9e6.js Normal file
View file

File diff suppressed because one or more lines are too long

4
litellm/proxy/_experimental/out/_next/static/chunks/521-d97d355792d44830.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/527-d9b7316e990a0539.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5319-43bd4ee5bb7e50d1.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5809-33437b27a2ac76d5.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/5830-47ce1a4897188f14.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/603-4d26e1c19f911024.js Normal file
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/_next/static/chunks/6433-c9b3a95c5de0b59f.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/6494-7124dea6b90175e7.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/6836-e30124a71aafae62.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/6925-5f3f11523497a729.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7155-f019e996be6de8e3.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7281-41cef56aa2b3df92.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7382-c293a81095a18d6c.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7406-8dbad47afbc571f1.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/773-da80cf7b30837a84.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7851-62c5dbe52d2b953b.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7906-11071e9e2e7b8318.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/7908-07a76cfe29c543c9.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/8049-0a4ef0c3396390d0.js Normal file
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more