mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
Merge branch 'BerriAI:main' into wandb-inference-docs
This commit is contained in:
Anubhav Singh
GitHub
commit
f59f1971cc
341 changed files with 7994 additions and 1185 deletions
|
|
@ -2,7 +2,7 @@ import Tabs from '@theme/Tabs';
|
|||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# /mcp - Model Context Protocol
|
||||
# MCP Overview
|
||||
|
||||
LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint for all MCP tools and control MCP access by Key, Team.
|
||||
|
||||
|
|
@ -246,8 +246,203 @@ litellm_settings:
|
|||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## MCP Tool Filtering
|
||||
## Converting OpenAPI Specs to MCP Servers
|
||||
|
||||
LiteLLM can automatically convert OpenAPI specifications into MCP servers, allowing you to expose any REST API as MCP tools. This is useful when you have existing APIs with OpenAPI/Swagger documentation and want to make them available as MCP tools.
|
||||
|
||||
### Benefits
|
||||
|
||||
- **Rapid Integration**: Convert existing APIs to MCP tools without writing custom MCP server code
|
||||
- **Automatic Tool Generation**: LiteLLM automatically generates MCP tools from your OpenAPI spec
|
||||
- **Unified Interface**: Use the same MCP interface for both native MCP servers and OpenAPI-based APIs
|
||||
- **Easy Testing**: Test and iterate on API integrations quickly
|
||||
|
||||
### Configuration
|
||||
|
||||
Add your OpenAPI-based MCP server to your `config.yaml`:
|
||||
|
||||
```yaml title="config.yaml - OpenAPI to MCP" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-xxxxxxx
|
||||
|
||||
mcp_servers:
|
||||
# OpenAPI Spec Example - Petstore API
|
||||
petstore_mcp:
|
||||
url: "https://petstore.swagger.io/v2"
|
||||
spec_path: "/path/to/openapi.json"
|
||||
auth_type: "none"
|
||||
|
||||
# OpenAPI Spec with API Key Authentication
|
||||
my_api_mcp:
|
||||
url: "http://0.0.0.0:8090"
|
||||
spec_path: "/path/to/openapi.json"
|
||||
auth_type: "api_key"
|
||||
auth_value: "your-api-key-here"
|
||||
|
||||
# OpenAPI Spec with Bearer Token
|
||||
secured_api_mcp:
|
||||
url: "https://api.example.com"
|
||||
spec_path: "/path/to/openapi.json"
|
||||
auth_type: "bearer_token"
|
||||
auth_value: "your-bearer-token"
|
||||
```
|
||||
|
||||
### Configuration Parameters
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| `url` | Yes | The base URL of your API endpoint |
|
||||
| `spec_path` | Yes | Path or URL to your OpenAPI specification file (JSON or YAML) |
|
||||
| `auth_type` | No | Authentication type: `none`, `api_key`, `bearer_token`, `basic`, `authorization` |
|
||||
| `auth_value` | No | Authentication value (required if `auth_type` is set) |
|
||||
| `description` | No | Optional description for the MCP server |
|
||||
| `allowed_tools` | No | List of specific tools to allow (see [MCP Tool Filtering](#mcp-tool-filtering)) |
|
||||
| `disallowed_tools` | No | List of specific tools to block (see [MCP Tool Filtering](#mcp-tool-filtering)) |
|
||||
|
||||
### Usage Example
|
||||
|
||||
Once configured, you can use the OpenAPI-based MCP server just like any other MCP server:
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="fastmcp" label="Python FastMCP">
|
||||
|
||||
```python title="Using OpenAPI-based MCP Server" showLineNumbers
|
||||
from fastmcp import Client
|
||||
import asyncio
|
||||
|
||||
# Standard MCP configuration
|
||||
config = {
|
||||
"mcpServers": {
|
||||
"petstore": {
|
||||
"url": "http://localhost:4000/petstore_mcp/mcp",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer sk-1234"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Create a client that connects to the server
|
||||
client = Client(config)
|
||||
|
||||
async def main():
|
||||
async with client:
|
||||
# List available tools generated from OpenAPI spec
|
||||
tools = await client.list_tools()
|
||||
print(f"Available tools: {[tool.name for tool in tools]}")
|
||||
|
||||
# Example: Get a pet by ID (from Petstore API)
|
||||
response = await client.call_tool(
|
||||
name="getpetbyid",
|
||||
arguments={"petId": "1"}
|
||||
)
|
||||
print(f"Response:\n{response}\n")
|
||||
|
||||
# Example: Find pets by status
|
||||
response = await client.call_tool(
|
||||
name="findpetsbystatus",
|
||||
arguments={"status": "available"}
|
||||
)
|
||||
print(f"Response:\n{response}\n")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="cursor" label="Cursor IDE">
|
||||
|
||||
```json title="Cursor MCP Configuration for OpenAPI Server" showLineNumbers
|
||||
{
|
||||
"mcpServers": {
|
||||
"Petstore": {
|
||||
"url": "http://localhost:4000/petstore_mcp/mcp",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="openai" label="OpenAI Responses API">
|
||||
|
||||
```bash title="Using OpenAPI MCP Server with OpenAI" showLineNumbers
|
||||
curl --location 'https://api.openai.com/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
--data '{
|
||||
"model": "gpt-4o",
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "petstore",
|
||||
"server_url": "http://localhost:4000/petstore_mcp/mcp",
|
||||
"require_approval": "never",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY"
|
||||
}
|
||||
}
|
||||
],
|
||||
"input": "Find all available pets in the petstore",
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Spec Loading**: LiteLLM loads your OpenAPI specification from the provided `spec_path`
|
||||
2. **Tool Generation**: Each API endpoint in the spec becomes an MCP tool
|
||||
3. **Parameter Mapping**: OpenAPI parameters are automatically mapped to MCP tool parameters
|
||||
4. **Request Handling**: When a tool is called, LiteLLM converts the MCP request to the appropriate HTTP request
|
||||
5. **Response Translation**: API responses are converted back to MCP format
|
||||
|
||||
### OpenAPI Spec Requirements
|
||||
|
||||
Your OpenAPI specification should follow standard OpenAPI/Swagger conventions:
|
||||
- **Supported versions**: OpenAPI 3.0.x, OpenAPI 3.1.x, Swagger 2.0
|
||||
- **Required fields**: `paths`, `info` sections should be properly defined
|
||||
- **Operation IDs**: Each operation should have a unique `operationId` (this becomes the tool name)
|
||||
- **Parameters**: Request parameters should be properly documented with types and descriptions
|
||||
|
||||
### Example OpenAPI Spec Structure
|
||||
|
||||
```yaml title="sample-openapi.yaml" showLineNumbers
|
||||
openapi: 3.0.0
|
||||
info:
|
||||
title: My API
|
||||
version: 1.0.0
|
||||
paths:
|
||||
/pets/{petId}:
|
||||
get:
|
||||
operationId: getPetById
|
||||
summary: Get a pet by ID
|
||||
parameters:
|
||||
- name: petId
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: integer
|
||||
responses:
|
||||
'200':
|
||||
description: Successful response
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
```
|
||||
|
||||
## Allow/Disallow MCP Tools
|
||||
|
||||
Control which tools are available from your MCP servers. You can either allow only specific tools or block dangerous ones.
|
||||
|
||||
<Tabs>
|
||||
|
|
@ -306,210 +501,118 @@ mcp_servers:
|
|||
- If you specify both `allowed_tools` and `disallowed_tools`, the allowed list takes priority
|
||||
- Tool names are case-sensitive
|
||||
|
||||
## Using your MCP
|
||||
---
|
||||
|
||||
### Use on LiteLLM UI
|
||||
## Allow/Disallow MCP Tool Parameters
|
||||
|
||||
Follow this walkthrough to use your MCP on LiteLLM UI
|
||||
Control which parameters are allowed for specific MCP tools using the `allowed_params` configuration. This provides fine-grained control over tool usage by restricting the parameters that can be passed to each tool.
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/57e0763267254bc79dbe6658d0b8758c" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
### Configuration
|
||||
|
||||
### Use with Responses API
|
||||
`allowed_params` is a dictionary that maps tool names to lists of allowed parameter names. When configured, only the specified parameters will be accepted for that tool - any other parameters will be rejected with a 403 error.
|
||||
|
||||
Replace `http://localhost:4000` with your LiteLLM Proxy base URL.
|
||||
|
||||
Demo Video Using Responses API with LiteLLM Proxy: [Demo video here](https://www.loom.com/share/34587e618c5c47c0b0d67b4e4d02718f?sid=2caf3d45-ead4-4490-bcc1-8d6dd6041c02)
|
||||
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="cURL">
|
||||
|
||||
```bash title="cURL Example" showLineNumbers
|
||||
curl --location 'http://localhost:4000/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer sk-1234" \
|
||||
--data '{
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never"
|
||||
}
|
||||
],
|
||||
"stream": true,
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```yaml title="config.yaml with allowed_params" showLineNumbers
|
||||
mcp_servers:
|
||||
deepwiki_mcp:
|
||||
url: https://mcp.deepwiki.com/mcp
|
||||
transport: "http"
|
||||
auth_type: "none"
|
||||
allowed_params:
|
||||
# Tool name: list of allowed parameters
|
||||
read_wiki_contents: ["status"]
|
||||
|
||||
my_api_mcp:
|
||||
url: "https://my-api-server.com"
|
||||
auth_type: "api_key"
|
||||
auth_value: "my-key"
|
||||
allowed_params:
|
||||
# Using unprefixed tool name
|
||||
getpetbyid: ["status"]
|
||||
# Using prefixed tool name (both formats work)
|
||||
my_api_mcp-findpetsbystatus: ["status", "limit"]
|
||||
# Another tool with multiple allowed params
|
||||
create_issue: ["title", "body", "labels"]
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python SDK">
|
||||
### How It Works
|
||||
|
||||
```python title="Python SDK Example" showLineNumbers
|
||||
"""
|
||||
Use LiteLLM Proxy MCP Gateway to call MCP tools.
|
||||
1. **Tool-specific filtering**: Each tool can have its own list of allowed parameters
|
||||
2. **Flexible naming**: Tool names can be specified with or without the server prefix (e.g., both `"getpetbyid"` and `"my_api_mcp-getpetbyid"` work)
|
||||
3. **Whitelist approach**: Only parameters in the allowed list are permitted
|
||||
4. **Unlisted tools**: If `allowed_params` is not set, all parameters are allowed
|
||||
5. **Error handling**: Requests with disallowed parameters receive a 403 error with details about which parameters are allowed
|
||||
|
||||
When using LiteLLM Proxy, you can use the same MCP tools across all your LLM providers.
|
||||
"""
|
||||
import openai
|
||||
### Example Request Behavior
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="sk-1234", # paste your litellm proxy api key here
|
||||
base_url="http://localhost:4000" # paste your litellm proxy base url here
|
||||
)
|
||||
print("Making API request to Responses API with MCP tools")
|
||||
With the configuration above, here's how requests would be handled:
|
||||
|
||||
response = client.responses.create(
|
||||
model="gpt-5",
|
||||
input=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
tools=[
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never"
|
||||
}
|
||||
],
|
||||
stream=True,
|
||||
tool_choice="required"
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
print("response chunk: ", chunk)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
#### Specifying MCP Tools
|
||||
|
||||
You can specify which MCP tools are available by using the `allowed_tools` parameter. This allows you to restrict access to specific tools within an MCP server.
|
||||
|
||||
To get the list of allowed tools when using LiteLLM MCP Gateway, you can naigate to the LiteLLM UI on MCP Servers > MCP Tools > Click the Tool > Copy Tool Name.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="cURL">
|
||||
|
||||
```bash title="cURL Example with allowed_tools" showLineNumbers
|
||||
curl --location 'http://localhost:4000/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer sk-1234" \
|
||||
--data '{
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy/mcp",
|
||||
"require_approval": "never",
|
||||
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
|
||||
}
|
||||
],
|
||||
"stream": true,
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python SDK">
|
||||
|
||||
```python title="Python SDK Example with allowed_tools" showLineNumbers
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="sk-1234",
|
||||
base_url="http://localhost:4000"
|
||||
)
|
||||
|
||||
response = client.responses.create(
|
||||
model="gpt-5",
|
||||
input=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
tools=[
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy/mcp",
|
||||
"require_approval": "never",
|
||||
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
|
||||
}
|
||||
],
|
||||
stream=True,
|
||||
tool_choice="required"
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Use with Cursor IDE
|
||||
|
||||
Use tools directly from Cursor IDE with LiteLLM MCP:
|
||||
|
||||
**Setup Instructions:**
|
||||
|
||||
1. **Open Cursor Settings**: Use `⇧+⌘+J` (Mac) or `Ctrl+Shift+J` (Windows/Linux)
|
||||
2. **Navigate to MCP Tools**: Go to the "MCP Tools" tab and click "New MCP Server"
|
||||
3. **Add Configuration**: Copy and paste the JSON configuration below, then save with `Cmd+S` or `Ctrl+S`
|
||||
|
||||
```json title="Basic Cursor MCP Configuration" showLineNumbers
|
||||
**✅ Allowed Request:**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"LiteLLM": {
|
||||
"url": "litellm_proxy",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
|
||||
}
|
||||
}
|
||||
"tool": "read_wiki_contents",
|
||||
"arguments": {
|
||||
"status": "active"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### How it works when server_url="litellm_proxy"
|
||||
**❌ Rejected Request:**
|
||||
```json
|
||||
{
|
||||
"tool": "read_wiki_contents",
|
||||
"arguments": {
|
||||
"status": "active",
|
||||
"limit": 10 // This parameter is not allowed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When server_url="litellm_proxy", LiteLLM bridges non-MCP providers to your MCP tools.
|
||||
**Error Response:**
|
||||
```json
|
||||
{
|
||||
"error": "Parameters ['limit'] are not allowed for tool read_wiki_contents. Allowed parameters: ['status']. Contact proxy admin to allow these parameters."
|
||||
}
|
||||
```
|
||||
|
||||
- Tool Discovery: LiteLLM fetches MCP tools and converts them to OpenAI-compatible definitions
|
||||
- LLM Call: Tools are sent to the LLM with your input; LLM selects which tools to call
|
||||
- Tool Execution: LiteLLM automatically parses arguments, routes calls to MCP servers, executes tools, and retrieves results
|
||||
- Response Integration: Tool results are sent back to LLM for final response generation
|
||||
- Output: Complete response combining LLM reasoning with tool execution results
|
||||
### Use Cases
|
||||
|
||||
This enables MCP tool usage with any LiteLLM-supported provider, regardless of native MCP support.
|
||||
- **Security**: Prevent users from accessing sensitive parameters or dangerous operations
|
||||
- **Cost control**: Restrict expensive parameters (e.g., limiting result counts)
|
||||
- **Compliance**: Enforce parameter usage policies for regulatory requirements
|
||||
- **Staged rollouts**: Gradually enable parameters as tools are tested
|
||||
- **Multi-tenant isolation**: Different parameter access for different user groups
|
||||
|
||||
#### Auto-execution for require_approval: "never"
|
||||
### Combining with Tool Filtering
|
||||
|
||||
Setting require_approval: "never" triggers automatic tool execution, returning the final response in a single API call without additional user interaction.
|
||||
`allowed_params` works alongside `allowed_tools` and `disallowed_tools` for complete control:
|
||||
|
||||
```yaml title="Combined filtering example" showLineNumbers
|
||||
mcp_servers:
|
||||
github_mcp:
|
||||
url: "https://api.githubcopilot.com/mcp"
|
||||
auth_type: oauth2
|
||||
authorization_url: https://github.com/login/oauth/authorize
|
||||
token_url: https://github.com/login/oauth/access_token
|
||||
client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
|
||||
client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
|
||||
scopes: ["public_repo", "user:email"]
|
||||
# Only allow specific tools
|
||||
allowed_tools: ["create_issue", "list_issues", "search_issues"]
|
||||
# Block dangerous operations
|
||||
disallowed_tools: ["delete_repo"]
|
||||
# Restrict parameters per tool
|
||||
allowed_params:
|
||||
create_issue: ["title", "body", "labels"]
|
||||
list_issues: ["state", "sort", "perPage"]
|
||||
search_issues: ["query", "sort", "order", "perPage"]
|
||||
```
|
||||
|
||||
This configuration ensures that:
|
||||
1. Only the three listed tools are available
|
||||
2. The `delete_repo` tool is explicitly blocked
|
||||
3. Each tool can only use its specified parameters
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Access Control
|
||||
|
||||
|
|
@ -1489,221 +1592,6 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
|
|||
}'
|
||||
```
|
||||
|
||||
|
||||
|
||||
## MCP Cost Tracking
|
||||
|
||||
LiteLLM provides two ways to track costs for MCP tool calls:
|
||||
|
||||
| Method | When to Use | What It Does |
|
||||
|--------|-------------|--------------|
|
||||
| **Config-based Cost Tracking** | Simple cost tracking with fixed costs per tool/server | Automatically tracks costs based on configuration |
|
||||
| **Custom Post-MCP Hook** | Dynamic cost tracking with custom logic | Allows custom cost calculations and response modifications |
|
||||
|
||||
### Config-based Cost Tracking
|
||||
|
||||
Configure fixed costs for MCP servers directly in your config.yaml:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-xxxxxxx
|
||||
|
||||
mcp_servers:
|
||||
zapier_server:
|
||||
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
|
||||
mcp_info:
|
||||
mcp_server_cost_info:
|
||||
# Default cost for all tools in this server
|
||||
default_cost_per_query: 0.01
|
||||
# Custom cost for specific tools
|
||||
tool_name_to_cost_per_query:
|
||||
send_email: 0.05
|
||||
create_document: 0.03
|
||||
|
||||
expensive_api_server:
|
||||
url: "https://api.expensive-service.com/mcp"
|
||||
mcp_info:
|
||||
mcp_server_cost_info:
|
||||
default_cost_per_query: 1.50
|
||||
```
|
||||
|
||||
### Custom Post-MCP Hook
|
||||
|
||||
Use this when you need dynamic cost calculation or want to modify the MCP response before it's returned to the user.
|
||||
|
||||
#### 1. Create a custom MCP hook file
|
||||
|
||||
```python title="custom_mcp_hook.py" showLineNumbers
|
||||
from typing import Optional
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.types.mcp import MCPPostCallResponseObject
|
||||
|
||||
|
||||
class CustomMCPCostTracker(CustomLogger):
|
||||
"""
|
||||
Custom handler for MCP cost tracking and response modification
|
||||
"""
|
||||
|
||||
async def async_post_mcp_tool_call_hook(
|
||||
self,
|
||||
kwargs,
|
||||
response_obj: MCPPostCallResponseObject,
|
||||
start_time,
|
||||
end_time
|
||||
) -> Optional[MCPPostCallResponseObject]:
|
||||
"""
|
||||
Called after each MCP tool call.
|
||||
Modify costs and response before returning to user.
|
||||
"""
|
||||
|
||||
# Extract tool information from kwargs
|
||||
tool_name = kwargs.get("name", "")
|
||||
server_name = kwargs.get("server_name", "")
|
||||
|
||||
# Calculate custom cost based on your logic
|
||||
custom_cost = 42.00
|
||||
|
||||
# Set the response cost
|
||||
response_obj.hidden_params.response_cost = custom_cost
|
||||
|
||||
|
||||
|
||||
return response_obj
|
||||
|
||||
|
||||
# Create instance for LiteLLM to use
|
||||
custom_mcp_cost_tracker = CustomMCPCostTracker()
|
||||
```
|
||||
|
||||
#### 2. Configure in config.yaml
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-xxxxxxx
|
||||
|
||||
# Add your custom MCP hook
|
||||
callbacks:
|
||||
- custom_mcp_hook.custom_mcp_cost_tracker
|
||||
|
||||
mcp_servers:
|
||||
zapier_server:
|
||||
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
|
||||
```
|
||||
|
||||
#### 3. Start the proxy
|
||||
|
||||
```shell
|
||||
$ litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
When MCP tools are called, your custom hook will:
|
||||
1. Calculate costs based on your custom logic
|
||||
2. Modify the response if needed
|
||||
3. Track costs in LiteLLM's logging system
|
||||
|
||||
## MCP Guardrails
|
||||
|
||||
LiteLLM supports applying guardrails to MCP tool calls to ensure security and compliance. You can configure guardrails to run before or during MCP calls to validate inputs and block or mask sensitive information.
|
||||
|
||||
### Supported MCP Guardrail Modes
|
||||
|
||||
MCP guardrails support the following modes:
|
||||
|
||||
- `pre_mcp_call`: Run **before** MCP call, on **input**. Use this mode when you want to apply validation/masking/blocking for MCP requests
|
||||
- `during_mcp_call`: Run **during** MCP call execution. Use this mode for real-time monitoring and intervention
|
||||
|
||||
### Configuration Examples
|
||||
|
||||
Configure guardrails to run before MCP tool calls to validate and sanitize inputs:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
guardrails:
|
||||
- guardrail_name: "mcp-input-validation"
|
||||
litellm_params:
|
||||
guardrail: presidio # or other supported guardrails
|
||||
mode: "pre_mcp_call" # or during_mcp_call
|
||||
pii_entities_config:
|
||||
CREDIT_CARD: "BLOCK" # Will block requests containing credit card numbers
|
||||
EMAIL_ADDRESS: "MASK" # Will mask email addresses
|
||||
PHONE_NUMBER: "MASK" # Will mask phone numbers
|
||||
default_on: true
|
||||
```
|
||||
|
||||
|
||||
### Usage Examples
|
||||
|
||||
#### Testing Pre-MCP Call Guardrails
|
||||
|
||||
Test your MCP guardrails with a request that includes sensitive information:
|
||||
|
||||
```bash title="Test MCP Guardrail" showLineNumbers
|
||||
curl http://localhost:4000/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "user", "content": "My credit card is 4111-1111-1111-1111 and my email is john@example.com"}
|
||||
],
|
||||
"guardrails": ["mcp-input-validation"]
|
||||
}'
|
||||
```
|
||||
|
||||
The request will be processed as follows:
|
||||
1. Credit card number will be blocked (request rejected)
|
||||
2. Email address will be masked (e.g., replaced with `<EMAIL_ADDRESS>`)
|
||||
|
||||
#### Using with MCP Tools
|
||||
|
||||
When using MCP tools, guardrails will be applied to the tool inputs:
|
||||
|
||||
```python title="Python Example with MCP Guardrails" showLineNumbers
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="your-api-key",
|
||||
base_url="http://localhost:4000"
|
||||
)
|
||||
|
||||
# This request will trigger MCP guardrails
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[
|
||||
{"role": "user", "content": "Send an email to 555-123-4567 with my SSN 123-45-6789"}
|
||||
],
|
||||
tools=[{"type": "mcp", "server_label": "litellm", "server_url": "litellm_proxy"}],
|
||||
guardrails=["mcp-input-validation"]
|
||||
)
|
||||
```
|
||||
|
||||
### Supported Guardrail Providers
|
||||
|
||||
MCP guardrails work with all LiteLLM-supported guardrail providers:
|
||||
|
||||
- **Presidio**: PII detection and masking
|
||||
- **Bedrock**: AWS Bedrock guardrails
|
||||
- **Lakera**: Content moderation
|
||||
- **Aporia**: Custom guardrails
|
||||
- **Custom**: Your own guardrail implementations
|
||||
|
||||
## MCP Permission Management
|
||||
|
||||
LiteLLM supports managing permissions for MCP Servers by Keys, Teams, Organizations (entities) on LiteLLM. When a MCP client attempts to list tools, LiteLLM will only return the tools the entity has permissions to access.
|
||||
|
||||
When Creating a Key, Team, or Organization, you can select the allowed MCP Servers that the entity has access to.
|
||||
|
||||
<Image
|
||||
img={require('../img/mcp_key.png')}
|
||||
style={{width: '80%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
|
||||
## LiteLLM Proxy - Walk through MCP Gateway
|
||||
LiteLLM exposes an MCP Gateway for admins to add all their MCP servers to LiteLLM. The key benefits of using LiteLLM Proxy with MCP are:
|
||||
|
||||
|
|
|
|||
45
docs/my-website/docs/mcp_control.md
Normal file
45
docs/my-website/docs/mcp_control.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# MCP Permission Management
|
||||
|
||||
Control which MCP servers and tools can be accessed by specific keys, teams, or organizations in LiteLLM. When a client attempts to list or call tools, LiteLLM enforces access controls based on configured permissions.
|
||||
|
||||
## Overview
|
||||
|
||||
LiteLLM provides fine-grained permission management for MCP servers, allowing you to:
|
||||
|
||||
- **Restrict MCP access by entity**: Control which keys, teams, or organizations can access specific MCP servers
|
||||
- **Tool-level filtering**: Automatically filter available tools based on entity permissions
|
||||
- **Centralized control**: Manage all MCP permissions from the LiteLLM Admin UI or API
|
||||
|
||||
This ensures that only authorized entities can discover and use MCP tools, providing an additional security layer for your MCP infrastructure.
|
||||
|
||||
:::info Related Documentation
|
||||
- [MCP Overview](./mcp.md) - Learn about MCP in LiteLLM
|
||||
- [MCP Cost Tracking](./mcp_cost.md) - Track costs for MCP tool calls
|
||||
- [MCP Guardrails](./mcp_guardrail.md) - Apply security guardrails to MCP calls
|
||||
- [Using MCP](./mcp_usage.md) - How to use MCP with LiteLLM
|
||||
:::
|
||||
|
||||
## How It Works
|
||||
|
||||
LiteLLM supports managing permissions for MCP Servers by Keys, Teams, Organizations (entities) on LiteLLM. When a MCP client attempts to list tools, LiteLLM will only return the tools the entity has permissions to access.
|
||||
|
||||
When Creating a Key, Team, or Organization, you can select the allowed MCP Servers that the entity has access to.
|
||||
|
||||
<Image
|
||||
img={require('../img/mcp_key.png')}
|
||||
style={{width: '80%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
|
||||
## Set Allowed Tools for a Key, Team, or Organization
|
||||
|
||||
Control which tools different teams can access from the same MCP server. For example, give your Engineering team access to `list_repositories`, `create_issue`, and `search_code`, while Sales only gets `search_code` and `close_issue`.
|
||||
|
||||
|
||||
This video shows how to set allowed tools for a Key, Team, or Organization.
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/7464d444c3324078892367272fe50745" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
121
docs/my-website/docs/mcp_cost.md
Normal file
121
docs/my-website/docs/mcp_cost.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# MCP Cost Tracking
|
||||
|
||||
LiteLLM provides two ways to track costs for MCP tool calls:
|
||||
|
||||
| Method | When to Use | What It Does |
|
||||
|--------|-------------|--------------|
|
||||
| **Config-based Cost Tracking** | Simple cost tracking with fixed costs per tool/server | Automatically tracks costs based on configuration |
|
||||
| **Custom Post-MCP Hook** | Dynamic cost tracking with custom logic | Allows custom cost calculations and response modifications |
|
||||
|
||||
### Config-based Cost Tracking
|
||||
|
||||
Configure fixed costs for MCP servers directly in your config.yaml:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-xxxxxxx
|
||||
|
||||
mcp_servers:
|
||||
zapier_server:
|
||||
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
|
||||
mcp_info:
|
||||
mcp_server_cost_info:
|
||||
# Default cost for all tools in this server
|
||||
default_cost_per_query: 0.01
|
||||
# Custom cost for specific tools
|
||||
tool_name_to_cost_per_query:
|
||||
send_email: 0.05
|
||||
create_document: 0.03
|
||||
|
||||
expensive_api_server:
|
||||
url: "https://api.expensive-service.com/mcp"
|
||||
mcp_info:
|
||||
mcp_server_cost_info:
|
||||
default_cost_per_query: 1.50
|
||||
```
|
||||
|
||||
### Custom Post-MCP Hook
|
||||
|
||||
Use this when you need dynamic cost calculation or want to modify the MCP response before it's returned to the user.
|
||||
|
||||
#### 1. Create a custom MCP hook file
|
||||
|
||||
```python title="custom_mcp_hook.py" showLineNumbers
|
||||
from typing import Optional
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.types.mcp import MCPPostCallResponseObject
|
||||
|
||||
|
||||
class CustomMCPCostTracker(CustomLogger):
|
||||
"""
|
||||
Custom handler for MCP cost tracking and response modification
|
||||
"""
|
||||
|
||||
async def async_post_mcp_tool_call_hook(
|
||||
self,
|
||||
kwargs,
|
||||
response_obj: MCPPostCallResponseObject,
|
||||
start_time,
|
||||
end_time
|
||||
) -> Optional[MCPPostCallResponseObject]:
|
||||
"""
|
||||
Called after each MCP tool call.
|
||||
Modify costs and response before returning to user.
|
||||
"""
|
||||
|
||||
# Extract tool information from kwargs
|
||||
tool_name = kwargs.get("name", "")
|
||||
server_name = kwargs.get("server_name", "")
|
||||
|
||||
# Calculate custom cost based on your logic
|
||||
custom_cost = 42.00
|
||||
|
||||
# Set the response cost
|
||||
response_obj.hidden_params.response_cost = custom_cost
|
||||
|
||||
|
||||
|
||||
return response_obj
|
||||
|
||||
|
||||
# Create instance for LiteLLM to use
|
||||
custom_mcp_cost_tracker = CustomMCPCostTracker()
|
||||
```
|
||||
|
||||
#### 2. Configure in config.yaml
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-xxxxxxx
|
||||
|
||||
# Add your custom MCP hook
|
||||
callbacks:
|
||||
- custom_mcp_hook.custom_mcp_cost_tracker
|
||||
|
||||
mcp_servers:
|
||||
zapier_server:
|
||||
url: "https://actions.zapier.com/mcp/sk-xxxxx/sse"
|
||||
```
|
||||
|
||||
#### 3. Start the proxy
|
||||
|
||||
```shell
|
||||
$ litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
When MCP tools are called, your custom hook will:
|
||||
1. Calculate costs based on your custom logic
|
||||
2. Modify the response if needed
|
||||
3. Track costs in LiteLLM's logging system
|
||||
|
||||
88
docs/my-website/docs/mcp_guardrail.md
Normal file
88
docs/my-website/docs/mcp_guardrail.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# MCP Guardrails
|
||||
|
||||
LiteLLM supports applying guardrails to MCP tool calls to ensure security and compliance. You can configure guardrails to run before or during MCP calls to validate inputs and block or mask sensitive information.
|
||||
|
||||
### Supported MCP Guardrail Modes
|
||||
|
||||
MCP guardrails support the following modes:
|
||||
|
||||
- `pre_mcp_call`: Run **before** MCP call, on **input**. Use this mode when you want to apply validation/masking/blocking for MCP requests
|
||||
- `during_mcp_call`: Run **during** MCP call execution. Use this mode for real-time monitoring and intervention
|
||||
|
||||
### Configuration Examples
|
||||
|
||||
Configure guardrails to run before MCP tool calls to validate and sanitize inputs:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
guardrails:
|
||||
- guardrail_name: "mcp-input-validation"
|
||||
litellm_params:
|
||||
guardrail: presidio # or other supported guardrails
|
||||
mode: "pre_mcp_call" # or during_mcp_call
|
||||
pii_entities_config:
|
||||
CREDIT_CARD: "BLOCK" # Will block requests containing credit card numbers
|
||||
EMAIL_ADDRESS: "MASK" # Will mask email addresses
|
||||
PHONE_NUMBER: "MASK" # Will mask phone numbers
|
||||
default_on: true
|
||||
```
|
||||
|
||||
|
||||
### Usage Examples
|
||||
|
||||
#### Testing Pre-MCP Call Guardrails
|
||||
|
||||
Test your MCP guardrails with a request that includes sensitive information:
|
||||
|
||||
```bash title="Test MCP Guardrail" showLineNumbers
|
||||
curl http://localhost:4000/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "user", "content": "My credit card is 4111-1111-1111-1111 and my email is john@example.com"}
|
||||
],
|
||||
"guardrails": ["mcp-input-validation"]
|
||||
}'
|
||||
```
|
||||
|
||||
The request will be processed as follows:
|
||||
1. Credit card number will be blocked (request rejected)
|
||||
2. Email address will be masked (e.g., replaced with `<EMAIL_ADDRESS>`)
|
||||
|
||||
#### Using with MCP Tools
|
||||
|
||||
When using MCP tools, guardrails will be applied to the tool inputs:
|
||||
|
||||
```python title="Python Example with MCP Guardrails" showLineNumbers
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="your-api-key",
|
||||
base_url="http://localhost:4000"
|
||||
)
|
||||
|
||||
# This request will trigger MCP guardrails
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[
|
||||
{"role": "user", "content": "Send an email to 555-123-4567 with my SSN 123-45-6789"}
|
||||
],
|
||||
tools=[{"type": "mcp", "server_label": "litellm", "server_url": "litellm_proxy"}],
|
||||
guardrails=["mcp-input-validation"]
|
||||
)
|
||||
```
|
||||
|
||||
### Supported Guardrail Providers
|
||||
|
||||
MCP guardrails work with all LiteLLM-supported guardrail providers:
|
||||
|
||||
- **Presidio**: PII detection and masking
|
||||
- **Bedrock**: AWS Bedrock guardrails
|
||||
- **Lakera**: Content moderation
|
||||
- **Aporia**: Custom guardrails
|
||||
- **Custom**: Your own guardrail implementations
|
||||
209
docs/my-website/docs/mcp_usage.md
Normal file
209
docs/my-website/docs/mcp_usage.md
Normal file
|
|
@ -0,0 +1,209 @@
|
|||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# Using your MCP
|
||||
|
||||
This document covers how to use LiteLLM as an MCP Gateway. You can see how to use it with Responses API, Cursor IDE, and OpenAI SDK.
|
||||
|
||||
### Use on LiteLLM UI
|
||||
|
||||
Follow this walkthrough to use your MCP on LiteLLM UI
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/57e0763267254bc79dbe6658d0b8758c" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
|
||||
### Use with Responses API
|
||||
|
||||
Replace `http://localhost:4000` with your LiteLLM Proxy base URL.
|
||||
|
||||
Demo Video Using Responses API with LiteLLM Proxy: [Demo video here](https://www.loom.com/share/34587e618c5c47c0b0d67b4e4d02718f?sid=2caf3d45-ead4-4490-bcc1-8d6dd6041c02)
|
||||
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="cURL">
|
||||
|
||||
```bash title="cURL Example" showLineNumbers
|
||||
curl --location 'http://localhost:4000/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer sk-1234" \
|
||||
--data '{
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never"
|
||||
}
|
||||
],
|
||||
"stream": true,
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python SDK">
|
||||
|
||||
```python title="Python SDK Example" showLineNumbers
|
||||
"""
|
||||
Use LiteLLM Proxy MCP Gateway to call MCP tools.
|
||||
|
||||
When using LiteLLM Proxy, you can use the same MCP tools across all your LLM providers.
|
||||
"""
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="sk-1234", # paste your litellm proxy api key here
|
||||
base_url="http://localhost:4000" # paste your litellm proxy base url here
|
||||
)
|
||||
print("Making API request to Responses API with MCP tools")
|
||||
|
||||
response = client.responses.create(
|
||||
model="gpt-5",
|
||||
input=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
tools=[
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never"
|
||||
}
|
||||
],
|
||||
stream=True,
|
||||
tool_choice="required"
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
print("response chunk: ", chunk)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
#### Specifying MCP Tools
|
||||
|
||||
You can specify which MCP tools are available by using the `allowed_tools` parameter. This allows you to restrict access to specific tools within an MCP server.
|
||||
|
||||
To get the list of allowed tools when using LiteLLM MCP Gateway, you can naigate to the LiteLLM UI on MCP Servers > MCP Tools > Click the Tool > Copy Tool Name.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="cURL">
|
||||
|
||||
```bash title="cURL Example with allowed_tools" showLineNumbers
|
||||
curl --location 'http://localhost:4000/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer sk-1234" \
|
||||
--data '{
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy/mcp",
|
||||
"require_approval": "never",
|
||||
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
|
||||
}
|
||||
],
|
||||
"stream": true,
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="python" label="Python SDK">
|
||||
|
||||
```python title="Python SDK Example with allowed_tools" showLineNumbers
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="sk-1234",
|
||||
base_url="http://localhost:4000"
|
||||
)
|
||||
|
||||
response = client.responses.create(
|
||||
model="gpt-5",
|
||||
input=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "give me TLDR of what BerriAI/litellm repo is about",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
tools=[
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy/mcp",
|
||||
"require_approval": "never",
|
||||
"allowed_tools": ["GitMCP-fetch_litellm_documentation"]
|
||||
}
|
||||
],
|
||||
stream=True,
|
||||
tool_choice="required"
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Use with Cursor IDE
|
||||
|
||||
Use tools directly from Cursor IDE with LiteLLM MCP:
|
||||
|
||||
**Setup Instructions:**
|
||||
|
||||
1. **Open Cursor Settings**: Use `⇧+⌘+J` (Mac) or `Ctrl+Shift+J` (Windows/Linux)
|
||||
2. **Navigate to MCP Tools**: Go to the "MCP Tools" tab and click "New MCP Server"
|
||||
3. **Add Configuration**: Copy and paste the JSON configuration below, then save with `Cmd+S` or `Ctrl+S`
|
||||
|
||||
```json title="Basic Cursor MCP Configuration" showLineNumbers
|
||||
{
|
||||
"mcpServers": {
|
||||
"LiteLLM": {
|
||||
"url": "litellm_proxy",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### How it works when server_url="litellm_proxy"
|
||||
|
||||
When server_url="litellm_proxy", LiteLLM bridges non-MCP providers to your MCP tools.
|
||||
|
||||
- Tool Discovery: LiteLLM fetches MCP tools and converts them to OpenAI-compatible definitions
|
||||
- LLM Call: Tools are sent to the LLM with your input; LLM selects which tools to call
|
||||
- Tool Execution: LiteLLM automatically parses arguments, routes calls to MCP servers, executes tools, and retrieves results
|
||||
- Response Integration: Tool results are sent back to LLM for final response generation
|
||||
- Output: Complete response combining LLM reasoning with tool execution results
|
||||
|
||||
This enables MCP tool usage with any LiteLLM-supported provider, regardless of native MCP support.
|
||||
|
||||
#### Auto-execution for require_approval: "never"
|
||||
|
||||
Setting require_approval: "never" triggers automatic tool execution, returning the final response in a single API call without additional user interaction.
|
||||
|
|
@ -265,6 +265,7 @@ print(response)
|
|||
| TwelveLabs Marengo Embed 2.7 | `embedding(model="bedrock/us.twelvelabs.marengo-embed-2-7-v1:0", input=input)` | Supports multimodal input (text, video, audio, image) |
|
||||
| Cohere Embeddings - English | `embedding(model="bedrock/cohere.embed-english-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18)
|
||||
| Cohere Embeddings - Multilingual | `embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18)
|
||||
| Cohere Embed v4 | `embedding(model="bedrock/cohere.embed-v4:0", input=input)` | Supports text and image input, configurable dimensions (256, 512, 1024, 1536), 128k context length |
|
||||
|
||||
### Advanced - [Drop Unsupported Params](https://docs.litellm.ai/docs/completion/drop_params#openai-proxy-usage)
|
||||
|
||||
|
|
|
|||
|
|
@ -37,6 +37,29 @@ for event in response:
|
|||
print(event)
|
||||
```
|
||||
|
||||
#### Image Generation with Streaming
|
||||
```python showLineNumbers title="OpenAI Streaming Image Generation"
|
||||
import litellm
|
||||
import base64
|
||||
|
||||
# Streaming image generation with partial images
|
||||
stream = litellm.responses(
|
||||
model="gpt-4.1", # Use an actual image generation model
|
||||
input="Generate a gorgeous image of a river made of white owl feathers",
|
||||
stream=True,
|
||||
tools=[{"type": "image_generation", "partial_images": 2}],
|
||||
|
||||
)
|
||||
|
||||
for event in stream:
|
||||
if event.type == "response.image_generation_call.partial_image":
|
||||
idx = event.partial_image_index
|
||||
image_base64 = event.partial_image_b64
|
||||
image_bytes = base64.b64decode(image_base64)
|
||||
with open(f"river{idx}.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
#### GET a Response
|
||||
```python showLineNumbers title="Get Response by ID"
|
||||
import litellm
|
||||
|
|
@ -150,6 +173,33 @@ for event in response:
|
|||
print(event)
|
||||
```
|
||||
|
||||
#### Image Generation with Streaming
|
||||
```python showLineNumbers title="OpenAI Proxy Streaming Image Generation"
|
||||
from openai import OpenAI
|
||||
import base64
|
||||
|
||||
# Initialize client with your proxy URL
|
||||
client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000")
|
||||
|
||||
stream = client.responses.create(
|
||||
model="gpt-4.1",
|
||||
input="Draw a gorgeous image of a river made of white owl feathers, snaking its way through a serene winter landscape",
|
||||
stream=True,
|
||||
tools=[{"type": "image_generation", "partial_images": 2}],
|
||||
)
|
||||
|
||||
|
||||
for event in stream:
|
||||
print(f"event: {event}")
|
||||
if event.type == "response.image_generation_call.partial_image":
|
||||
idx = event.partial_image_index
|
||||
image_base64 = event.partial_image_b64
|
||||
image_bytes = base64.b64decode(image_base64)
|
||||
with open(f"river{idx}.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
|
||||
```
|
||||
|
||||
#### GET a Response
|
||||
```python showLineNumbers title="Get Response by ID with OpenAI SDK"
|
||||
from openai import OpenAI
|
||||
|
|
|
|||
|
|
@ -81,6 +81,23 @@ MICROSOFT_TENANT="5a39737
|
|||
http://localhost:4000/sso/callback
|
||||
```
|
||||
|
||||
**Using App Roles for User Permissions**
|
||||
|
||||
You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token and assign the corresponding role to the user.
|
||||
|
||||
Supported roles:
|
||||
- `proxy_admin` - Admin over the platform
|
||||
- `proxy_admin_viewer` - Can login, view all keys, view all spend (read-only)
|
||||
- `internal_user` - Normal user. Can login, view spend and depending on team-member permissions - view/create/delete their own keys.
|
||||
|
||||
|
||||
To set up app roles:
|
||||
1. Navigate to your App Registration on https://portal.azure.com/
|
||||
2. Go to "App roles" and create a new app role
|
||||
3. Use one of the supported role names above (e.g., `proxy_admin`)
|
||||
4. Assign users to these roles in your Enterprise Application
|
||||
5. When users sign in via SSO, LiteLLM will automatically assign them the corresponding role
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="Generic" label="Generic SSO Provider">
|
||||
|
|
|
|||
|
|
@ -278,6 +278,8 @@ Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable cac
|
|||
REDIS_HOST = "" # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
|
||||
REDIS_PORT = "" # REDIS_PORT='18841'
|
||||
REDIS_PASSWORD = "" # REDIS_PASSWORD='liteLlmIsAmazing'
|
||||
REDIS_USERNAME = "" # REDIS_USERNAME='my-redis-username' [OPTIONAL] if your redis server requires a username
|
||||
REDIS_SSL = "True" # REDIS_SSL='True' to enable SSL by default is False
|
||||
```
|
||||
|
||||
**Additional kwargs**
|
||||
|
|
|
|||
|
|
@ -141,6 +141,7 @@ litellm_settings:
|
|||
"dev": 0.1 # 10% reserved for development (1 RPM)
|
||||
priority_reservation_settings:
|
||||
default_priority: 0 # Weight (0%) assigned to keys without explicit priority metadata
|
||||
saturation_threshold: 0.50 # A model is saturated if it has hit 50% of its RPM limit
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234 # OR set `LITELLM_MASTER_KEY=".."` in your .env
|
||||
|
|
@ -156,6 +157,8 @@ general_settings:
|
|||
|
||||
`priority_reservation_settings`: Object (Optional)
|
||||
- **default_priority (float)**: Weight/percentage (0.0 to 1.0) assigned to API keys that have no priority metadata set (defaults to 0.5)
|
||||
- **saturation_threshold (float)**: Saturation level (0.0 to 1.0) at which strict priority enforcement begins for a model. Saturation is calculated as `max(current_rpm/max_rpm, current_tpm/max_tpm)`. Below this threshold, generous mode allows priority borrowing from unused capacity. Above this threshold, strict mode enforces normalized priority limits.
|
||||
- Example: When model usage is low, keys can use more than their allocated share. When model usage is high, keys are strictly limited to their allocated share.
|
||||
|
||||
**Start Proxy**
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ Requests to /chat/completions may be bridged here automatically when the provide
|
|||
| Logging | ✅ | Works across all integrations |
|
||||
| End-user Tracking | ✅ | |
|
||||
| Streaming | ✅ | |
|
||||
| Image Generation Streaming | ✅ | Progressive image generation with partial images (1-3) |
|
||||
| Fallbacks | ✅ | Works between supported models |
|
||||
| Loadbalancing | ✅ | Works between supported models |
|
||||
| Supported operations | Create a response, Get a response, Delete a response | |
|
||||
|
|
@ -56,6 +57,29 @@ for event in response:
|
|||
print(event)
|
||||
```
|
||||
|
||||
#### Image Generation with Streaming
|
||||
```python showLineNumbers title="OpenAI Streaming Image Generation"
|
||||
import litellm
|
||||
import base64
|
||||
|
||||
# Streaming image generation with partial images
|
||||
stream = litellm.responses(
|
||||
model="gpt-4.1", # Use an actual image generation model
|
||||
input="Generate a gorgeous image of a river made of white owl feathers",
|
||||
stream=True,
|
||||
tools=[{"type": "image_generation", "partial_images": 2}],
|
||||
|
||||
)
|
||||
|
||||
for event in stream:
|
||||
if event.type == "response.image_generation_call.partial_image":
|
||||
idx = event.partial_image_index
|
||||
image_base64 = event.partial_image_b64
|
||||
image_bytes = base64.b64decode(image_base64)
|
||||
with open(f"river{idx}.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
#### GET a Response
|
||||
```python showLineNumbers title="Get Response by ID"
|
||||
import litellm
|
||||
|
|
@ -380,6 +404,32 @@ for event in response:
|
|||
print(event)
|
||||
```
|
||||
|
||||
#### Image Generation with Streaming
|
||||
```python showLineNumbers title="OpenAI Proxy Streaming Image Generation"
|
||||
from openai import OpenAI
|
||||
import base64
|
||||
|
||||
client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000")
|
||||
|
||||
stream = client.responses.create(
|
||||
model="gpt-4.1",
|
||||
input="Draw a gorgeous image of a river made of white owl feathers, snaking its way through a serene winter landscape",
|
||||
stream=True,
|
||||
tools=[{"type": "image_generation", "partial_images": 2}],
|
||||
)
|
||||
|
||||
|
||||
for event in stream:
|
||||
print(f"event: {event}")
|
||||
if event.type == "response.image_generation_call.partial_image":
|
||||
idx = event.partial_image_index
|
||||
image_base64 = event.partial_image_b64
|
||||
image_bytes = base64.b64decode(image_base64)
|
||||
with open(f"river{idx}.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
|
||||
```
|
||||
|
||||
#### GET a Response
|
||||
```python showLineNumbers title="Get Response by ID with OpenAI SDK"
|
||||
from openai import OpenAI
|
||||
|
|
|
|||
|
|
@ -140,6 +140,54 @@ litellm_settings:
|
|||
<Image img={require('../../img/msft_default_settings.png')} style={{ width: '900px', height: 'auto' }} />
|
||||
|
||||
|
||||
## 4. Using Entra ID App Roles for User Permissions
|
||||
|
||||
You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token during SSO sign-in and assign the corresponding role to the user.
|
||||
|
||||
### 4.1 Supported Roles
|
||||
|
||||
LiteLLM supports the following app roles (case-insensitive):
|
||||
|
||||
- `proxy_admin` - Admin over the entire LiteLLM platform
|
||||
- `proxy_admin_viewer` - Read-only admin access (can view all keys and spend)
|
||||
- `org_admin` - Admin over a specific organization (can create teams and users within their org)
|
||||
- `internal_user` - Standard user (can create/view/delete their own keys and view their own spend)
|
||||
|
||||
### 4.2 Create App Roles in Entra ID
|
||||
|
||||
1. Navigate to your App Registration on https://portal.azure.com/
|
||||
2. Go to **App roles** > **Create app role**
|
||||
|
||||
3. Configure the app role:
|
||||
- **Display name**: Proxy Admin (or your preferred display name)
|
||||
- **Value**: `proxy_admin` (use one of the supported role values above)
|
||||
- **Description**: Administrator access to LiteLLM proxy
|
||||
- **Allowed member types**: Users/Groups
|
||||
|
||||
|
||||
4. Click **Apply** to save the role
|
||||
|
||||
### 4.3 Assign Users to App Roles
|
||||
|
||||
1. Navigate to **Enterprise Applications** on https://portal.azure.com/
|
||||
2. Select your LiteLLM application
|
||||
3. Go to **Users and groups** > **Add user/group**
|
||||
4. Select the user and assign them to one of the app roles you created
|
||||
|
||||
|
||||
### 4.4 Test the Role Assignment
|
||||
|
||||
1. Sign in to LiteLLM UI via SSO as a user with an assigned app role
|
||||
2. LiteLLM will automatically extract the app role from the JWT token
|
||||
3. The user will be assigned the corresponding LiteLLM role in the database
|
||||
4. The user's permissions will reflect their assigned role
|
||||
|
||||
**How it works:**
|
||||
- When a user signs in via Microsoft SSO, LiteLLM extracts the `roles` claim from the JWT `id_token`
|
||||
- If any of the roles match a valid LiteLLM role (case-insensitive), that role is assigned to the user
|
||||
- If multiple roles are present, LiteLLM uses the first valid role it finds
|
||||
- This role assignment persists in the LiteLLM database and determines the user's access level
|
||||
|
||||
## Video Walkthrough
|
||||
|
||||
This walks through setting up sso auto-add for **Microsoft Entra ID**
|
||||
|
|
|
|||
|
|
@ -333,7 +333,17 @@ const sidebars = {
|
|||
"image_variations",
|
||||
]
|
||||
},
|
||||
"mcp",
|
||||
{
|
||||
type: "category",
|
||||
label: "/mcp - Model Context Protocol",
|
||||
items: [
|
||||
"mcp",
|
||||
"mcp_usage",
|
||||
"mcp_control",
|
||||
"mcp_cost",
|
||||
"mcp_guardrail",
|
||||
]
|
||||
},
|
||||
"moderation",
|
||||
{
|
||||
type: "category",
|
||||
|
|
@ -529,16 +539,10 @@ const sidebars = {
|
|||
type: "category",
|
||||
label: "Guides",
|
||||
items: [
|
||||
{
|
||||
type: "category",
|
||||
label: "Tools",
|
||||
items: [
|
||||
"completion/computer_use",
|
||||
"completion/web_search",
|
||||
"completion/web_fetch",
|
||||
"completion/function_call",
|
||||
]
|
||||
},
|
||||
"completion/computer_use",
|
||||
"completion/web_search",
|
||||
"completion/web_fetch",
|
||||
"completion/function_call",
|
||||
"completion/audio",
|
||||
"completion/document_understanding",
|
||||
"completion/drop_params",
|
||||
|
|
|
|||
|
|
@ -1190,6 +1190,9 @@ from .llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
|
|||
from .llms.azure.responses.o_series_transformation import (
|
||||
AzureOpenAIOSeriesResponsesAPIConfig,
|
||||
)
|
||||
from .llms.litellm_proxy.responses.transformation import (
|
||||
LiteLLMProxyResponsesAPIConfig,
|
||||
)
|
||||
from .llms.openai.chat.o_series_transformation import (
|
||||
OpenAIOSeriesConfig as OpenAIO1Config, # maintain backwards compatibility
|
||||
OpenAIOSeriesConfig,
|
||||
|
|
|
|||
|
|
@ -177,14 +177,21 @@ def get_redis_url_from_environment():
|
|||
raise ValueError(
|
||||
"Either 'REDIS_URL' or both 'REDIS_HOST' and 'REDIS_PORT' must be specified for Redis."
|
||||
)
|
||||
|
||||
if "REDIS_PASSWORD" in os.environ:
|
||||
redis_password = f":{os.environ['REDIS_PASSWORD']}@"
|
||||
|
||||
if "REDIS_SSL" in os.environ and os.environ["REDIS_SSL"].lower() == "true":
|
||||
redis_protocol = "rediss"
|
||||
else:
|
||||
redis_password = ""
|
||||
|
||||
redis_protocol = "redis"
|
||||
|
||||
# Build authentication part of URL
|
||||
auth_part = ""
|
||||
if "REDIS_USERNAME" in os.environ and "REDIS_PASSWORD" in os.environ:
|
||||
auth_part = f"{os.environ['REDIS_USERNAME']}:{os.environ['REDIS_PASSWORD']}@"
|
||||
elif "REDIS_PASSWORD" in os.environ:
|
||||
auth_part = f"{os.environ['REDIS_PASSWORD']}@"
|
||||
|
||||
return (
|
||||
f"redis://{redis_password}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
|
||||
f"{redis_protocol}://{auth_part}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -14,10 +14,10 @@ It utilizes the (RedisCache, s3Cache, RedisSemanticCache, QdrantSemanticCache, I
|
|||
In each method it will call the appropriate method from caching.py
|
||||
"""
|
||||
|
||||
import time
|
||||
import asyncio
|
||||
import datetime
|
||||
import inspect
|
||||
import time
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
Any,
|
||||
|
|
@ -62,12 +62,10 @@ else:
|
|||
LiteLLMLoggingObj = Any
|
||||
|
||||
|
||||
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
|
||||
|
||||
|
||||
from litellm.litellm_core_utils.core_helpers import (
|
||||
_get_parent_otel_span_from_kwargs,
|
||||
_get_parent_otel_span_from_kwargs,
|
||||
)
|
||||
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
|
||||
|
||||
|
||||
class CachingHandlerResponse(BaseModel):
|
||||
|
|
@ -214,9 +212,7 @@ class LLMCachingHandler:
|
|||
end_time=end_time,
|
||||
cache_hit=cache_hit,
|
||||
)
|
||||
cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
|
||||
**kwargs
|
||||
)
|
||||
cache_key = litellm.cache.get_cache_key(**kwargs)
|
||||
if (
|
||||
isinstance(cached_result, BaseModel)
|
||||
or isinstance(cached_result, CustomStreamWrapper)
|
||||
|
|
@ -330,9 +326,7 @@ class LLMCachingHandler:
|
|||
end_time=end_time,
|
||||
cache_hit=cache_hit
|
||||
)
|
||||
cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
|
||||
**kwargs
|
||||
)
|
||||
cache_key = litellm.cache.get_cache_key(**kwargs)
|
||||
if (
|
||||
isinstance(cached_result, BaseModel)
|
||||
or isinstance(cached_result, CustomStreamWrapper)
|
||||
|
|
|
|||
|
|
@ -871,6 +871,7 @@ bedrock_embedding_models: set = set(
|
|||
"amazon.titan-embed-text-v1",
|
||||
"cohere.embed-english-v3",
|
||||
"cohere.embed-multilingual-v3",
|
||||
"cohere.embed-v4:0",
|
||||
"twelvelabs.marengo-embed-2-7-v1:0",
|
||||
]
|
||||
)
|
||||
|
|
|
|||
|
|
@ -268,6 +268,7 @@ def create_file(
|
|||
raise e
|
||||
|
||||
|
||||
@client
|
||||
async def afile_retrieve(
|
||||
file_id: str,
|
||||
custom_llm_provider: Literal["openai", "azure"] = "openai",
|
||||
|
|
@ -308,6 +309,7 @@ async def afile_retrieve(
|
|||
raise e
|
||||
|
||||
|
||||
@client
|
||||
def file_retrieve(
|
||||
file_id: str,
|
||||
custom_llm_provider: Literal["openai", "azure"] = "openai",
|
||||
|
|
@ -422,6 +424,7 @@ def file_retrieve(
|
|||
|
||||
|
||||
# Delete file
|
||||
@client
|
||||
async def afile_delete(
|
||||
file_id: str,
|
||||
custom_llm_provider: Literal["openai", "azure"] = "openai",
|
||||
|
|
@ -462,6 +465,7 @@ async def afile_delete(
|
|||
raise e
|
||||
|
||||
|
||||
@client
|
||||
def file_delete(
|
||||
file_id: str,
|
||||
custom_llm_provider: Literal["openai", "azure"] = "openai",
|
||||
|
|
@ -577,6 +581,7 @@ def file_delete(
|
|||
|
||||
|
||||
# List files
|
||||
@client
|
||||
async def afile_list(
|
||||
custom_llm_provider: Literal["openai", "azure"] = "openai",
|
||||
purpose: Optional[str] = None,
|
||||
|
|
@ -617,6 +622,7 @@ async def afile_list(
|
|||
raise e
|
||||
|
||||
|
||||
@client
|
||||
def file_list(
|
||||
custom_llm_provider: Literal["openai", "azure"] = "openai",
|
||||
purpose: Optional[str] = None,
|
||||
|
|
@ -729,6 +735,7 @@ def file_list(
|
|||
raise e
|
||||
|
||||
|
||||
@client
|
||||
async def afile_content(
|
||||
file_id: str,
|
||||
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
|
||||
|
|
@ -771,6 +778,7 @@ async def afile_content(
|
|||
raise e
|
||||
|
||||
|
||||
@client
|
||||
def file_content(
|
||||
file_id: str,
|
||||
model: Optional[str] = None,
|
||||
|
|
|
|||
|
|
@ -81,12 +81,12 @@ from litellm.types.llms.openai import (
|
|||
)
|
||||
from litellm.types.mcp import MCPPostCallResponseObject
|
||||
from litellm.types.rerank import RerankResponse
|
||||
from litellm.types.router import CustomPricingLiteLLMParams
|
||||
from litellm.types.utils import (
|
||||
CachingDetails,
|
||||
CallTypes,
|
||||
CostBreakdown,
|
||||
CostResponseTypes,
|
||||
CustomPricingLiteLLMParams,
|
||||
DynamicPromptManagementParamLiteral,
|
||||
EmbeddingResponse,
|
||||
GuardrailStatus,
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ import copy
|
|||
import json
|
||||
import mimetypes
|
||||
import re
|
||||
from litellm._uuid import uuid
|
||||
import xml.etree.ElementTree as ET
|
||||
from enum import Enum
|
||||
from typing import Any, List, Optional, Tuple, cast, overload
|
||||
|
|
@ -13,6 +12,7 @@ import litellm
|
|||
import litellm.types
|
||||
import litellm.types.llms
|
||||
from litellm import verbose_logger
|
||||
from litellm._uuid import uuid
|
||||
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_async_httpx_client
|
||||
from litellm.types.files import get_file_extension_from_mime_type
|
||||
from litellm.types.llms.anthropic import *
|
||||
|
|
@ -232,7 +232,6 @@ def ollama_pt(
|
|||
## MERGE CONSECUTIVE ASSISTANT CONTENT ##
|
||||
while msg_i < len(messages) and messages[msg_i]["role"] == "assistant":
|
||||
assistant_content_str += convert_content_list_to_str(messages[msg_i])
|
||||
msg_i += 1
|
||||
|
||||
tool_calls = messages[msg_i].get("tool_calls")
|
||||
ollama_tool_calls = []
|
||||
|
|
@ -258,7 +257,7 @@ def ollama_pt(
|
|||
f"Tool Calls: {json.dumps(ollama_tool_calls, indent=2)}"
|
||||
)
|
||||
|
||||
msg_i += 1
|
||||
msg_i += 1
|
||||
|
||||
if assistant_content_str:
|
||||
prompt += f"### Assistant:\n{assistant_content_str}\n\n"
|
||||
|
|
@ -365,62 +364,20 @@ def phind_codellama_pt(messages):
|
|||
return prompt
|
||||
|
||||
|
||||
def hf_chat_template( # noqa: PLR0915
|
||||
model: str, messages: list, chat_template: Optional[Any] = None
|
||||
):
|
||||
# Define Jinja2 environment
|
||||
env = ImmutableSandboxedEnvironment()
|
||||
|
||||
def raise_exception(message):
|
||||
raise Exception(f"Error message - {message}")
|
||||
|
||||
# Create a template object from the template text
|
||||
env.globals["raise_exception"] = raise_exception
|
||||
|
||||
## get the tokenizer config from huggingface
|
||||
bos_token = ""
|
||||
eos_token = ""
|
||||
if chat_template is None:
|
||||
|
||||
def _get_tokenizer_config(hf_model_name):
|
||||
try:
|
||||
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
|
||||
# Make a GET request to fetch the JSON data
|
||||
client = HTTPHandler(concurrent_limit=1)
|
||||
|
||||
response = client.get(url)
|
||||
except Exception as e:
|
||||
raise e
|
||||
if response.status_code == 200:
|
||||
# Parse the JSON data
|
||||
tokenizer_config = json.loads(response.content)
|
||||
return {"status": "success", "tokenizer": tokenizer_config}
|
||||
else:
|
||||
return {"status": "failure"}
|
||||
|
||||
if model in litellm.known_tokenizer_config:
|
||||
tokenizer_config = litellm.known_tokenizer_config[model]
|
||||
else:
|
||||
tokenizer_config = _get_tokenizer_config(model)
|
||||
litellm.known_tokenizer_config.update({model: tokenizer_config})
|
||||
|
||||
if (
|
||||
tokenizer_config["status"] == "failure"
|
||||
or "chat_template" not in tokenizer_config["tokenizer"]
|
||||
):
|
||||
raise Exception("No chat template found")
|
||||
## read the bos token, eos token and chat template from the json
|
||||
tokenizer_config = tokenizer_config["tokenizer"] # type: ignore
|
||||
|
||||
bos_token = tokenizer_config["bos_token"] # type: ignore
|
||||
if bos_token is not None and not isinstance(bos_token, str):
|
||||
if isinstance(bos_token, dict):
|
||||
bos_token = bos_token.get("content", None)
|
||||
eos_token = tokenizer_config["eos_token"] # type: ignore
|
||||
if eos_token is not None and not isinstance(eos_token, str):
|
||||
if isinstance(eos_token, dict):
|
||||
eos_token = eos_token.get("content", None)
|
||||
chat_template = tokenizer_config["chat_template"] # type: ignore
|
||||
def _render_chat_template(env, chat_template: str, bos_token: str, eos_token: str, messages: list) -> str:
|
||||
"""
|
||||
Shared template rendering logic for both sync and async hf_chat_template
|
||||
|
||||
Args:
|
||||
env: Jinja2 environment
|
||||
chat_template: Chat template string
|
||||
bos_token: Beginning of sequence token
|
||||
eos_token: End of sequence token
|
||||
messages: Messages to render
|
||||
|
||||
Returns:
|
||||
Rendered template string
|
||||
"""
|
||||
try:
|
||||
template = env.from_string(chat_template) # type: ignore
|
||||
except Exception as e:
|
||||
|
|
@ -435,7 +392,6 @@ def hf_chat_template( # noqa: PLR0915
|
|||
bos_token="<bos>",
|
||||
)
|
||||
return True
|
||||
|
||||
# This will be raised if Jinja attempts to render the system message and it can't
|
||||
except Exception:
|
||||
return False
|
||||
|
|
@ -469,7 +425,7 @@ def hf_chat_template( # noqa: PLR0915
|
|||
)
|
||||
except Exception as e:
|
||||
if "Conversation roles must alternate user/assistant" in str(e):
|
||||
# reformat messages to ensure user/assistant are alternating, if there's either 2 consecutive 'user' messages or 2 consecutive 'assistant' message, add a blank 'user' or 'assistant' message to ensure compatibility
|
||||
# reformat messages to ensure user/assistant are alternating
|
||||
new_messages = []
|
||||
for i in range(len(reformatted_messages) - 1):
|
||||
new_messages.append(reformatted_messages[i])
|
||||
|
|
@ -495,6 +451,188 @@ def hf_chat_template( # noqa: PLR0915
|
|||
) # don't use verbose_logger.exception, if exception is raised
|
||||
|
||||
|
||||
async def _afetch_and_extract_template(
|
||||
model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
|
||||
) -> Tuple[str, str, str]:
|
||||
"""
|
||||
Async version: Fetch template and tokens from HuggingFace.
|
||||
|
||||
Returns: (chat_template, bos_token, eos_token)
|
||||
"""
|
||||
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
|
||||
_extract_token_value,
|
||||
)
|
||||
|
||||
bos_token = ""
|
||||
eos_token = ""
|
||||
|
||||
if chat_template is None:
|
||||
# Fetch or retrieve cached tokenizer config
|
||||
if model in litellm.known_tokenizer_config:
|
||||
tokenizer_config = litellm.known_tokenizer_config[model]
|
||||
else:
|
||||
tokenizer_config = await get_config_fn(hf_model_name=model)
|
||||
litellm.known_tokenizer_config.update({model: tokenizer_config})
|
||||
|
||||
# Try to get chat template from tokenizer_config.json first
|
||||
if (
|
||||
tokenizer_config.get("status") == "success"
|
||||
and "tokenizer" in tokenizer_config
|
||||
and isinstance(tokenizer_config["tokenizer"], dict)
|
||||
and "chat_template" in tokenizer_config["tokenizer"]
|
||||
):
|
||||
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
|
||||
bos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("bos_token")
|
||||
)
|
||||
eos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("eos_token")
|
||||
)
|
||||
chat_template = tokenizer_data["chat_template"]
|
||||
else:
|
||||
# Fallback: Try to fetch chat template from separate .jinja file
|
||||
template_result = await get_template_fn(hf_model_name=model)
|
||||
if template_result.get("status") == "success":
|
||||
chat_template = template_result["chat_template"]
|
||||
# Still try to get tokens from tokenizer_config if available
|
||||
if (
|
||||
tokenizer_config.get("status") == "success"
|
||||
and "tokenizer" in tokenizer_config
|
||||
and isinstance(tokenizer_config["tokenizer"], dict)
|
||||
):
|
||||
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
|
||||
bos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("bos_token")
|
||||
)
|
||||
eos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("eos_token")
|
||||
)
|
||||
else:
|
||||
raise Exception("No chat template found")
|
||||
|
||||
return chat_template, bos_token, eos_token # type: ignore
|
||||
|
||||
|
||||
def _fetch_and_extract_template(
|
||||
model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
|
||||
) -> Tuple[str, str, str]:
|
||||
"""
|
||||
Sync version: Fetch template and tokens from HuggingFace.
|
||||
|
||||
Returns: (chat_template, bos_token, eos_token)
|
||||
"""
|
||||
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
|
||||
_extract_token_value,
|
||||
)
|
||||
|
||||
bos_token = ""
|
||||
eos_token = ""
|
||||
|
||||
if chat_template is None:
|
||||
# Fetch or retrieve cached tokenizer config
|
||||
if model in litellm.known_tokenizer_config:
|
||||
tokenizer_config = litellm.known_tokenizer_config[model]
|
||||
else:
|
||||
tokenizer_config = get_config_fn(hf_model_name=model)
|
||||
litellm.known_tokenizer_config.update({model: tokenizer_config})
|
||||
|
||||
# Try to get chat template from tokenizer_config.json first
|
||||
if (
|
||||
tokenizer_config.get("status") == "success"
|
||||
and "tokenizer" in tokenizer_config
|
||||
and isinstance(tokenizer_config["tokenizer"], dict)
|
||||
and "chat_template" in tokenizer_config["tokenizer"]
|
||||
):
|
||||
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
|
||||
bos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("bos_token")
|
||||
)
|
||||
eos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("eos_token")
|
||||
)
|
||||
chat_template = tokenizer_data["chat_template"]
|
||||
else:
|
||||
# Fallback: Try to fetch chat template from separate .jinja file
|
||||
template_result = get_template_fn(hf_model_name=model)
|
||||
if template_result.get("status") == "success":
|
||||
chat_template = template_result["chat_template"]
|
||||
# Still try to get tokens from tokenizer_config if available
|
||||
if (
|
||||
tokenizer_config.get("status") == "success"
|
||||
and "tokenizer" in tokenizer_config
|
||||
and isinstance(tokenizer_config["tokenizer"], dict)
|
||||
):
|
||||
tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
|
||||
bos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("bos_token")
|
||||
)
|
||||
eos_token = _extract_token_value(
|
||||
token_value=tokenizer_data.get("eos_token")
|
||||
)
|
||||
else:
|
||||
raise Exception("No chat template found")
|
||||
|
||||
return chat_template, bos_token, eos_token # type: ignore
|
||||
|
||||
|
||||
async def ahf_chat_template(
|
||||
model: str, messages: list, chat_template: Optional[Any] = None
|
||||
):
|
||||
"""HuggingFace chat template (async version)"""
|
||||
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
|
||||
_aget_chat_template_file,
|
||||
_aget_tokenizer_config,
|
||||
strftime_now,
|
||||
)
|
||||
|
||||
env = ImmutableSandboxedEnvironment()
|
||||
env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
|
||||
env.globals["strftime_now"] = strftime_now
|
||||
|
||||
template, bos_token, eos_token = await _afetch_and_extract_template(
|
||||
model=model,
|
||||
chat_template=chat_template,
|
||||
get_config_fn=_aget_tokenizer_config,
|
||||
get_template_fn=_aget_chat_template_file,
|
||||
)
|
||||
return _render_chat_template(
|
||||
env=env,
|
||||
chat_template=template,
|
||||
bos_token=bos_token,
|
||||
eos_token=eos_token,
|
||||
messages=messages,
|
||||
)
|
||||
|
||||
|
||||
def hf_chat_template(
|
||||
model: str, messages: list, chat_template: Optional[Any] = None
|
||||
):
|
||||
"""HuggingFace chat template (sync version)"""
|
||||
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
|
||||
_get_chat_template_file,
|
||||
_get_tokenizer_config,
|
||||
strftime_now,
|
||||
)
|
||||
|
||||
env = ImmutableSandboxedEnvironment()
|
||||
env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
|
||||
env.globals["strftime_now"] = strftime_now
|
||||
|
||||
template, bos_token, eos_token = _fetch_and_extract_template(
|
||||
model=model,
|
||||
chat_template=chat_template,
|
||||
get_config_fn=_get_tokenizer_config,
|
||||
get_template_fn=_get_chat_template_file,
|
||||
)
|
||||
return _render_chat_template(
|
||||
env=env,
|
||||
chat_template=template,
|
||||
bos_token=bos_token,
|
||||
eos_token=eos_token,
|
||||
messages=messages,
|
||||
)
|
||||
|
||||
|
||||
def deepseek_r1_pt(messages):
|
||||
return hf_chat_template(
|
||||
model="deepseek-r1/deepseek-r1-7b-instruct", messages=messages
|
||||
|
|
@ -4032,33 +4170,9 @@ def prompt_factory(
|
|||
elif custom_llm_provider == "azure_text":
|
||||
return azure_text_pt(messages=messages)
|
||||
elif custom_llm_provider == "watsonx":
|
||||
if "granite" in model and "chat" in model:
|
||||
# granite-13b-chat-v1 and granite-13b-chat-v2 use a specific prompt template
|
||||
return ibm_granite_pt(messages=messages)
|
||||
elif "ibm-mistral" in model and "instruct" in model:
|
||||
# models like ibm-mistral/mixtral-8x7b-instruct-v01-q use the mistral instruct prompt template
|
||||
return mistral_instruct_pt(messages=messages)
|
||||
elif "meta-llama/llama-3" in model and "instruct" in model:
|
||||
# https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/
|
||||
return custom_prompt(
|
||||
role_dict={
|
||||
"system": {
|
||||
"pre_message": "<|start_header_id|>system<|end_header_id|>\n",
|
||||
"post_message": "<|eot_id|>",
|
||||
},
|
||||
"user": {
|
||||
"pre_message": "<|start_header_id|>user<|end_header_id|>\n",
|
||||
"post_message": "<|eot_id|>",
|
||||
},
|
||||
"assistant": {
|
||||
"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
|
||||
"post_message": "<|eot_id|>",
|
||||
},
|
||||
},
|
||||
messages=messages,
|
||||
initial_prompt_value="<|begin_of_text|>",
|
||||
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
|
||||
)
|
||||
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
|
||||
return IBMWatsonXChatConfig.apply_prompt_template(model=model, messages=messages)
|
||||
|
||||
try:
|
||||
if "meta-llama/llama-2" in model and "chat" in model:
|
||||
return llama_2_chat_pt(messages=messages)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,139 @@
|
|||
import json
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, Union
|
||||
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
_get_httpx_client,
|
||||
get_async_httpx_client,
|
||||
)
|
||||
from litellm.types.llms.custom_http import httpxSpecialProvider
|
||||
|
||||
|
||||
def strftime_now(fmt: str) -> str:
|
||||
"""
|
||||
Custom function for templates that need current date/time formatting (e.g., gpt-oss)
|
||||
|
||||
Args:
|
||||
fmt: Format string for datetime.now().strftime()
|
||||
|
||||
Returns:
|
||||
Formatted string
|
||||
"""
|
||||
return datetime.now().strftime(fmt)
|
||||
|
||||
|
||||
def _get_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch tokenizer_config.json from HuggingFace (sync)
|
||||
|
||||
Args:
|
||||
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
|
||||
|
||||
Returns:
|
||||
Dict with 'status' and optionally 'tokenizer' keys
|
||||
"""
|
||||
try:
|
||||
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
|
||||
client = _get_httpx_client()
|
||||
response = client.get(url=url)
|
||||
except Exception as e:
|
||||
raise e
|
||||
if response.status_code == 200:
|
||||
tokenizer_config = json.loads(response.content)
|
||||
return {"status": "success", "tokenizer": tokenizer_config}
|
||||
else:
|
||||
return {"status": "failure"}
|
||||
|
||||
|
||||
async def _aget_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch tokenizer_config.json from HuggingFace (async)
|
||||
|
||||
Args:
|
||||
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
|
||||
|
||||
Returns:
|
||||
Dict with 'status' and optionally 'tokenizer' keys
|
||||
"""
|
||||
try:
|
||||
url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
|
||||
client = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.PromptFactory,
|
||||
)
|
||||
response = await client.get(url=url)
|
||||
except Exception as e:
|
||||
raise e
|
||||
if response.status_code == 200:
|
||||
tokenizer_config = json.loads(response.content)
|
||||
return {"status": "success", "tokenizer": tokenizer_config}
|
||||
else:
|
||||
return {"status": "failure"}
|
||||
|
||||
|
||||
def _get_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch chat template from separate .jinja file (sync)
|
||||
|
||||
Args:
|
||||
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
|
||||
|
||||
Returns:
|
||||
Dict with 'status' and optionally 'chat_template' keys
|
||||
"""
|
||||
template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
|
||||
client = _get_httpx_client()
|
||||
|
||||
for filename in template_filenames:
|
||||
try:
|
||||
url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
|
||||
response = client.get(url=url)
|
||||
if response.status_code == 200:
|
||||
return {"status": "success", "chat_template": response.content.decode("utf-8")}
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return {"status": "failure"}
|
||||
|
||||
|
||||
async def _aget_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch chat template from separate .jinja file (async)
|
||||
|
||||
Args:
|
||||
hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
|
||||
|
||||
Returns:
|
||||
Dict with 'status' and optionally 'chat_template' keys
|
||||
"""
|
||||
template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
|
||||
client = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.PromptFactory,
|
||||
)
|
||||
|
||||
for filename in template_filenames:
|
||||
try:
|
||||
url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
|
||||
response = await client.get(url=url)
|
||||
if response.status_code == 200:
|
||||
return {"status": "success", "chat_template": response.content.decode("utf-8")}
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return {"status": "failure"}
|
||||
|
||||
|
||||
def _extract_token_value(token_value: Union[None, str, Dict[str, Any]]) -> str:
|
||||
"""
|
||||
Extract token string from various formats (string, dict, etc.)
|
||||
|
||||
Args:
|
||||
token_value: Token value in various formats (None, str, or dict with 'content' key)
|
||||
|
||||
Returns:
|
||||
Extracted token string
|
||||
"""
|
||||
if token_value is None or isinstance(token_value, str):
|
||||
return token_value or ""
|
||||
if isinstance(token_value, dict):
|
||||
return token_value.get("content", "")
|
||||
return ""
|
||||
|
|
@ -878,7 +878,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
|
|||
|
||||
prompt_tokens_details = PromptTokensDetailsWrapper(
|
||||
cached_tokens=cache_read_input_tokens,
|
||||
cache_creation_tokens=cache_read_input_tokens,
|
||||
cache_creation_tokens=cache_creation_input_tokens,
|
||||
cache_creation_token_details=cache_creation_token_details,
|
||||
)
|
||||
completion_token_details = (
|
||||
|
|
|
|||
|
|
@ -270,7 +270,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
|
|||
processed_chunk.get("delta", {}).get("stop_reason")
|
||||
is not None
|
||||
):
|
||||
|
||||
self.holding_stop_reason_chunk = processed_chunk
|
||||
else:
|
||||
self.chunk_queue.append(processed_chunk)
|
||||
|
|
@ -380,4 +379,11 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
|
|||
self.current_content_block_start = content_block_start
|
||||
return True
|
||||
|
||||
# For parallel tool calls, we'll necessarily have a new content block
|
||||
# if we get a function name since it signals a new tool call
|
||||
if block_type == "tool_use" and content_block_start.get("name"):
|
||||
self.current_content_block_type = block_type
|
||||
self.current_content_block_start = content_block_start
|
||||
return True
|
||||
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -365,6 +365,11 @@ def get_azure_ad_token(
|
|||
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
|
||||
except ValueError:
|
||||
verbose_logger.debug("Azure AD Token Provider could not be used.")
|
||||
except Exception as e:
|
||||
verbose_logger.error(
|
||||
f"Error calling Azure AD token provider: {str(e)}. Follow docs - https://docs.litellm.ai/docs/providers/azure/#azure-ad-token-refresh---defaultazurecredential"
|
||||
)
|
||||
raise e
|
||||
|
||||
#########################################################
|
||||
# If litellm.enable_azure_ad_token_refresh is True and no other token provider is available,
|
||||
|
|
@ -561,7 +566,9 @@ class BaseAzureLLM(BaseOpenAILLM):
|
|||
"Using Azure AD token provider based on Service Principal with Secret workflow for Azure Auth"
|
||||
)
|
||||
try:
|
||||
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
|
||||
azure_ad_token_provider = get_azure_ad_token_provider(
|
||||
azure_scope=scope,
|
||||
)
|
||||
except ValueError:
|
||||
verbose_logger.debug("Azure AD Token Provider could not be used.")
|
||||
if api_version is None:
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
|
|||
# Ensure required fields are present for ResponseReasoningItem
|
||||
item_data = dict(item)
|
||||
if "id" not in item_data:
|
||||
item_data["id"] = f"reasoning_{hash(str(item_data))}"
|
||||
item_data["id"] = f"rs_{hash(str(item_data))}"
|
||||
if "summary" not in item_data:
|
||||
item_data["summary"] = (
|
||||
item_data.get("reasoning_content", "")[:100] + "..."
|
||||
|
|
|
|||
|
|
@ -1092,10 +1092,8 @@ class AmazonConverseConfig(BaseConfig):
|
|||
cache_read_input_tokens = usage["cacheReadInputTokens"]
|
||||
input_tokens += cache_read_input_tokens
|
||||
if "cacheWriteInputTokens" in usage:
|
||||
"""
|
||||
Do not increment prompt_tokens with cacheWriteInputTokens
|
||||
"""
|
||||
cache_creation_input_tokens = usage["cacheWriteInputTokens"]
|
||||
input_tokens += cache_creation_input_tokens
|
||||
|
||||
prompt_tokens_details = PromptTokensDetailsWrapper(
|
||||
cached_tokens=cache_read_input_tokens
|
||||
|
|
|
|||
|
|
@ -164,7 +164,7 @@ class AsyncHTTPHandler:
|
|||
self,
|
||||
timeout: Optional[Union[float, httpx.Timeout]] = None,
|
||||
event_hooks: Optional[Mapping[str, List[Callable[..., Any]]]] = None,
|
||||
concurrent_limit=1000,
|
||||
concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
|
||||
client_alias: Optional[str] = None, # name for client in logs
|
||||
ssl_verify: Optional[VerifyTypes] = None,
|
||||
shared_session: Optional["ClientSession"] = None,
|
||||
|
|
@ -173,7 +173,6 @@ class AsyncHTTPHandler:
|
|||
self.event_hooks = event_hooks
|
||||
self.client = self.create_client(
|
||||
timeout=timeout,
|
||||
concurrent_limit=concurrent_limit,
|
||||
event_hooks=event_hooks,
|
||||
ssl_verify=ssl_verify,
|
||||
shared_session=shared_session,
|
||||
|
|
@ -183,7 +182,6 @@ class AsyncHTTPHandler:
|
|||
def create_client(
|
||||
self,
|
||||
timeout: Optional[Union[float, httpx.Timeout]],
|
||||
concurrent_limit: int,
|
||||
event_hooks: Optional[Mapping[str, List[Callable[..., Any]]]],
|
||||
ssl_verify: Optional[VerifyTypes] = None,
|
||||
shared_session: Optional["ClientSession"] = None,
|
||||
|
|
@ -209,10 +207,6 @@ class AsyncHTTPHandler:
|
|||
transport=transport,
|
||||
event_hooks=event_hooks,
|
||||
timeout=timeout,
|
||||
limits=httpx.Limits(
|
||||
max_connections=concurrent_limit,
|
||||
max_keepalive_connections=concurrent_limit,
|
||||
),
|
||||
verify=ssl_config,
|
||||
cert=cert,
|
||||
headers=headers,
|
||||
|
|
@ -286,7 +280,7 @@ class AsyncHTTPHandler:
|
|||
except (httpx.RemoteProtocolError, httpx.ConnectError):
|
||||
# Retry the request with a new session if there is a connection error
|
||||
new_client = self.create_client(
|
||||
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
|
||||
timeout=timeout, event_hooks=self.event_hooks
|
||||
)
|
||||
try:
|
||||
return await self.single_connection_post_request(
|
||||
|
|
@ -352,7 +346,7 @@ class AsyncHTTPHandler:
|
|||
except (httpx.RemoteProtocolError, httpx.ConnectError):
|
||||
# Retry the request with a new session if there is a connection error
|
||||
new_client = self.create_client(
|
||||
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
|
||||
timeout=timeout, event_hooks=self.event_hooks
|
||||
)
|
||||
try:
|
||||
return await self.single_connection_post_request(
|
||||
|
|
@ -412,7 +406,7 @@ class AsyncHTTPHandler:
|
|||
except (httpx.RemoteProtocolError, httpx.ConnectError):
|
||||
# Retry the request with a new session if there is a connection error
|
||||
new_client = self.create_client(
|
||||
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
|
||||
timeout=timeout, event_hooks=self.event_hooks
|
||||
)
|
||||
try:
|
||||
return await self.single_connection_post_request(
|
||||
|
|
@ -471,7 +465,7 @@ class AsyncHTTPHandler:
|
|||
except (httpx.RemoteProtocolError, httpx.ConnectError):
|
||||
# Retry the request with a new session if there is a connection error
|
||||
new_client = self.create_client(
|
||||
timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
|
||||
timeout=timeout, event_hooks=self.event_hooks
|
||||
)
|
||||
try:
|
||||
return await self.single_connection_post_request(
|
||||
|
|
@ -657,7 +651,7 @@ class AsyncHTTPHandler:
|
|||
)
|
||||
return LiteLLMAiohttpTransport(
|
||||
client=lambda: ClientSession(
|
||||
connector=TCPConnector(**connector_kwargs),
|
||||
connector=TCPConnector(limit=0, **connector_kwargs), # 0 = unlimited connections per host
|
||||
trust_env=trust_env,
|
||||
),
|
||||
)
|
||||
|
|
@ -680,7 +674,7 @@ class HTTPHandler:
|
|||
def __init__(
|
||||
self,
|
||||
timeout: Optional[Union[float, httpx.Timeout]] = None,
|
||||
concurrent_limit=1000,
|
||||
concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
|
||||
client: Optional[httpx.Client] = None,
|
||||
ssl_verify: Optional[Union[bool, str]] = None,
|
||||
):
|
||||
|
|
@ -701,10 +695,6 @@ class HTTPHandler:
|
|||
self.client = httpx.Client(
|
||||
transport=transport,
|
||||
timeout=timeout,
|
||||
limits=httpx.Limits(
|
||||
max_connections=concurrent_limit,
|
||||
max_keepalive_connections=concurrent_limit,
|
||||
),
|
||||
verify=ssl_config,
|
||||
cert=cert,
|
||||
headers=headers,
|
||||
|
|
|
|||
48
litellm/llms/litellm_proxy/responses/transformation.py
Normal file
48
litellm/llms/litellm_proxy/responses/transformation.py
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
"""
|
||||
Responses API transformation for LiteLLM Proxy provider.
|
||||
|
||||
LiteLLM Proxy supports the OpenAI Responses API natively when the underlying model supports it.
|
||||
This config enables pass-through behavior to the proxy's /v1/responses endpoint.
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
from litellm.types.utils import LlmProviders
|
||||
|
||||
|
||||
class LiteLLMProxyResponsesAPIConfig(OpenAIResponsesAPIConfig):
|
||||
"""
|
||||
Configuration for LiteLLM Proxy Responses API support.
|
||||
|
||||
Extends OpenAI's config since the proxy follows OpenAI's API spec,
|
||||
but uses LITELLM_PROXY_API_BASE for the base URL.
|
||||
"""
|
||||
|
||||
@property
|
||||
def custom_llm_provider(self) -> LlmProviders:
|
||||
return LlmProviders.LITELLM_PROXY
|
||||
|
||||
def get_complete_url(
|
||||
self,
|
||||
api_base: Optional[str],
|
||||
litellm_params: dict,
|
||||
) -> str:
|
||||
"""
|
||||
Get the endpoint for LiteLLM Proxy responses API.
|
||||
|
||||
Uses LITELLM_PROXY_API_BASE environment variable if api_base is not provided.
|
||||
"""
|
||||
api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
|
||||
|
||||
if api_base is None:
|
||||
raise ValueError(
|
||||
"api_base not set for LiteLLM Proxy responses API. "
|
||||
"Set via api_base parameter or LITELLM_PROXY_API_BASE environment variable"
|
||||
)
|
||||
|
||||
# Remove trailing slashes
|
||||
api_base = api_base.rstrip("/")
|
||||
|
||||
return f"{api_base}/responses"
|
||||
|
|
@ -397,13 +397,13 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
|
|||
)
|
||||
from litellm.types.llms.openai import ChatCompletionToolParam
|
||||
|
||||
for message in messages:
|
||||
message = cast(
|
||||
for i, message in enumerate(messages):
|
||||
messages[i] = cast(
|
||||
AllMessageValues, filter_value_from_dict(message, "cache_control") # type: ignore
|
||||
)
|
||||
if tools is not None:
|
||||
for tool in tools:
|
||||
tool = cast(
|
||||
for i, tool in enumerate(tools):
|
||||
tools[i] = cast(
|
||||
ChatCompletionToolParam,
|
||||
filter_value_from_dict(tool, "cache_control"), # type: ignore
|
||||
)
|
||||
|
|
|
|||
|
|
@ -207,7 +207,6 @@ class BaseOpenAILLM:
|
|||
ssl_config = get_ssl_configuration()
|
||||
|
||||
return httpx.AsyncClient(
|
||||
limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100),
|
||||
verify=ssl_config,
|
||||
transport=AsyncHTTPHandler._create_async_transport(
|
||||
ssl_context=ssl_config
|
||||
|
|
@ -228,7 +227,6 @@ class BaseOpenAILLM:
|
|||
ssl_config = get_ssl_configuration()
|
||||
|
||||
return httpx.Client(
|
||||
limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100),
|
||||
verify=ssl_config,
|
||||
follow_redirects=True,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -124,7 +124,7 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
|
|||
# Ensure required fields are present for ResponseReasoningItem
|
||||
item_data = dict(item)
|
||||
if "id" not in item_data:
|
||||
item_data["id"] = f"reasoning_{hash(str(item_data))}"
|
||||
item_data["id"] = f"rs_{hash(str(item_data))}"
|
||||
if "summary" not in item_data:
|
||||
item_data["summary"] = (
|
||||
item_data.get("reasoning_content", "")[:100] + "..."
|
||||
|
|
@ -271,6 +271,7 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
|
|||
ResponsesAPIStreamEvents.MCP_CALL_ARGUMENTS_DONE: MCPCallArgumentsDoneEvent,
|
||||
ResponsesAPIStreamEvents.MCP_CALL_COMPLETED: MCPCallCompletedEvent,
|
||||
ResponsesAPIStreamEvents.MCP_CALL_FAILED: MCPCallFailedEvent,
|
||||
ResponsesAPIStreamEvents.IMAGE_GENERATION_PARTIAL_IMAGE: ImageGenerationPartialImageEvent,
|
||||
ResponsesAPIStreamEvents.ERROR: ErrorEvent,
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ Calls done in OpenAI/openai.py as OpenRouter is openai-compatible.
|
|||
Docs: https://openrouter.ai/docs/parameters
|
||||
"""
|
||||
|
||||
from enum import Enum
|
||||
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union
|
||||
|
||||
import httpx
|
||||
|
|
@ -20,6 +21,12 @@ from ...openai.chat.gpt_transformation import OpenAIGPTConfig
|
|||
from ..common_utils import OpenRouterException
|
||||
|
||||
|
||||
class CacheControlSupportedModels(str, Enum):
|
||||
"""Models that support cache_control in content blocks."""
|
||||
CLAUDE = "claude"
|
||||
GEMINI = "gemini"
|
||||
|
||||
|
||||
class OpenrouterConfig(OpenAIGPTConfig):
|
||||
def map_openai_params(
|
||||
self,
|
||||
|
|
@ -48,19 +55,73 @@ class OpenrouterConfig(OpenAIGPTConfig):
|
|||
)
|
||||
return mapped_openai_params
|
||||
|
||||
def _supports_cache_control_in_content(self, model: str) -> bool:
|
||||
"""
|
||||
Check if the model supports cache_control in content blocks.
|
||||
|
||||
Returns:
|
||||
bool: True if model supports cache_control (Claude or Gemini models)
|
||||
"""
|
||||
model_lower = model.lower()
|
||||
return any(
|
||||
supported_model.value in model_lower
|
||||
for supported_model in CacheControlSupportedModels
|
||||
)
|
||||
|
||||
def remove_cache_control_flag_from_messages_and_tools(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
tools: Optional[List["ChatCompletionToolParam"]] = None,
|
||||
) -> Tuple[List[AllMessageValues], Optional[List["ChatCompletionToolParam"]]]:
|
||||
if "claude" in model.lower(): # don't remove 'cache_control' flag
|
||||
if self._supports_cache_control_in_content(model):
|
||||
return messages, tools
|
||||
else:
|
||||
return super().remove_cache_control_flag_from_messages_and_tools(
|
||||
model, messages, tools
|
||||
)
|
||||
|
||||
def _move_cache_control_to_content(
|
||||
self, messages: List[AllMessageValues]
|
||||
) -> List[AllMessageValues]:
|
||||
"""
|
||||
Move cache_control from message level to content blocks.
|
||||
OpenRouter requires cache_control to be inside content blocks, not at message level.
|
||||
|
||||
When cache_control is at message level, it's added to ALL content blocks
|
||||
to cache the entire message content.
|
||||
"""
|
||||
transformed_messages = []
|
||||
for message in messages:
|
||||
message_copy = dict(message)
|
||||
cache_control = message_copy.pop("cache_control", None)
|
||||
|
||||
if cache_control is not None:
|
||||
content = message_copy.get("content")
|
||||
|
||||
if isinstance(content, list):
|
||||
# Content is already a list, add cache_control to all blocks
|
||||
if len(content) > 0:
|
||||
content_copy = []
|
||||
for block in content:
|
||||
block_copy = dict(block)
|
||||
block_copy["cache_control"] = cache_control
|
||||
content_copy.append(block_copy)
|
||||
message_copy["content"] = content_copy
|
||||
else:
|
||||
# Content is a string, convert to structured format
|
||||
message_copy["content"] = [
|
||||
{
|
||||
"type": "text",
|
||||
"text": content,
|
||||
"cache_control": cache_control,
|
||||
}
|
||||
]
|
||||
|
||||
transformed_messages.append(message_copy)
|
||||
|
||||
return transformed_messages
|
||||
|
||||
def transform_request(
|
||||
self,
|
||||
model: str,
|
||||
|
|
@ -75,6 +136,9 @@ class OpenrouterConfig(OpenAIGPTConfig):
|
|||
Returns:
|
||||
dict: The transformed request. Sent as the body of the API call.
|
||||
"""
|
||||
if self._supports_cache_control_in_content(model):
|
||||
messages = self._move_cache_control_to_content(messages)
|
||||
|
||||
extra_body = optional_params.pop("extra_body", {})
|
||||
response = super().transform_request(
|
||||
model, messages, optional_params, litellm_params, headers
|
||||
|
|
|
|||
|
|
@ -4,10 +4,14 @@ Translation from OpenAI's `/chat/completions` endpoint to IBM WatsonX's `/text/c
|
|||
Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-chat
|
||||
"""
|
||||
|
||||
from typing import List, Optional, Tuple, Union
|
||||
from typing import Dict, List, Optional, Tuple, Union
|
||||
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
from litellm.types.llms.watsonx import WatsonXAIEndpoint, WatsonXAPIParams
|
||||
from litellm.types.llms.watsonx import (
|
||||
WatsonXAIEndpoint,
|
||||
WatsonXAPIParams,
|
||||
WatsonXModelPattern,
|
||||
)
|
||||
|
||||
from ....utils import _remove_additional_properties, _remove_strict_from_schema
|
||||
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
|
||||
|
|
@ -120,3 +124,95 @@ class IBMWatsonXChatConfig(IBMWatsonXMixin, OpenAIGPTConfig):
|
|||
None if model.startswith("deployment/") else api_params["project_id"]
|
||||
)
|
||||
return payload
|
||||
|
||||
@staticmethod
|
||||
def _apply_prompt_template_core(model: str, messages: List[Dict[str, str]], hf_template_fn) -> Optional[str]:
|
||||
"""Core logic for applying prompt templates"""
|
||||
from litellm.litellm_core_utils.prompt_templates.factory import (
|
||||
custom_prompt,
|
||||
ibm_granite_pt,
|
||||
mistral_instruct_pt,
|
||||
)
|
||||
|
||||
if WatsonXModelPattern.GRANITE_CHAT.value in model:
|
||||
return ibm_granite_pt(messages=messages)
|
||||
elif WatsonXModelPattern.IBM_MISTRAL.value in model:
|
||||
return mistral_instruct_pt(messages=messages)
|
||||
elif WatsonXModelPattern.GPT_OSS.value in model:
|
||||
hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
|
||||
try:
|
||||
return hf_template_fn(model=hf_model, messages=messages)
|
||||
except Exception:
|
||||
pass
|
||||
elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
|
||||
return custom_prompt(
|
||||
role_dict={
|
||||
"system": {"pre_message": "<|start_header_id|>system<|end_header_id|>\n", "post_message": "<|eot_id|>"},
|
||||
"user": {"pre_message": "<|start_header_id|>user<|end_header_id|>\n", "post_message": "<|eot_id|>"},
|
||||
"assistant": {"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n", "post_message": "<|eot_id|>"},
|
||||
},
|
||||
messages=messages,
|
||||
initial_prompt_value="<|begin_of_text|>",
|
||||
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
|
||||
)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
async def aapply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
|
||||
"""Apply prompt template (async version)"""
|
||||
import litellm
|
||||
from litellm.litellm_core_utils.prompt_templates.factory import (
|
||||
ahf_chat_template,
|
||||
custom_prompt,
|
||||
hf_chat_template,
|
||||
ibm_granite_pt,
|
||||
mistral_instruct_pt,
|
||||
)
|
||||
|
||||
if WatsonXModelPattern.GRANITE_CHAT.value in model:
|
||||
return ibm_granite_pt(messages=messages)
|
||||
elif WatsonXModelPattern.IBM_MISTRAL.value in model:
|
||||
return mistral_instruct_pt(messages=messages)
|
||||
elif WatsonXModelPattern.GPT_OSS.value in model:
|
||||
hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
|
||||
try:
|
||||
# Use sync if cached, async if not
|
||||
if hf_model in litellm.known_tokenizer_config:
|
||||
return hf_chat_template(model=hf_model, messages=messages)
|
||||
else:
|
||||
return await ahf_chat_template(model=hf_model, messages=messages)
|
||||
except Exception:
|
||||
pass
|
||||
elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
|
||||
return custom_prompt(
|
||||
role_dict={
|
||||
"system": {
|
||||
"pre_message": "<|start_header_id|>system<|end_header_id|>\n",
|
||||
"post_message": "<|eot_id|>",
|
||||
},
|
||||
"user": {
|
||||
"pre_message": "<|start_header_id|>user<|end_header_id|>\n",
|
||||
"post_message": "<|eot_id|>",
|
||||
},
|
||||
"assistant": {
|
||||
"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
|
||||
"post_message": "<|eot_id|>",
|
||||
},
|
||||
},
|
||||
messages=messages,
|
||||
initial_prompt_value="<|begin_of_text|>",
|
||||
final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
|
||||
)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def apply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
|
||||
"""Apply prompt template (sync version)"""
|
||||
from litellm.litellm_core_utils.prompt_templates.factory import (
|
||||
hf_chat_template,
|
||||
)
|
||||
|
||||
return IBMWatsonXChatConfig._apply_prompt_template_core(
|
||||
model=model, messages=messages, hf_template_fn=hf_chat_template
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -131,34 +131,102 @@ def _get_api_params(
|
|||
)
|
||||
|
||||
|
||||
def convert_watsonx_messages_to_prompt(
|
||||
async def _aconvert_watsonx_messages_core(
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
provider: str,
|
||||
custom_prompt_dict: Dict,
|
||||
apply_template_fn,
|
||||
) -> str:
|
||||
"""Async core logic for converting watsonx messages to prompt"""
|
||||
from litellm.types.llms.watsonx import WatsonXModelPattern
|
||||
|
||||
# handle anthropic prompts and amazon titan prompts
|
||||
if model in custom_prompt_dict:
|
||||
# check if the model has a registered custom prompt
|
||||
model_prompt_dict = custom_prompt_dict[model]
|
||||
prompt = ptf.custom_prompt(
|
||||
return ptf.custom_prompt(
|
||||
messages=messages,
|
||||
role_dict=model_prompt_dict.get(
|
||||
"role_dict", model_prompt_dict.get("roles")
|
||||
),
|
||||
role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
|
||||
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
|
||||
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
|
||||
bos_token=model_prompt_dict.get("bos_token", ""),
|
||||
eos_token=model_prompt_dict.get("eos_token", ""),
|
||||
)
|
||||
return prompt
|
||||
elif provider == "ibm-mistralai":
|
||||
prompt = ptf.mistral_instruct_pt(messages=messages)
|
||||
elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
|
||||
return ptf.mistral_instruct_pt(messages=messages)
|
||||
else:
|
||||
prompt: str = ptf.prompt_factory( # type: ignore
|
||||
# Try applying specific template first
|
||||
result = await apply_template_fn(model=model, messages=messages)
|
||||
if result:
|
||||
return result
|
||||
# Fallback to default
|
||||
return ptf.prompt_factory(
|
||||
model=model, messages=messages, custom_llm_provider="watsonx"
|
||||
) # type: ignore
|
||||
|
||||
|
||||
def _convert_watsonx_messages_core(
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
provider: str,
|
||||
custom_prompt_dict: Dict,
|
||||
apply_template_fn,
|
||||
) -> str:
|
||||
"""Sync core logic for converting watsonx messages to prompt"""
|
||||
from litellm.types.llms.watsonx import WatsonXModelPattern
|
||||
|
||||
# handle anthropic prompts and amazon titan prompts
|
||||
if model in custom_prompt_dict:
|
||||
model_prompt_dict = custom_prompt_dict[model]
|
||||
return ptf.custom_prompt(
|
||||
messages=messages,
|
||||
role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
|
||||
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
|
||||
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
|
||||
bos_token=model_prompt_dict.get("bos_token", ""),
|
||||
eos_token=model_prompt_dict.get("eos_token", ""),
|
||||
)
|
||||
return prompt
|
||||
elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
|
||||
return ptf.mistral_instruct_pt(messages=messages)
|
||||
else:
|
||||
# Try applying specific template first
|
||||
result = apply_template_fn(model=model, messages=messages)
|
||||
if result:
|
||||
return result
|
||||
# Fallback to default
|
||||
return ptf.prompt_factory(
|
||||
model=model, messages=messages, custom_llm_provider="watsonx"
|
||||
) # type: ignore
|
||||
|
||||
|
||||
async def aconvert_watsonx_messages_to_prompt(
|
||||
model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
|
||||
) -> str:
|
||||
"""Async version of convert_watsonx_messages_to_prompt"""
|
||||
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
|
||||
|
||||
return await _aconvert_watsonx_messages_core(
|
||||
model=model,
|
||||
messages=messages,
|
||||
provider=provider,
|
||||
custom_prompt_dict=custom_prompt_dict,
|
||||
apply_template_fn=IBMWatsonXChatConfig.aapply_prompt_template,
|
||||
)
|
||||
|
||||
|
||||
def convert_watsonx_messages_to_prompt(
|
||||
model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
|
||||
) -> str:
|
||||
"""Sync version of convert_watsonx_messages_to_prompt"""
|
||||
from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
|
||||
|
||||
return _convert_watsonx_messages_core(
|
||||
model=model,
|
||||
messages=messages,
|
||||
provider=provider,
|
||||
custom_prompt_dict=custom_prompt_dict,
|
||||
apply_template_fn=IBMWatsonXChatConfig.apply_prompt_template,
|
||||
)
|
||||
|
||||
|
||||
# Mixin class for shared IBM Watson X functionality
|
||||
|
|
|
|||
|
|
@ -228,39 +228,35 @@ class IBMWatsonXAIConfig(IBMWatsonXMixin, BaseConfig):
|
|||
"us-south",
|
||||
]
|
||||
|
||||
def transform_request(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
optional_params: Dict,
|
||||
litellm_params: Dict,
|
||||
headers: Dict,
|
||||
) -> Dict:
|
||||
provider = model.split("/")[0]
|
||||
prompt = convert_watsonx_messages_to_prompt(
|
||||
model=model,
|
||||
messages=messages,
|
||||
provider=provider,
|
||||
custom_prompt_dict={},
|
||||
)
|
||||
def _build_request_payload(self, model: str, prompt: str, optional_params: Dict) -> Dict:
|
||||
"""Shared logic to build request payload"""
|
||||
extra_body_params = optional_params.pop("extra_body", {})
|
||||
optional_params.update(extra_body_params)
|
||||
watsonx_api_params = _get_api_params(params=optional_params)
|
||||
|
||||
watsonx_auth_payload = self._prepare_payload(
|
||||
model=model,
|
||||
api_params=watsonx_api_params,
|
||||
)
|
||||
|
||||
# init the payload to the text generation call
|
||||
payload = {
|
||||
watsonx_auth_payload = self._prepare_payload(model=model, api_params=watsonx_api_params)
|
||||
|
||||
return {
|
||||
"input": prompt,
|
||||
"moderations": optional_params.pop("moderations", {}),
|
||||
"parameters": optional_params,
|
||||
**watsonx_auth_payload,
|
||||
}
|
||||
|
||||
return payload
|
||||
async def atransform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
|
||||
"""Async version of transform_request"""
|
||||
from litellm.llms.watsonx.common_utils import (
|
||||
aconvert_watsonx_messages_to_prompt,
|
||||
)
|
||||
|
||||
provider = model.split("/")[0]
|
||||
prompt = await aconvert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
|
||||
return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
|
||||
|
||||
def transform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
|
||||
"""Sync version of transform_request"""
|
||||
provider = model.split("/")[0]
|
||||
prompt = convert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
|
||||
return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
|
||||
|
||||
def transform_response(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -5294,6 +5294,16 @@
|
|||
"output_cost_per_token": 0.0,
|
||||
"supports_embedding_image_input": true
|
||||
},
|
||||
"cohere.embed-v4:0": {
|
||||
"input_cost_per_token": 1.2e-07,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "embedding",
|
||||
"output_cost_per_token": 0.0,
|
||||
"output_vector_size": 1536,
|
||||
"supports_embedding_image_input": true
|
||||
},
|
||||
"cohere.rerank-v3-5:0": {
|
||||
"input_cost_per_query": 0.002,
|
||||
"input_cost_per_token": 0.0,
|
||||
|
|
@ -12965,6 +12975,39 @@
|
|||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"gpt-5-pro-2025-10-06": {
|
||||
"input_cost_per_token": 1.5e-05,
|
||||
"input_cost_per_token_batches": 7.5e-06,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 400000,
|
||||
"max_output_tokens": 272000,
|
||||
"max_tokens": 272000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.2e-04,
|
||||
"output_cost_per_token_batches": 6e-05,
|
||||
"supported_endpoints": [
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": false,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"gpt-5-codex": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
"input_cost_per_token": 1.25e-06,
|
||||
|
|
@ -13293,8 +13336,7 @@
|
|||
"supported_endpoints": [
|
||||
"/v1/images/generations",
|
||||
"/v1/images/edits"
|
||||
],
|
||||
"supports_vision": true
|
||||
]
|
||||
},
|
||||
"gpt-realtime": {
|
||||
"cache_creation_input_audio_token_cost": 4e-07,
|
||||
|
|
@ -13328,6 +13370,37 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true
|
||||
},
|
||||
"gpt-realtime-mini": {
|
||||
"cache_creation_input_audio_token_cost": 3e-07,
|
||||
"cache_read_input_audio_token_cost": 3e-07,
|
||||
"input_cost_per_audio_token": 1e-05,
|
||||
"input_cost_per_token": 6e-07,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 4096,
|
||||
"max_tokens": 4096,
|
||||
"mode": "chat",
|
||||
"output_cost_per_audio_token": 2e-05,
|
||||
"output_cost_per_token": 2.4e-06,
|
||||
"supported_endpoints": [
|
||||
"/v1/realtime"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image",
|
||||
"audio"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text",
|
||||
"audio"
|
||||
],
|
||||
"supports_audio_input": true,
|
||||
"supports_audio_output": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true
|
||||
},
|
||||
"gpt-realtime-2025-08-28": {
|
||||
"cache_creation_input_audio_token_cost": 4e-07,
|
||||
"cache_read_input_token_cost": 4e-07,
|
||||
|
|
|
|||
|
|
@ -109,10 +109,21 @@ class MCPRequestHandler:
|
|||
request.body = mock_body # type: ignore
|
||||
if ".well-known" in str(request.url): # public routes
|
||||
validated_user_api_key_auth = UserAPIKeyAuth()
|
||||
# elif litellm_api_key == "":
|
||||
# from fastapi import HTTPException
|
||||
|
||||
# raise HTTPException(
|
||||
# status_code=401,
|
||||
# detail="LiteLLM API key is missing. Please add it or use OAuth authentication.",
|
||||
# headers={
|
||||
# "WWW-Authenticate": f'Bearer resource_metadata=f"{request.base_url}/.well-known/oauth-protected-resource"',
|
||||
# },
|
||||
# )
|
||||
else:
|
||||
validated_user_api_key_auth = await user_api_key_auth(
|
||||
api_key=litellm_api_key, request=request
|
||||
)
|
||||
|
||||
return (
|
||||
validated_user_api_key_auth,
|
||||
mcp_auth_header,
|
||||
|
|
@ -344,14 +355,14 @@ class MCPRequestHandler:
|
|||
proxy_logging_obj,
|
||||
user_api_key_cache,
|
||||
)
|
||||
|
||||
|
||||
if not user_api_key_auth:
|
||||
return None
|
||||
|
||||
|
||||
# Already loaded
|
||||
if user_api_key_auth.object_permission:
|
||||
return user_api_key_auth.object_permission
|
||||
|
||||
|
||||
# Need to fetch from DB
|
||||
if user_api_key_auth.object_permission_id and prisma_client:
|
||||
return await get_object_permission(
|
||||
|
|
@ -361,7 +372,7 @@ class MCPRequestHandler:
|
|||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -369,16 +380,20 @@ class MCPRequestHandler:
|
|||
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
|
||||
):
|
||||
"""Helper to get team object_permission from cache or DB."""
|
||||
from litellm.proxy.auth.auth_checks import get_team_object
|
||||
from litellm.proxy.auth.auth_checks import (
|
||||
get_object_permission,
|
||||
get_team_object,
|
||||
)
|
||||
from litellm.proxy.proxy_server import (
|
||||
prisma_client,
|
||||
proxy_logging_obj,
|
||||
user_api_key_cache,
|
||||
)
|
||||
|
||||
|
||||
if not user_api_key_auth or not user_api_key_auth.team_id or not prisma_client:
|
||||
return None
|
||||
|
||||
|
||||
# First get the team object (which may have object_permission already loaded)
|
||||
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
|
||||
team_id=user_api_key_auth.team_id,
|
||||
prisma_client=prisma_client,
|
||||
|
|
@ -386,8 +401,25 @@ class MCPRequestHandler:
|
|||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
return team_obj.object_permission if team_obj else None
|
||||
|
||||
if not team_obj:
|
||||
return None
|
||||
|
||||
# Already loaded
|
||||
if team_obj.object_permission:
|
||||
return team_obj.object_permission
|
||||
|
||||
# Need to fetch from DB using object_permission_id
|
||||
if team_obj.object_permission_id:
|
||||
return await get_object_permission(
|
||||
object_permission_id=team_obj.object_permission_id,
|
||||
prisma_client=prisma_client,
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
async def get_allowed_tools_for_server(
|
||||
|
|
@ -397,26 +429,38 @@ class MCPRequestHandler:
|
|||
"""
|
||||
Get list of allowed tool names for a specific server based on key/team permissions.
|
||||
Follows same inheritance logic as get_allowed_mcp_servers.
|
||||
|
||||
|
||||
Args:
|
||||
server_id: Server ID to check permissions for
|
||||
user_api_key_auth: User auth
|
||||
|
||||
|
||||
Returns:
|
||||
List[str] if restrictions exist, None if no restrictions (allow all)
|
||||
"""
|
||||
if not user_api_key_auth:
|
||||
return None
|
||||
|
||||
|
||||
try:
|
||||
# Get key and team object permissions
|
||||
key_obj_perm = await MCPRequestHandler._get_key_object_permission(user_api_key_auth)
|
||||
team_obj_perm = await MCPRequestHandler._get_team_object_permission(user_api_key_auth)
|
||||
|
||||
key_obj_perm = await MCPRequestHandler._get_key_object_permission(
|
||||
user_api_key_auth
|
||||
)
|
||||
team_obj_perm = await MCPRequestHandler._get_team_object_permission(
|
||||
user_api_key_auth
|
||||
)
|
||||
|
||||
# Extract tool permissions for this server
|
||||
key_tools = key_obj_perm.mcp_tool_permissions.get(server_id) if key_obj_perm and key_obj_perm.mcp_tool_permissions else None
|
||||
team_tools = team_obj_perm.mcp_tool_permissions.get(server_id) if team_obj_perm and team_obj_perm.mcp_tool_permissions else None
|
||||
|
||||
key_tools = (
|
||||
key_obj_perm.mcp_tool_permissions.get(server_id)
|
||||
if key_obj_perm and key_obj_perm.mcp_tool_permissions
|
||||
else None
|
||||
)
|
||||
team_tools = (
|
||||
team_obj_perm.mcp_tool_permissions.get(server_id)
|
||||
if team_obj_perm and team_obj_perm.mcp_tool_permissions
|
||||
else None
|
||||
)
|
||||
|
||||
# Apply same inheritance logic as get_allowed_mcp_servers
|
||||
if team_tools:
|
||||
if key_tools:
|
||||
|
|
@ -428,7 +472,7 @@ class MCPRequestHandler:
|
|||
else:
|
||||
# No team restrictions → use key restrictions
|
||||
return key_tools
|
||||
|
||||
|
||||
except Exception as e:
|
||||
verbose_logger.warning(f"Failed to get allowed tools for server: {str(e)}")
|
||||
return None
|
||||
|
|
@ -441,12 +485,12 @@ class MCPRequestHandler:
|
|||
) -> bool:
|
||||
"""
|
||||
Check if a specific tool is allowed for a server based on key/team permissions.
|
||||
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool to check
|
||||
server_id: Server ID
|
||||
user_api_key_auth: User auth
|
||||
|
||||
|
||||
Returns:
|
||||
True if allowed, False if blocked
|
||||
"""
|
||||
|
|
@ -454,15 +498,15 @@ class MCPRequestHandler:
|
|||
server_id=server_id,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
|
||||
|
||||
# None means no restrictions (allow all)
|
||||
if allowed_tools is None:
|
||||
return True
|
||||
|
||||
|
||||
# Empty list means no tools allowed
|
||||
if not allowed_tools:
|
||||
return False
|
||||
|
||||
|
||||
# Check if tool is in allowed list
|
||||
return tool_name in allowed_tools
|
||||
|
||||
|
|
@ -536,41 +580,24 @@ class MCPRequestHandler:
|
|||
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
|
||||
) -> List[str]:
|
||||
"""
|
||||
The `object_permission` for a team is not stored on the user_api_key_auth object
|
||||
Get allowed MCP servers for a team.
|
||||
|
||||
first we check if the team has a object_permission_id attached
|
||||
- if it does then we look up the object_permission for the team
|
||||
Uses the helper _get_team_object_permission which:
|
||||
1. First checks if object_permission is already loaded on the team
|
||||
2. If not, fetches from DB using object_permission_id if it exists
|
||||
"""
|
||||
from litellm.proxy.auth.auth_checks import get_team_object
|
||||
from litellm.proxy.proxy_server import (
|
||||
prisma_client,
|
||||
proxy_logging_obj,
|
||||
user_api_key_cache,
|
||||
)
|
||||
|
||||
if user_api_key_auth is None:
|
||||
return []
|
||||
|
||||
if user_api_key_auth.team_id is None:
|
||||
return []
|
||||
|
||||
if prisma_client is None:
|
||||
verbose_logger.debug("prisma_client is None")
|
||||
return []
|
||||
|
||||
try:
|
||||
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
|
||||
team_id=user_api_key_auth.team_id,
|
||||
prisma_client=prisma_client,
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
# Use the helper method that properly handles fetching from DB if needed
|
||||
object_permissions = await MCPRequestHandler._get_team_object_permission(
|
||||
user_api_key_auth
|
||||
)
|
||||
if team_obj is None:
|
||||
verbose_logger.debug("team_obj is None")
|
||||
return []
|
||||
|
||||
object_permissions = team_obj.object_permission
|
||||
if object_permissions is None:
|
||||
return []
|
||||
|
||||
|
|
|
|||
|
|
@ -195,6 +195,7 @@ class MCPServerManager:
|
|||
name=name_for_prefix,
|
||||
alias=alias,
|
||||
server_name=server_name,
|
||||
spec_path=server_config.get("spec_path", None),
|
||||
url=server_config.get("url", None) or "",
|
||||
command=server_config.get("command", None) or "",
|
||||
args=server_config.get("args", None) or [],
|
||||
|
|
@ -215,15 +216,167 @@ class MCPServerManager:
|
|||
extra_headers=server_config.get("extra_headers", None),
|
||||
allowed_tools=server_config.get("allowed_tools", None),
|
||||
disallowed_tools=server_config.get("disallowed_tools", None),
|
||||
allowed_params=server_config.get("allowed_params", None),
|
||||
access_groups=server_config.get("access_groups", None),
|
||||
)
|
||||
self.config_mcp_servers[server_id] = new_server
|
||||
|
||||
# Check if this is an OpenAPI-based server
|
||||
spec_path = server_config.get("spec_path", None)
|
||||
if spec_path:
|
||||
verbose_logger.info(
|
||||
f"Loading OpenAPI spec from {spec_path} for server {server_name}"
|
||||
)
|
||||
self._register_openapi_tools(
|
||||
spec_path=spec_path,
|
||||
server=new_server,
|
||||
base_url=server_config.get("url", ""),
|
||||
)
|
||||
|
||||
verbose_logger.debug(
|
||||
f"Loaded MCP Servers: {json.dumps(self.config_mcp_servers, indent=4, default=str)}"
|
||||
)
|
||||
|
||||
self.initialize_tool_name_to_mcp_server_name_mapping()
|
||||
|
||||
def _register_openapi_tools(self, spec_path: str, server: MCPServer, base_url: str):
|
||||
"""
|
||||
Register tools from an OpenAPI specification for a given server.
|
||||
|
||||
This creates "virtual" MCP tools from OpenAPI endpoints that are:
|
||||
1. Registered in the global tool registry with server prefix
|
||||
2. Mapped to the server for routing
|
||||
3. Executed via the local tool handler
|
||||
|
||||
Args:
|
||||
spec_path: Path to the OpenAPI specification file
|
||||
server: The MCPServer instance to register tools for
|
||||
base_url: Base URL for API calls
|
||||
"""
|
||||
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
|
||||
build_input_schema,
|
||||
create_tool_function,
|
||||
)
|
||||
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
|
||||
get_base_url as get_openapi_base_url,
|
||||
)
|
||||
from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
|
||||
load_openapi_spec,
|
||||
)
|
||||
from litellm.proxy._experimental.mcp_server.tool_registry import (
|
||||
global_mcp_tool_registry,
|
||||
)
|
||||
|
||||
try:
|
||||
# Load OpenAPI spec
|
||||
spec = load_openapi_spec(spec_path)
|
||||
|
||||
# Use base_url from config if provided, otherwise extract from spec
|
||||
if not base_url:
|
||||
base_url = get_openapi_base_url(spec)
|
||||
|
||||
verbose_logger.info(
|
||||
f"Registering OpenAPI tools for server {server.name} with base URL: {base_url}"
|
||||
)
|
||||
|
||||
# Get server prefix for tool naming
|
||||
server_prefix = get_server_prefix(server)
|
||||
|
||||
# Build headers from server configuration
|
||||
headers = {}
|
||||
|
||||
# Add authentication headers if configured
|
||||
if server.authentication_token:
|
||||
from litellm.types.mcp import MCPAuth
|
||||
|
||||
if server.auth_type == MCPAuth.bearer_token:
|
||||
headers["Authorization"] = f"Bearer {server.authentication_token}"
|
||||
elif server.auth_type == MCPAuth.api_key:
|
||||
headers["Authorization"] = f"ApiKey {server.authentication_token}"
|
||||
elif server.auth_type == MCPAuth.basic:
|
||||
headers["Authorization"] = f"Basic {server.authentication_token}"
|
||||
|
||||
# Add any extra headers from server config
|
||||
# Note: extra_headers is a List[str] of header names to forward, not a dict
|
||||
# For OpenAPI tools, we'll just use the authentication headers
|
||||
# If extra_headers were needed, they would be processed separately
|
||||
|
||||
verbose_logger.debug(
|
||||
f"Using headers for OpenAPI tools (excluding sensitive values): "
|
||||
f"{list(headers.keys())}"
|
||||
)
|
||||
|
||||
# Extract and register tools from OpenAPI paths
|
||||
paths = spec.get("paths", {})
|
||||
registered_count = 0
|
||||
|
||||
verbose_logger.debug(f"Processing {len(paths)} paths from OpenAPI spec")
|
||||
|
||||
for path, path_item in paths.items():
|
||||
for method in ["get", "post", "put", "delete", "patch"]:
|
||||
if method not in path_item:
|
||||
continue
|
||||
|
||||
operation = path_item[method]
|
||||
|
||||
# Generate tool name (without prefix initially)
|
||||
operation_id = operation.get(
|
||||
"operationId", f"{method}_{path.replace('/', '_')}"
|
||||
)
|
||||
base_tool_name = operation_id.replace(" ", "_").lower()
|
||||
|
||||
# Add server prefix to tool name
|
||||
prefixed_tool_name = add_server_prefix_to_tool_name(
|
||||
base_tool_name, server_prefix
|
||||
)
|
||||
|
||||
# Get description
|
||||
description = operation.get(
|
||||
"summary",
|
||||
operation.get("description", f"{method.upper()} {path}"),
|
||||
)
|
||||
|
||||
# Build input schema using imported function
|
||||
input_schema = build_input_schema(operation)
|
||||
|
||||
# Create tool function with headers using imported function
|
||||
tool_func = create_tool_function(
|
||||
path, method, operation, base_url, headers=headers
|
||||
)
|
||||
tool_func.__name__ = prefixed_tool_name
|
||||
tool_func.__doc__ = description
|
||||
|
||||
# Register tool with prefixed name in global registry
|
||||
global_mcp_tool_registry.register_tool(
|
||||
name=prefixed_tool_name,
|
||||
description=description,
|
||||
input_schema=input_schema,
|
||||
handler=tool_func,
|
||||
)
|
||||
|
||||
# Update tool name to server name mapping (for both prefixed and base names)
|
||||
self.tool_name_to_mcp_server_name_mapping[base_tool_name] = (
|
||||
server_prefix
|
||||
)
|
||||
self.tool_name_to_mcp_server_name_mapping[prefixed_tool_name] = (
|
||||
server_prefix
|
||||
)
|
||||
|
||||
registered_count += 1
|
||||
verbose_logger.debug(
|
||||
f"Registered OpenAPI tool: {prefixed_tool_name} for server {server.name}"
|
||||
)
|
||||
|
||||
verbose_logger.info(
|
||||
f"Successfully registered {registered_count} OpenAPI tools for server {server.name}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
verbose_logger.error(
|
||||
f"Failed to register OpenAPI tools for server {server.name}: {str(e)}"
|
||||
)
|
||||
raise e
|
||||
|
||||
def remove_server(self, mcp_server: LiteLLM_MCPServerTable):
|
||||
"""
|
||||
Remove a server from the registry
|
||||
|
|
@ -469,6 +622,10 @@ class MCPServerManager:
|
|||
Returns:
|
||||
List[MCPTool]: List of tools available on the server with prefixed names
|
||||
"""
|
||||
from litellm.proxy._experimental.mcp_server.tool_registry import (
|
||||
global_mcp_tool_registry,
|
||||
)
|
||||
|
||||
verbose_logger.debug(f"Connecting to url: {server.url}")
|
||||
verbose_logger.info(f"_get_tools_from_server for {server.name}...")
|
||||
|
||||
|
|
@ -481,7 +638,14 @@ class MCPServerManager:
|
|||
extra_headers=extra_headers,
|
||||
)
|
||||
|
||||
tools = await self._fetch_tools_with_timeout(client, server.name)
|
||||
## HANDLE OPENAPI TOOLS
|
||||
if server.spec_path:
|
||||
_tools = global_mcp_tool_registry.list_tools(tool_prefix=server.name)
|
||||
tools = global_mcp_tool_registry.convert_tools_to_mcp_sdk_tool_type(
|
||||
_tools
|
||||
)
|
||||
else:
|
||||
tools = await self._fetch_tools_with_timeout(client, server.name)
|
||||
|
||||
prefixed_or_original_tools = self._create_prefixed_tools(
|
||||
tools, server, add_prefix=add_prefix
|
||||
|
|
@ -597,11 +761,69 @@ class MCPServerManager:
|
|||
Check if the tool is allowed or banned for the given server
|
||||
"""
|
||||
if server.allowed_tools:
|
||||
return tool_name in server.allowed_tools
|
||||
return (
|
||||
tool_name in server.allowed_tools
|
||||
or f"{server.name}-{tool_name}" in server.allowed_tools
|
||||
)
|
||||
if server.disallowed_tools:
|
||||
return tool_name not in server.disallowed_tools
|
||||
return (
|
||||
tool_name not in server.disallowed_tools
|
||||
and f"{server.name}-{tool_name}" not in server.disallowed_tools
|
||||
)
|
||||
return True
|
||||
|
||||
def validate_allowed_params(
|
||||
self, tool_name: str, arguments: Dict[str, Any], server: MCPServer
|
||||
) -> None:
|
||||
"""
|
||||
Filter arguments to only include allowed parameters for the given tool.
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool (with or without prefix)
|
||||
arguments: Dictionary of arguments to filter
|
||||
server: MCPServer configuration
|
||||
|
||||
Returns:
|
||||
Filtered dictionary containing only allowed parameters
|
||||
|
||||
Raises:
|
||||
HTTPException: If allowed_params is configured for this tool but arguments contain disallowed params
|
||||
"""
|
||||
from litellm.proxy._experimental.mcp_server.utils import (
|
||||
get_server_name_prefix_tool_mcp,
|
||||
)
|
||||
|
||||
# If no allowed_params configured, return all arguments
|
||||
if not server.allowed_params:
|
||||
return
|
||||
|
||||
# Get the unprefixed tool name to match against config
|
||||
unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(tool_name)
|
||||
|
||||
# Check both prefixed and unprefixed tool names
|
||||
allowed_params_list = server.allowed_params.get(
|
||||
tool_name
|
||||
) or server.allowed_params.get(unprefixed_tool_name)
|
||||
|
||||
# If this tool doesn't have allowed_params specified, allow all params
|
||||
if allowed_params_list is None:
|
||||
return None
|
||||
|
||||
# Filter arguments to only include allowed parameters
|
||||
disallowed_params = [
|
||||
param for param in arguments.keys() if param not in allowed_params_list
|
||||
]
|
||||
|
||||
if disallowed_params:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail={
|
||||
"error": f"Parameters {disallowed_params} are not allowed for tool {tool_name}. "
|
||||
f"Allowed parameters: {allowed_params_list}. "
|
||||
f"Contact proxy admin to allow these parameters."
|
||||
},
|
||||
)
|
||||
|
||||
async def check_tool_permission_for_key_team(
|
||||
self,
|
||||
tool_name: str,
|
||||
|
|
@ -621,18 +843,20 @@ class MCPServerManager:
|
|||
Raises:
|
||||
HTTPException: If tool is not allowed for this key/team
|
||||
"""
|
||||
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import MCPRequestHandler
|
||||
|
||||
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
|
||||
MCPRequestHandler,
|
||||
)
|
||||
|
||||
if not user_api_key_auth:
|
||||
return
|
||||
|
||||
|
||||
# Check if tool is allowed
|
||||
is_allowed = await MCPRequestHandler.is_tool_allowed_for_server(
|
||||
tool_name=tool_name,
|
||||
server_id=server.server_id,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
|
||||
|
||||
if not is_allowed:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
|
|
@ -641,6 +865,64 @@ class MCPServerManager:
|
|||
},
|
||||
)
|
||||
|
||||
async def _call_openapi_tool_handler(
|
||||
self,
|
||||
server: MCPServer,
|
||||
tool_name: str,
|
||||
arguments: Dict[str, Any],
|
||||
) -> CallToolResult:
|
||||
"""
|
||||
Call an OpenAPI tool handler directly.
|
||||
|
||||
For OpenAPI servers, instead of using MCP protocol, we call the tool handler
|
||||
that was registered during OpenAPI spec parsing. This handler makes direct
|
||||
HTTP requests to the API.
|
||||
|
||||
Args:
|
||||
tool_name: The full tool name (with prefix) to call
|
||||
arguments: Tool arguments to pass to the handler
|
||||
|
||||
Returns:
|
||||
CallToolResult with the response from the API
|
||||
"""
|
||||
from mcp.types import TextContent
|
||||
|
||||
from litellm.proxy._experimental.mcp_server.tool_registry import (
|
||||
global_mcp_tool_registry,
|
||||
)
|
||||
|
||||
# Get the tool from the registry
|
||||
tool = global_mcp_tool_registry.get_tool(f"{server.name}-{tool_name}")
|
||||
if tool is None:
|
||||
# Tool not found in registry
|
||||
error_msg = f"OpenAPI tool {tool_name} not found in registry"
|
||||
verbose_logger.error(error_msg)
|
||||
return CallToolResult(
|
||||
content=[TextContent(type="text", text=error_msg)],
|
||||
isError=True,
|
||||
)
|
||||
|
||||
try:
|
||||
# Call the tool handler with the arguments
|
||||
# The handler is an async function that makes the HTTP request
|
||||
handler_result = await tool.handler(**arguments)
|
||||
|
||||
# Convert the handler result (string response) to CallToolResult format
|
||||
result = CallToolResult(
|
||||
content=[TextContent(type="text", text=str(handler_result))],
|
||||
isError=False,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Error calling OpenAPI tool {tool_name}: {str(e)}"
|
||||
verbose_logger.error(error_msg)
|
||||
return CallToolResult(
|
||||
content=[TextContent(type="text", text=error_msg)],
|
||||
isError=True,
|
||||
)
|
||||
|
||||
async def pre_call_tool_check(
|
||||
self,
|
||||
name: str,
|
||||
|
|
@ -667,6 +949,13 @@ class MCPServerManager:
|
|||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
|
||||
## filter parameters based on allowed_params configuration
|
||||
self.validate_allowed_params(
|
||||
tool_name=name,
|
||||
arguments=arguments,
|
||||
server=server,
|
||||
)
|
||||
|
||||
pre_hook_kwargs = {
|
||||
"name": name,
|
||||
"arguments": arguments,
|
||||
|
|
@ -793,95 +1082,109 @@ class MCPServerManager:
|
|||
server=mcp_server,
|
||||
)
|
||||
|
||||
# Get server-specific auth header if available
|
||||
server_auth_header: Optional[Union[Dict[str, str], str]] = None
|
||||
if mcp_server_auth_headers and mcp_server.alias:
|
||||
server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
|
||||
elif mcp_server_auth_headers and mcp_server.server_name:
|
||||
server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
|
||||
# Prepare tasks for during hooks
|
||||
tasks = []
|
||||
if proxy_logging_obj:
|
||||
# Create synthetic LLM data for during hook processing
|
||||
from litellm.types.llms.base import HiddenParams
|
||||
from litellm.types.mcp import MCPDuringCallRequestObject
|
||||
|
||||
# Fall back to deprecated mcp_auth_header if no server-specific header found
|
||||
if server_auth_header is None:
|
||||
server_auth_header = mcp_auth_header
|
||||
|
||||
# oauth2 headers
|
||||
extra_headers: Optional[Dict[str, str]] = None
|
||||
if mcp_server.auth_type == MCPAuth.oauth2:
|
||||
extra_headers = oauth2_headers
|
||||
|
||||
if mcp_server.extra_headers and raw_headers:
|
||||
if extra_headers is None:
|
||||
extra_headers = {}
|
||||
for header in mcp_server.extra_headers:
|
||||
if header in raw_headers:
|
||||
extra_headers[header] = raw_headers[header]
|
||||
|
||||
client = self._create_mcp_client(
|
||||
server=mcp_server,
|
||||
mcp_auth_header=server_auth_header,
|
||||
extra_headers=extra_headers,
|
||||
)
|
||||
|
||||
async with client:
|
||||
# Use the original tool name (without prefix) for the actual call
|
||||
call_tool_params = MCPCallToolRequestParams(
|
||||
name=original_tool_name,
|
||||
request_obj = MCPDuringCallRequestObject(
|
||||
tool_name=name,
|
||||
arguments=arguments,
|
||||
server_name=server_name_from_prefix,
|
||||
start_time=start_time.timestamp() if start_time else None,
|
||||
hidden_params=HiddenParams(),
|
||||
)
|
||||
tasks = []
|
||||
if proxy_logging_obj:
|
||||
# Create synthetic LLM data for during hook processing
|
||||
from litellm.types.llms.base import HiddenParams
|
||||
from litellm.types.mcp import MCPDuringCallRequestObject
|
||||
|
||||
request_obj = MCPDuringCallRequestObject(
|
||||
tool_name=name,
|
||||
during_hook_kwargs = {
|
||||
"name": name,
|
||||
"arguments": arguments,
|
||||
"server_name": server_name_from_prefix,
|
||||
"user_api_key_auth": user_api_key_auth,
|
||||
}
|
||||
|
||||
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
|
||||
request_obj, during_hook_kwargs
|
||||
)
|
||||
|
||||
during_hook_task = asyncio.create_task(
|
||||
proxy_logging_obj.during_call_hook(
|
||||
user_api_key_dict=user_api_key_auth,
|
||||
data=synthetic_llm_data,
|
||||
call_type="mcp_call", # type: ignore
|
||||
)
|
||||
)
|
||||
tasks.append(during_hook_task)
|
||||
|
||||
# For OpenAPI servers, call the tool handler directly instead of via MCP client
|
||||
if mcp_server.spec_path:
|
||||
verbose_logger.debug(
|
||||
f"Calling OpenAPI tool {name} directly via HTTP handler"
|
||||
)
|
||||
tasks.append(
|
||||
asyncio.create_task(
|
||||
self._call_openapi_tool_handler(mcp_server, name, arguments)
|
||||
)
|
||||
)
|
||||
else:
|
||||
# For regular MCP servers, use the MCP client
|
||||
# Get server-specific auth header if available
|
||||
server_auth_header: Optional[Union[Dict[str, str], str]] = None
|
||||
if mcp_server_auth_headers and mcp_server.alias:
|
||||
server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
|
||||
elif mcp_server_auth_headers and mcp_server.server_name:
|
||||
server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
|
||||
|
||||
# Fall back to deprecated mcp_auth_header if no server-specific header found
|
||||
if server_auth_header is None:
|
||||
server_auth_header = mcp_auth_header
|
||||
|
||||
# oauth2 headers
|
||||
extra_headers: Optional[Dict[str, str]] = None
|
||||
if mcp_server.auth_type == MCPAuth.oauth2:
|
||||
extra_headers = oauth2_headers
|
||||
|
||||
if mcp_server.extra_headers and raw_headers:
|
||||
if extra_headers is None:
|
||||
extra_headers = {}
|
||||
for header in mcp_server.extra_headers:
|
||||
if header in raw_headers:
|
||||
extra_headers[header] = raw_headers[header]
|
||||
|
||||
client = self._create_mcp_client(
|
||||
server=mcp_server,
|
||||
mcp_auth_header=server_auth_header,
|
||||
extra_headers=extra_headers,
|
||||
)
|
||||
|
||||
async with client:
|
||||
# Use the original tool name (without prefix) for the actual call
|
||||
call_tool_params = MCPCallToolRequestParams(
|
||||
name=original_tool_name,
|
||||
arguments=arguments,
|
||||
server_name=server_name_from_prefix,
|
||||
start_time=start_time.timestamp() if start_time else None,
|
||||
hidden_params=HiddenParams(),
|
||||
)
|
||||
tasks.append(asyncio.create_task(client.call_tool(call_tool_params)))
|
||||
|
||||
during_hook_kwargs = {
|
||||
"name": name,
|
||||
"arguments": arguments,
|
||||
"server_name": server_name_from_prefix,
|
||||
"user_api_key_auth": user_api_key_auth,
|
||||
}
|
||||
try:
|
||||
mcp_responses = await asyncio.gather(*tasks)
|
||||
|
||||
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
|
||||
request_obj, during_hook_kwargs
|
||||
)
|
||||
# If proxy_logging_obj is None, the tool call result is at index 0
|
||||
# If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
|
||||
result_index = 1 if proxy_logging_obj else 0
|
||||
result = mcp_responses[result_index]
|
||||
|
||||
during_hook_task = asyncio.create_task(
|
||||
proxy_logging_obj.during_call_hook(
|
||||
user_api_key_dict=user_api_key_auth,
|
||||
data=synthetic_llm_data,
|
||||
call_type="mcp_call", # type: ignore
|
||||
)
|
||||
)
|
||||
tasks.append(during_hook_task)
|
||||
|
||||
tasks.append(asyncio.create_task(client.call_tool(call_tool_params)))
|
||||
try:
|
||||
mcp_responses = await asyncio.gather(*tasks)
|
||||
|
||||
# If proxy_logging_obj is None, the tool call result is at index 0
|
||||
# If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
|
||||
result_index = 1 if proxy_logging_obj else 0
|
||||
result = mcp_responses[result_index]
|
||||
|
||||
return cast(CallToolResult, result)
|
||||
except (
|
||||
BlockedPiiEntityError,
|
||||
GuardrailRaisedException,
|
||||
HTTPException,
|
||||
) as e:
|
||||
# Re-raise guardrail exceptions to properly fail the MCP call
|
||||
verbose_logger.error(
|
||||
f"Guardrail blocked MCP tool call during result check: {str(e)}"
|
||||
)
|
||||
raise e
|
||||
return cast(CallToolResult, result)
|
||||
except (
|
||||
BlockedPiiEntityError,
|
||||
GuardrailRaisedException,
|
||||
HTTPException,
|
||||
) as e:
|
||||
# Re-raise guardrail exceptions to properly fail the MCP call
|
||||
verbose_logger.error(
|
||||
f"Guardrail blocked MCP tool call during result check: {str(e)}"
|
||||
)
|
||||
raise e
|
||||
|
||||
#########################################################
|
||||
# End of Methods that call the upstream MCP servers
|
||||
|
|
|
|||
|
|
@ -0,0 +1,236 @@
|
|||
"""
|
||||
This module is used to generate MCP tools from OpenAPI specs.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.proxy._experimental.mcp_server.tool_registry import (
|
||||
global_mcp_tool_registry,
|
||||
)
|
||||
|
||||
# Store the base URL and headers globally
|
||||
BASE_URL = ""
|
||||
HEADERS: Dict[str, str] = {}
|
||||
|
||||
|
||||
def load_openapi_spec(filepath: str) -> Dict[str, Any]:
|
||||
"""Load OpenAPI specification from JSON file."""
|
||||
with open(filepath, "r") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def get_base_url(spec: Dict[str, Any]) -> str:
|
||||
"""Extract base URL from OpenAPI spec."""
|
||||
# OpenAPI 3.x
|
||||
if "servers" in spec and spec["servers"]:
|
||||
return spec["servers"][0]["url"]
|
||||
# OpenAPI 2.x (Swagger)
|
||||
elif "host" in spec:
|
||||
scheme = spec.get("schemes", ["https"])[0]
|
||||
base_path = spec.get("basePath", "")
|
||||
return f"{scheme}://{spec['host']}{base_path}"
|
||||
return ""
|
||||
|
||||
|
||||
def extract_parameters(operation: Dict[str, Any]) -> tuple:
|
||||
"""Extract parameter names from OpenAPI operation."""
|
||||
path_params = []
|
||||
query_params = []
|
||||
body_params = []
|
||||
|
||||
# OpenAPI 3.x and 2.x parameters
|
||||
if "parameters" in operation:
|
||||
for param in operation["parameters"]:
|
||||
param_name = param["name"]
|
||||
if param.get("in") == "path":
|
||||
path_params.append(param_name)
|
||||
elif param.get("in") == "query":
|
||||
query_params.append(param_name)
|
||||
elif param.get("in") == "body":
|
||||
body_params.append(param_name)
|
||||
|
||||
# OpenAPI 3.x requestBody
|
||||
if "requestBody" in operation:
|
||||
body_params.append("body")
|
||||
|
||||
return path_params, query_params, body_params
|
||||
|
||||
|
||||
def build_input_schema(operation: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Build MCP input schema from OpenAPI operation."""
|
||||
properties = {}
|
||||
required = []
|
||||
|
||||
# Process parameters
|
||||
if "parameters" in operation:
|
||||
for param in operation["parameters"]:
|
||||
param_name = param["name"]
|
||||
param_schema = param.get("schema", {})
|
||||
param_type = param_schema.get("type", "string")
|
||||
|
||||
properties[param_name] = {
|
||||
"type": param_type,
|
||||
"description": param.get("description", ""),
|
||||
}
|
||||
|
||||
if param.get("required", False):
|
||||
required.append(param_name)
|
||||
|
||||
# Process requestBody (OpenAPI 3.x)
|
||||
if "requestBody" in operation:
|
||||
request_body = operation["requestBody"]
|
||||
content = request_body.get("content", {})
|
||||
|
||||
# Try to get JSON schema
|
||||
if "application/json" in content:
|
||||
schema = content["application/json"].get("schema", {})
|
||||
properties["body"] = {
|
||||
"type": "object",
|
||||
"description": request_body.get("description", "Request body"),
|
||||
"properties": schema.get("properties", {}),
|
||||
}
|
||||
if request_body.get("required", False):
|
||||
required.append("body")
|
||||
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
"required": required if required else [],
|
||||
}
|
||||
|
||||
|
||||
def create_tool_function(
|
||||
path: str,
|
||||
method: str,
|
||||
operation: Dict[str, Any],
|
||||
base_url: str,
|
||||
headers: Optional[Dict[str, str]] = None,
|
||||
):
|
||||
"""Create a tool function for an OpenAPI operation.
|
||||
|
||||
Args:
|
||||
path: API endpoint path
|
||||
method: HTTP method (get, post, put, delete, patch)
|
||||
operation: OpenAPI operation object
|
||||
base_url: Base URL for the API
|
||||
headers: Optional headers to include in requests (e.g., authentication)
|
||||
"""
|
||||
if headers is None:
|
||||
headers = {}
|
||||
|
||||
path_params, query_params, body_params = extract_parameters(operation)
|
||||
all_params = path_params + query_params + body_params
|
||||
|
||||
# Build function signature dynamically
|
||||
if all_params:
|
||||
params_str = ", ".join(f"{p}: str = ''" for p in all_params)
|
||||
else:
|
||||
params_str = ""
|
||||
|
||||
# Create the function code as a string
|
||||
func_code = f'''
|
||||
async def tool_function({params_str}) -> str:
|
||||
"""Dynamically generated tool function."""
|
||||
url = base_url + path
|
||||
|
||||
# Replace path parameters
|
||||
path_param_names = {path_params}
|
||||
for param_name in path_param_names:
|
||||
param_value = locals().get(param_name, "")
|
||||
if param_value:
|
||||
url = url.replace("{{" + param_name + "}}", str(param_value))
|
||||
|
||||
# Build query params
|
||||
query_param_names = {query_params}
|
||||
params = {{}}
|
||||
for param_name in query_param_names:
|
||||
param_value = locals().get(param_name, "")
|
||||
if param_value:
|
||||
params[param_name] = param_value
|
||||
|
||||
# Build request body
|
||||
body_param_names = {body_params}
|
||||
json_body = None
|
||||
if body_param_names:
|
||||
body_value = locals().get("body", {{}})
|
||||
if isinstance(body_value, dict):
|
||||
json_body = body_value
|
||||
elif body_value:
|
||||
# If it's a string, try to parse as JSON
|
||||
import json as json_module
|
||||
try:
|
||||
json_body = json_module.loads(body_value) if isinstance(body_value, str) else {{"data": body_value}}
|
||||
except:
|
||||
json_body = {{"data": body_value}}
|
||||
|
||||
# Make HTTP request
|
||||
async with httpx.AsyncClient() as client:
|
||||
if "{method.lower()}" == "get":
|
||||
response = await client.get(url, params=params, headers=headers)
|
||||
elif "{method.lower()}" == "post":
|
||||
response = await client.post(url, params=params, json=json_body, headers=headers)
|
||||
elif "{method.lower()}" == "put":
|
||||
response = await client.put(url, params=params, json=json_body, headers=headers)
|
||||
elif "{method.lower()}" == "delete":
|
||||
response = await client.delete(url, params=params, headers=headers)
|
||||
elif "{method.lower()}" == "patch":
|
||||
response = await client.patch(url, params=params, json=json_body, headers=headers)
|
||||
else:
|
||||
return "Unsupported HTTP method: {method}"
|
||||
|
||||
return response.text
|
||||
'''
|
||||
|
||||
# Execute the function code to create the actual function
|
||||
local_vars = {
|
||||
"httpx": httpx,
|
||||
"headers": headers,
|
||||
"base_url": base_url,
|
||||
"path": path,
|
||||
"method": method,
|
||||
}
|
||||
exec(func_code, local_vars)
|
||||
|
||||
return local_vars["tool_function"]
|
||||
|
||||
|
||||
def register_tools_from_openapi(spec: Dict[str, Any], base_url: str):
|
||||
"""Register MCP tools from OpenAPI specification."""
|
||||
paths = spec.get("paths", {})
|
||||
|
||||
for path, path_item in paths.items():
|
||||
for method in ["get", "post", "put", "delete", "patch"]:
|
||||
if method in path_item:
|
||||
operation = path_item[method]
|
||||
|
||||
# Generate tool name
|
||||
operation_id = operation.get(
|
||||
"operationId", f"{method}_{path.replace('/', '_')}"
|
||||
)
|
||||
tool_name = operation_id.replace(" ", "_").lower()
|
||||
|
||||
# Get description
|
||||
description = operation.get(
|
||||
"summary", operation.get("description", f"{method.upper()} {path}")
|
||||
)
|
||||
|
||||
# Build input schema
|
||||
input_schema = build_input_schema(operation)
|
||||
|
||||
# Create tool function
|
||||
tool_func = create_tool_function(path, method, operation, base_url)
|
||||
tool_func.__name__ = tool_name
|
||||
tool_func.__doc__ = description
|
||||
|
||||
# Register tool with local registry
|
||||
global_mcp_tool_registry.register_tool(
|
||||
name=tool_name,
|
||||
description=description,
|
||||
input_schema=input_schema,
|
||||
handler=tool_func,
|
||||
)
|
||||
verbose_logger.debug(f"Registered tool: {tool_name}")
|
||||
|
|
@ -364,25 +364,25 @@ if MCP_AVAILABLE:
|
|||
def _tool_name_matches(tool_name: str, filter_list: List[str]) -> bool:
|
||||
"""
|
||||
Check if a tool name matches any name in the filter list.
|
||||
|
||||
|
||||
Checks both the full tool name and unprefixed version (without server prefix).
|
||||
This allows users to configure simple tool names regardless of prefixing.
|
||||
|
||||
|
||||
Args:
|
||||
tool_name: The tool name to check (may be prefixed like "server-tool_name")
|
||||
filter_list: List of tool names to match against
|
||||
|
||||
|
||||
Returns:
|
||||
True if the tool name (prefixed or unprefixed) is in the filter list
|
||||
"""
|
||||
from litellm.proxy._experimental.mcp_server.utils import (
|
||||
get_server_name_prefix_tool_mcp,
|
||||
)
|
||||
|
||||
|
||||
# Check if the full name is in the list
|
||||
if tool_name in filter_list:
|
||||
return True
|
||||
|
||||
|
||||
# Check if the unprefixed name is in the list
|
||||
unprefixed_name, _ = get_server_name_prefix_tool_mcp(tool_name)
|
||||
return unprefixed_name in filter_list
|
||||
|
|
@ -393,34 +393,36 @@ if MCP_AVAILABLE:
|
|||
) -> List[MCPTool]:
|
||||
"""
|
||||
Filter tools by allowed/disallowed tools configuration.
|
||||
|
||||
|
||||
If allowed_tools is set, only tools in that list are returned.
|
||||
If disallowed_tools is set, tools in that list are excluded.
|
||||
Tool names are matched with and without server prefixes for flexibility.
|
||||
|
||||
|
||||
Args:
|
||||
tools: List of tools to filter
|
||||
mcp_server: Server configuration with allowed_tools/disallowed_tools
|
||||
|
||||
|
||||
Returns:
|
||||
Filtered list of tools
|
||||
"""
|
||||
tools_to_return = tools
|
||||
|
||||
|
||||
# Filter by allowed_tools (whitelist)
|
||||
if mcp_server.allowed_tools:
|
||||
tools_to_return = [
|
||||
tool for tool in tools
|
||||
tool
|
||||
for tool in tools
|
||||
if _tool_name_matches(tool.name, mcp_server.allowed_tools)
|
||||
]
|
||||
|
||||
|
||||
# Filter by disallowed_tools (blacklist)
|
||||
if mcp_server.disallowed_tools:
|
||||
tools_to_return = [
|
||||
tool for tool in tools_to_return
|
||||
tool
|
||||
for tool in tools_to_return
|
||||
if not _tool_name_matches(tool.name, mcp_server.disallowed_tools)
|
||||
]
|
||||
|
||||
|
||||
return tools_to_return
|
||||
|
||||
async def _get_tools_from_mcp_servers(
|
||||
|
|
@ -497,17 +499,17 @@ if MCP_AVAILABLE:
|
|||
extra_headers=extra_headers,
|
||||
add_prefix=add_prefix,
|
||||
)
|
||||
|
||||
|
||||
filtered_tools = filter_tools_by_allowed_tools(tools, server)
|
||||
|
||||
|
||||
filtered_tools = await filter_tools_by_key_team_permissions(
|
||||
tools=filtered_tools,
|
||||
server_id=server_id,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
|
||||
|
||||
all_tools.extend(filtered_tools)
|
||||
|
||||
|
||||
verbose_logger.debug(
|
||||
f"Successfully fetched {len(tools)} tools from server {server.name}, {len(filtered_tools)} after filtering"
|
||||
)
|
||||
|
|
@ -520,6 +522,7 @@ if MCP_AVAILABLE:
|
|||
verbose_logger.info(
|
||||
f"Successfully fetched {len(all_tools)} tools total from all MCP servers"
|
||||
)
|
||||
|
||||
return all_tools
|
||||
|
||||
async def filter_tools_by_key_team_permissions(
|
||||
|
|
@ -527,18 +530,31 @@ if MCP_AVAILABLE:
|
|||
server_id: str,
|
||||
user_api_key_auth: Optional[UserAPIKeyAuth],
|
||||
) -> List[MCPTool]:
|
||||
"""Filter tools based on key/team mcp_tool_permissions."""
|
||||
"""
|
||||
Filter tools based on key/team mcp_tool_permissions.
|
||||
|
||||
Note: Tool names in the DB are stored without server prefixes,
|
||||
but tool names from MCP servers are prefixed. We need to strip
|
||||
the prefix before comparing.
|
||||
"""
|
||||
# Filter by key/team tool-level permissions
|
||||
allowed_tool_names = await MCPRequestHandler.get_allowed_tools_for_server(
|
||||
server_id=server_id,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
if allowed_tool_names is not None:
|
||||
filtered_tools = [t for t in tools if t.name in allowed_tool_names]
|
||||
# Strip prefix from tool names before comparing
|
||||
# Tools are stored in DB without prefix, but come from MCP server with prefix
|
||||
filtered_tools = []
|
||||
for t in tools:
|
||||
# Get tool name without server prefix
|
||||
unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(t.name)
|
||||
if unprefixed_tool_name in allowed_tool_names:
|
||||
filtered_tools.append(t)
|
||||
else:
|
||||
# No restrictions, return all tools
|
||||
filtered_tools = tools
|
||||
|
||||
|
||||
return filtered_tools
|
||||
|
||||
async def _list_mcp_tools(
|
||||
|
|
@ -583,30 +599,7 @@ if MCP_AVAILABLE:
|
|||
)
|
||||
# Continue with empty managed tools list instead of failing completely
|
||||
|
||||
# Get tools from local registry
|
||||
local_tools = []
|
||||
try:
|
||||
local_tools_raw = global_mcp_tool_registry.list_tools()
|
||||
|
||||
# Convert local tools to MCPTool format
|
||||
for tool in local_tools_raw:
|
||||
# Convert from litellm.types.mcp_server.tool_registry.MCPTool to mcp.types.Tool
|
||||
mcp_tool = MCPTool(
|
||||
name=tool.name,
|
||||
description=tool.description,
|
||||
inputSchema=tool.input_schema,
|
||||
)
|
||||
local_tools.append(mcp_tool)
|
||||
except Exception as e:
|
||||
verbose_logger.exception(
|
||||
f"Error getting tools from local registry: {str(e)}"
|
||||
)
|
||||
# Continue with empty local tools list instead of failing completely
|
||||
|
||||
# Combine all tools
|
||||
all_tools = managed_tools + local_tools
|
||||
|
||||
return all_tools
|
||||
return managed_tools
|
||||
|
||||
@client
|
||||
async def call_mcp_tool(
|
||||
|
|
@ -667,33 +660,42 @@ if MCP_AVAILABLE:
|
|||
standard_logging_mcp_tool_call
|
||||
)
|
||||
litellm_logging_obj.model = f"MCP: {name}"
|
||||
# Try managed server tool first (pass the full prefixed name)
|
||||
# Primary and recommended way to use MCP servers
|
||||
# Check if tool exists in local registry first (for OpenAPI-based tools)
|
||||
# These tools are registered with their prefixed names
|
||||
#########################################################
|
||||
mcp_server: Optional[MCPServer] = (
|
||||
global_mcp_server_manager._get_mcp_server_from_tool_name(name)
|
||||
)
|
||||
if mcp_server:
|
||||
standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
|
||||
mcp_server.mcp_info or {}
|
||||
).get("mcp_server_cost_info")
|
||||
response = await _handle_managed_mcp_tool(
|
||||
name=name, # Pass the full name (potentially prefixed)
|
||||
arguments=arguments,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
mcp_auth_header=mcp_auth_header,
|
||||
mcp_server_auth_headers=mcp_server_auth_headers,
|
||||
oauth2_headers=oauth2_headers,
|
||||
raw_headers=raw_headers,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
)
|
||||
local_tool = global_mcp_tool_registry.get_tool(name)
|
||||
if local_tool:
|
||||
verbose_logger.debug(f"Executing local registry tool: {name}")
|
||||
response = await _handle_local_mcp_tool(name, arguments)
|
||||
|
||||
# Fall back to local tool registry (use original name)
|
||||
#########################################################
|
||||
# Deprecated: Local MCP Server Tool
|
||||
# Try managed MCP server tool (pass the full prefixed name)
|
||||
# Primary and recommended way to use external MCP servers
|
||||
#########################################################
|
||||
else:
|
||||
response = await _handle_local_mcp_tool(original_tool_name, arguments)
|
||||
mcp_server: Optional[MCPServer] = (
|
||||
global_mcp_server_manager._get_mcp_server_from_tool_name(name)
|
||||
)
|
||||
if mcp_server:
|
||||
standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
|
||||
mcp_server.mcp_info or {}
|
||||
).get("mcp_server_cost_info")
|
||||
response = await _handle_managed_mcp_tool(
|
||||
name=name, # Pass the full name (potentially prefixed)
|
||||
arguments=arguments,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
mcp_auth_header=mcp_auth_header,
|
||||
mcp_server_auth_headers=mcp_server_auth_headers,
|
||||
oauth2_headers=oauth2_headers,
|
||||
raw_headers=raw_headers,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
)
|
||||
|
||||
# Fall back to local tool registry with original name (legacy support)
|
||||
#########################################################
|
||||
# Deprecated: Local MCP Server Tool
|
||||
#########################################################
|
||||
else:
|
||||
response = await _handle_local_mcp_tool(original_tool_name, arguments)
|
||||
|
||||
#########################################################
|
||||
# Post MCP Tool Call Hook
|
||||
|
|
@ -765,14 +767,21 @@ if MCP_AVAILABLE:
|
|||
Handle tool execution for local registry tools
|
||||
Note: Local tools don't use prefixes, so we use the original name
|
||||
"""
|
||||
import inspect
|
||||
|
||||
tool = global_mcp_tool_registry.get_tool(name)
|
||||
if not tool:
|
||||
raise HTTPException(status_code=404, detail=f"Tool '{name}' not found")
|
||||
|
||||
try:
|
||||
result = tool.handler(**arguments)
|
||||
# Check if handler is async or sync
|
||||
if inspect.iscoroutinefunction(tool.handler):
|
||||
result = await tool.handler(**arguments)
|
||||
else:
|
||||
result = tool.handler(**arguments)
|
||||
return [TextContent(text=str(result), type="text")]
|
||||
except Exception as e:
|
||||
verbose_logger.exception(f"Error executing local tool {name}: {str(e)}")
|
||||
return [TextContent(text=f"Error: {str(e)}", type="text")]
|
||||
|
||||
def _get_mcp_servers_in_path(path: str) -> Optional[List[str]]:
|
||||
|
|
@ -893,6 +902,7 @@ if MCP_AVAILABLE:
|
|||
|
||||
await session_manager.handle_request(scope, receive, send)
|
||||
except Exception as e:
|
||||
raise e
|
||||
verbose_logger.exception(f"Error handling MCP request: {e}")
|
||||
# Instead of re-raising, try to send a graceful error response
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
import json
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
from mcp.types import Tool as MCPToolSDKTool
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.proxy.types_utils.utils import get_instance_fn
|
||||
from litellm.types.mcp_server.tool_registry import MCPTool
|
||||
|
|
@ -39,12 +41,30 @@ class MCPToolRegistry:
|
|||
"""
|
||||
return self.tools.get(name)
|
||||
|
||||
def list_tools(self) -> List[MCPTool]:
|
||||
def list_tools(self, tool_prefix: Optional[str] = None) -> List[MCPTool]:
|
||||
"""
|
||||
List all registered tools
|
||||
"""
|
||||
if tool_prefix:
|
||||
return [
|
||||
tool
|
||||
for tool in self.tools.values()
|
||||
if tool.name.startswith(tool_prefix)
|
||||
]
|
||||
return list(self.tools.values())
|
||||
|
||||
def convert_tools_to_mcp_sdk_tool_type(
|
||||
self, tools: List[MCPTool]
|
||||
) -> List[MCPToolSDKTool]:
|
||||
return [
|
||||
MCPToolSDKTool(
|
||||
name=tool.name,
|
||||
description=tool.description,
|
||||
inputSchema=tool.input_schema,
|
||||
)
|
||||
for tool in tools
|
||||
]
|
||||
|
||||
def load_tools_from_config(
|
||||
self, mcp_tools_config: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
self.__BUILD_MANIFEST={__rewrites:{afterFiles:[],beforeFiles:[],fallback:[]},"/_error":["static/chunks/pages/_error-28b803cb2479b966.js"],sortedPages:["/_app","/_error"]},self.__BUILD_MANIFEST_CB&&self.__BUILD_MANIFEST_CB();
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1 @@
|
|||
"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[3665],{84566:function(e,t,s){s.d(t,{GH$:function(){return l}});var c=s(2265);let l=({color:e="currentColor",size:t=24,className:s,...l})=>c.createElement("svg",{viewBox:"0 0 24 24",xmlns:"http://www.w3.org/2000/svg",width:t,height:t,fill:e,...l,className:"remixicon "+(s||"")},c.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM11.0026 16L6.75999 11.7574L8.17421 10.3431L11.0026 13.1716L16.6595 7.51472L18.0737 8.92893L11.0026 16Z"}))}}]);
|
||||
|
|
@ -1 +0,0 @@
|
|||
"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[665],{84566:function(e,t,s){s.d(t,{GH$:function(){return l}});var c=s(2265);let l=({color:e="currentColor",size:t=24,className:s,...l})=>c.createElement("svg",{viewBox:"0 0 24 24",xmlns:"http://www.w3.org/2000/svg",width:t,height:t,fill:e,...l,className:"remixicon "+(s||"")},c.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM11.0026 16L6.75999 11.7574L8.17421 10.3431L11.0026 13.1716L16.6595 7.51472L18.0737 8.92893L11.0026 16Z"}))}}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue