mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-09 22:31:41 +00:00
Merge branch 'BerriAI:main' into patch-2
This commit is contained in:
SamAcctX
GitHub
commit
cecffd335a
346 changed files with 12210 additions and 2316 deletions
19
.github/workflows/ghcr_deploy.yml
vendored
19
.github/workflows/ghcr_deploy.yml
vendored
|
|
@ -338,7 +338,9 @@ jobs:
|
|||
if [ -z "${CHART_LIST}" ]; then
|
||||
echo "current-version=0.1.0" | tee -a $GITHUB_OUTPUT
|
||||
else
|
||||
printf '%s' "${CHART_LIST}" | grep '^version:' | awk 'BEGIN{FS=":"}{print "current-version="$2}' | tr -d " " | tee -a $GITHUB_OUTPUT
|
||||
# Extract version and strip any prerelease suffix (e.g., 0.1.827-latest -> 0.1.827)
|
||||
VERSION=$(printf '%s' "${CHART_LIST}" | grep '^version:' | awk 'BEGIN{FS=":"}{print $2}' | tr -d " " | cut -d'-' -f1)
|
||||
echo "current-version=${VERSION}" | tee -a $GITHUB_OUTPUT
|
||||
fi
|
||||
env:
|
||||
HELM_EXPERIMENTAL_OCI: '1'
|
||||
|
|
@ -351,11 +353,24 @@ jobs:
|
|||
current-version: ${{ steps.current_version.outputs.current-version || '0.1.0' }}
|
||||
version-fragment: 'bug'
|
||||
|
||||
# Add suffix for non-stable releases (semantic versioning)
|
||||
- name: Calculate chart version with prerelease suffix
|
||||
id: chart_version
|
||||
shell: bash
|
||||
run: |
|
||||
BASE_VERSION="${{ steps.bump_version.outputs.next-version || '0.1.0' }}"
|
||||
RELEASE_TYPE="${{ github.event.inputs.release_type }}"
|
||||
if [ "$RELEASE_TYPE" = "stable" ]; then
|
||||
echo "version=${BASE_VERSION}" | tee -a $GITHUB_OUTPUT
|
||||
else
|
||||
echo "version=${BASE_VERSION}-${RELEASE_TYPE}" | tee -a $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- uses: ./.github/actions/helm-oci-chart-releaser
|
||||
with:
|
||||
name: ${{ env.CHART_NAME }}
|
||||
repository: ${{ env.REPO_OWNER }}
|
||||
tag: ${{ github.event.inputs.chartVersion || steps.bump_version.outputs.next-version || '0.1.0' }}
|
||||
tag: ${{ github.event.inputs.chartVersion || steps.chart_version.outputs.version || '0.1.0' }}
|
||||
app_version: ${{ steps.current_app_tag.outputs.latest_tag }}
|
||||
path: deploy/charts/${{ env.CHART_NAME }}
|
||||
registry: ${{ env.REGISTRY }}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,279 @@
|
|||
# Braintrust Prompt Wrapper for LiteLLM
|
||||
|
||||
This directory contains a wrapper server that enables LiteLLM to use prompts from [Braintrust](https://www.braintrust.dev/) through the generic prompt management API.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────────────┐ ┌─────────────┐
|
||||
│ LiteLLM │ ──────> │ Wrapper Server │ ──────> │ Braintrust │
|
||||
│ Client │ │ (This Server) │ │ API │
|
||||
└─────────────┘ └──────────────────────┘ └─────────────┘
|
||||
Uses generic Transforms Stores actual
|
||||
prompt manager Braintrust format prompt templates
|
||||
to LiteLLM format
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Generic Prompt Manager (`litellm/integrations/generic_prompt_management/`)
|
||||
|
||||
A generic client that can work with any API implementing the `/beta/litellm_prompt_management` endpoint.
|
||||
|
||||
**Expected API Response Format:**
|
||||
```json
|
||||
{
|
||||
"prompt_id": "string",
|
||||
"prompt_template": [
|
||||
{"role": "system", "content": "You are a helpful assistant"},
|
||||
{"role": "user", "content": "Hello {name}"}
|
||||
],
|
||||
"prompt_template_model": "gpt-4",
|
||||
"prompt_template_optional_params": {
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Braintrust Wrapper Server (`braintrust_prompt_wrapper_server.py`)
|
||||
|
||||
A FastAPI server that:
|
||||
- Implements the `/beta/litellm_prompt_management` endpoint
|
||||
- Fetches prompts from Braintrust API
|
||||
- Transforms Braintrust response format to LiteLLM format
|
||||
|
||||
## Setup
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install fastapi uvicorn httpx litellm
|
||||
```
|
||||
|
||||
### Set Environment Variables
|
||||
|
||||
```bash
|
||||
export BRAINTRUST_API_KEY="your-braintrust-api-key"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Step 1: Start the Wrapper Server
|
||||
|
||||
```bash
|
||||
python braintrust_prompt_wrapper_server.py
|
||||
```
|
||||
|
||||
The server will start on `http://localhost:8080` by default.
|
||||
|
||||
You can customize the port and host:
|
||||
```bash
|
||||
export PORT=8000
|
||||
export HOST=0.0.0.0
|
||||
python braintrust_prompt_wrapper_server.py
|
||||
```
|
||||
|
||||
### Step 2: Use with LiteLLM
|
||||
|
||||
```python
|
||||
import litellm
|
||||
from litellm.integrations.generic_prompt_management import GenericPromptManager
|
||||
|
||||
# Configure the generic prompt manager to use your wrapper server
|
||||
generic_config = {
|
||||
"api_base": "http://localhost:8080",
|
||||
"api_key": "your-braintrust-api-key", # Will be passed to Braintrust
|
||||
"timeout": 30,
|
||||
}
|
||||
|
||||
# Create the prompt manager
|
||||
prompt_manager = GenericPromptManager(**generic_config)
|
||||
|
||||
# Use with completion
|
||||
response = litellm.completion(
|
||||
model="generic_prompt/gpt-4",
|
||||
prompt_id="your-braintrust-prompt-id",
|
||||
prompt_variables={"name": "World"}, # Variables to substitute
|
||||
messages=[{"role": "user", "content": "Additional message"}]
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
### Step 3: Direct API Testing
|
||||
|
||||
You can also test the wrapper API directly:
|
||||
|
||||
```bash
|
||||
# Test with curl
|
||||
curl -H "Authorization: Bearer YOUR_BRAINTRUST_TOKEN" \
|
||||
"http://localhost:8080/beta/litellm_prompt_management?prompt_id=YOUR_PROMPT_ID"
|
||||
|
||||
# Health check
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Service info
|
||||
curl http://localhost:8080/
|
||||
```
|
||||
|
||||
## API Documentation
|
||||
|
||||
Once the server is running, visit:
|
||||
- Swagger UI: `http://localhost:8080/docs`
|
||||
- ReDoc: `http://localhost:8080/redoc`
|
||||
|
||||
## Braintrust Format Transformation
|
||||
|
||||
The wrapper automatically transforms Braintrust's response format:
|
||||
|
||||
**Braintrust API Response:**
|
||||
```json
|
||||
{
|
||||
"id": "prompt-123",
|
||||
"prompt_data": {
|
||||
"prompt": {
|
||||
"type": "chat",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a helpful assistant"
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"model": "gpt-4",
|
||||
"params": {
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Transformed to LiteLLM Format:**
|
||||
```json
|
||||
{
|
||||
"prompt_id": "prompt-123",
|
||||
"prompt_template": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a helpful assistant"
|
||||
}
|
||||
],
|
||||
"prompt_template_model": "gpt-4",
|
||||
"prompt_template_optional_params": {
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Supported Parameters
|
||||
|
||||
The wrapper automatically maps these Braintrust parameters to LiteLLM:
|
||||
|
||||
- `temperature`
|
||||
- `max_tokens` / `max_completion_tokens`
|
||||
- `top_p`
|
||||
- `frequency_penalty`
|
||||
- `presence_penalty`
|
||||
- `n`
|
||||
- `stop`
|
||||
- `response_format`
|
||||
- `tool_choice`
|
||||
- `function_call`
|
||||
- `tools`
|
||||
|
||||
## Variable Substitution
|
||||
|
||||
The generic prompt manager supports simple variable substitution:
|
||||
|
||||
```python
|
||||
# In your Braintrust prompt:
|
||||
# "Hello {name}, welcome to {place}!"
|
||||
|
||||
# In your code:
|
||||
prompt_variables = {
|
||||
"name": "Alice",
|
||||
"place": "Wonderland"
|
||||
}
|
||||
|
||||
# Result:
|
||||
# "Hello Alice, welcome to Wonderland!"
|
||||
```
|
||||
|
||||
Supports both `{variable}` and `{{variable}}` syntax.
|
||||
|
||||
## Error Handling
|
||||
|
||||
The wrapper provides detailed error messages:
|
||||
|
||||
- **401**: Missing or invalid Braintrust API token
|
||||
- **404**: Prompt not found in Braintrust
|
||||
- **502**: Failed to connect to Braintrust API
|
||||
- **500**: Error transforming response
|
||||
|
||||
## Production Deployment
|
||||
|
||||
For production use:
|
||||
|
||||
1. **Use HTTPS**: Deploy behind a reverse proxy with SSL
|
||||
2. **Authentication**: Add authentication to the wrapper endpoint if needed
|
||||
3. **Rate Limiting**: Implement rate limiting to prevent abuse
|
||||
4. **Caching**: Consider caching prompt responses
|
||||
5. **Monitoring**: Add logging and monitoring
|
||||
|
||||
Example with Docker:
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN pip install fastapi uvicorn httpx
|
||||
|
||||
COPY braintrust_prompt_wrapper_server.py .
|
||||
|
||||
ENV PORT=8080
|
||||
ENV HOST=0.0.0.0
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
CMD ["python", "braintrust_prompt_wrapper_server.py"]
|
||||
```
|
||||
|
||||
## Extending to Other Providers
|
||||
|
||||
This pattern can be used with any prompt management provider:
|
||||
|
||||
1. Create a wrapper server that implements `/beta/litellm_prompt_management`
|
||||
2. Transform the provider's response to LiteLLM format
|
||||
3. Use the generic prompt manager to connect
|
||||
|
||||
Example providers:
|
||||
- Langsmith
|
||||
- PromptLayer
|
||||
- Humanloop
|
||||
- Custom internal systems
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No Braintrust API token provided"
|
||||
- Set `BRAINTRUST_API_KEY` environment variable
|
||||
- Or pass token in `Authorization: Bearer TOKEN` header
|
||||
|
||||
### "Failed to connect to Braintrust API"
|
||||
- Check your internet connection
|
||||
- Verify Braintrust API is accessible
|
||||
- Check firewall settings
|
||||
|
||||
### "Prompt not found"
|
||||
- Verify the prompt ID exists in Braintrust
|
||||
- Check that your API token has access to the prompt
|
||||
|
||||
## License
|
||||
|
||||
This wrapper is part of the LiteLLM project and follows the same license.
|
||||
|
||||
|
|
@ -0,0 +1,274 @@
|
|||
"""
|
||||
Mock server that implements the /beta/litellm_prompt_management endpoint
|
||||
and acts as a wrapper for calling the Braintrust API.
|
||||
|
||||
This server transforms Braintrust's prompt API response into the format
|
||||
expected by LiteLLM's generic prompt management client.
|
||||
|
||||
Usage:
|
||||
python braintrust_prompt_wrapper_server.py
|
||||
|
||||
# Then test with:
|
||||
curl -H "Authorization: Bearer YOUR_BRAINTRUST_TOKEN" \
|
||||
"http://localhost:8080/beta/litellm_prompt_management?prompt_id=YOUR_PROMPT_ID"
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI, HTTPException, Header, Query
|
||||
from fastapi.responses import JSONResponse
|
||||
import uvicorn
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Braintrust Prompt Wrapper",
|
||||
description="Wrapper server for Braintrust prompts to work with LiteLLM",
|
||||
version="1.0.0",
|
||||
)
|
||||
|
||||
|
||||
def transform_braintrust_message(message: Dict[str, Any]) -> Dict[str, str]:
|
||||
"""
|
||||
Transform a Braintrust message to LiteLLM format.
|
||||
|
||||
Braintrust message format:
|
||||
{
|
||||
"role": "system",
|
||||
"content": "...",
|
||||
"name": "..." (optional)
|
||||
}
|
||||
|
||||
LiteLLM format:
|
||||
{
|
||||
"role": "system",
|
||||
"content": "..."
|
||||
}
|
||||
"""
|
||||
result = {
|
||||
"role": message.get("role", "user"),
|
||||
"content": message.get("content", ""),
|
||||
}
|
||||
|
||||
# Include name if present
|
||||
if "name" in message:
|
||||
result["name"] = message["name"]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def transform_braintrust_response(
|
||||
braintrust_response: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Transform Braintrust API response to LiteLLM prompt management format.
|
||||
|
||||
Braintrust response format:
|
||||
{
|
||||
"objects": [{
|
||||
"id": "prompt_id",
|
||||
"prompt_data": {
|
||||
"prompt": {
|
||||
"type": "chat",
|
||||
"messages": [...],
|
||||
"tools": "..."
|
||||
},
|
||||
"options": {
|
||||
"model": "gpt-4",
|
||||
"params": {
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100,
|
||||
...
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
|
||||
LiteLLM format:
|
||||
{
|
||||
"prompt_id": "prompt_id",
|
||||
"prompt_template": [...],
|
||||
"prompt_template_model": "gpt-4",
|
||||
"prompt_template_optional_params": {...}
|
||||
}
|
||||
"""
|
||||
# Extract the first object from the objects array if it exists
|
||||
if "objects" in braintrust_response and len(braintrust_response["objects"]) > 0:
|
||||
prompt_object = braintrust_response["objects"][0]
|
||||
else:
|
||||
prompt_object = braintrust_response
|
||||
|
||||
prompt_data = prompt_object.get("prompt_data", {})
|
||||
prompt_info = prompt_data.get("prompt", {})
|
||||
options = prompt_data.get("options", {})
|
||||
|
||||
# Extract messages
|
||||
messages = prompt_info.get("messages", [])
|
||||
transformed_messages = [transform_braintrust_message(msg) for msg in messages]
|
||||
|
||||
# Extract model
|
||||
model = options.get("model")
|
||||
|
||||
# Extract optional parameters
|
||||
params = options.get("params", {})
|
||||
optional_params: Dict[str, Any] = {}
|
||||
|
||||
# Map common parameters
|
||||
param_mapping = {
|
||||
"temperature": "temperature",
|
||||
"max_tokens": "max_tokens",
|
||||
"max_completion_tokens": "max_tokens", # Alternative name
|
||||
"top_p": "top_p",
|
||||
"frequency_penalty": "frequency_penalty",
|
||||
"presence_penalty": "presence_penalty",
|
||||
"n": "n",
|
||||
"stop": "stop",
|
||||
}
|
||||
|
||||
for braintrust_param, litellm_param in param_mapping.items():
|
||||
if braintrust_param in params:
|
||||
value = params[braintrust_param]
|
||||
if value is not None:
|
||||
optional_params[litellm_param] = value
|
||||
|
||||
# Handle response_format
|
||||
if "response_format" in params:
|
||||
optional_params["response_format"] = params["response_format"]
|
||||
|
||||
# Handle tool_choice
|
||||
if "tool_choice" in params:
|
||||
optional_params["tool_choice"] = params["tool_choice"]
|
||||
|
||||
# Handle function_call
|
||||
if "function_call" in params:
|
||||
optional_params["function_call"] = params["function_call"]
|
||||
|
||||
# Add tools if present
|
||||
if "tools" in prompt_info and prompt_info["tools"]:
|
||||
optional_params["tools"] = prompt_info["tools"]
|
||||
|
||||
# Handle tool_functions from prompt_data
|
||||
if "tool_functions" in prompt_data and prompt_data["tool_functions"]:
|
||||
optional_params["tool_functions"] = prompt_data["tool_functions"]
|
||||
|
||||
return {
|
||||
"prompt_id": prompt_object.get("id"),
|
||||
"prompt_template": transformed_messages,
|
||||
"prompt_template_model": model,
|
||||
"prompt_template_optional_params": optional_params if optional_params else None,
|
||||
}
|
||||
|
||||
|
||||
@app.get("/beta/litellm_prompt_management")
|
||||
async def get_prompt(
|
||||
prompt_id: str = Query(..., description="The Braintrust prompt ID to fetch"),
|
||||
authorization: Optional[str] = Header(
|
||||
None, description="Bearer token for Braintrust API"
|
||||
),
|
||||
) -> JSONResponse:
|
||||
"""
|
||||
Fetch a prompt from Braintrust and transform it to LiteLLM format.
|
||||
|
||||
Args:
|
||||
prompt_id: The Braintrust prompt ID
|
||||
authorization: Bearer token for Braintrust API (from header)
|
||||
|
||||
Returns:
|
||||
JSONResponse with the transformed prompt data
|
||||
"""
|
||||
# Extract token from Authorization header or environment
|
||||
braintrust_token = None
|
||||
if authorization and authorization.startswith("Bearer "):
|
||||
braintrust_token = authorization.replace("Bearer ", "")
|
||||
else:
|
||||
braintrust_token = os.getenv("BRAINTRUST_API_KEY")
|
||||
|
||||
if not braintrust_token:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="No Braintrust API token provided. Pass via Authorization header or set BRAINTRUST_API_KEY environment variable.",
|
||||
)
|
||||
|
||||
# Call Braintrust API
|
||||
braintrust_url = f"https://api.braintrust.dev/v1/prompt/{prompt_id}"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {braintrust_token}",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
print(f"headers: {headers}")
|
||||
print(f"braintrust_url: {braintrust_url}")
|
||||
print(f"braintrust_token: {braintrust_token}")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
response = await client.get(braintrust_url, headers=headers)
|
||||
response.raise_for_status()
|
||||
braintrust_data = response.json()
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise HTTPException(
|
||||
status_code=e.response.status_code,
|
||||
detail=f"Braintrust API error: {e.response.text}",
|
||||
)
|
||||
except httpx.RequestError as e:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to connect to Braintrust API: {str(e)}",
|
||||
)
|
||||
except json.JSONDecodeError as e:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to parse Braintrust API response: {str(e)}",
|
||||
)
|
||||
|
||||
print(f"braintrust_data: {braintrust_data}")
|
||||
# Transform the response
|
||||
try:
|
||||
transformed_data = transform_braintrust_response(braintrust_data)
|
||||
print(f"transformed_data: {transformed_data}")
|
||||
return JSONResponse(content=transformed_data)
|
||||
except Exception as e:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Failed to transform Braintrust response: {str(e)}",
|
||||
)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
"""Health check endpoint."""
|
||||
return {"status": "healthy", "service": "braintrust-prompt-wrapper"}
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def root():
|
||||
"""Root endpoint with service information."""
|
||||
return {
|
||||
"service": "Braintrust Prompt Wrapper for LiteLLM",
|
||||
"version": "1.0.0",
|
||||
"endpoints": {
|
||||
"prompt_management": "/beta/litellm_prompt_management?prompt_id=<id>",
|
||||
"health": "/health",
|
||||
},
|
||||
"documentation": "/docs",
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
"""Run the server."""
|
||||
port = int(os.getenv("PORT", "8080"))
|
||||
host = os.getenv("HOST", "0.0.0.0")
|
||||
|
||||
print(f"🚀 Starting Braintrust Prompt Wrapper Server on {host}:{port}")
|
||||
print(f"📚 API Documentation available at http://{host}:{port}/docs")
|
||||
print(
|
||||
f"🔑 Make sure to set BRAINTRUST_API_KEY environment variable or pass token in Authorization header"
|
||||
)
|
||||
|
||||
uvicorn.run(app, host=host, port=port)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
115
docs/my-website/docs/a2a_cost_tracking.md
Normal file
115
docs/my-website/docs/a2a_cost_tracking.md
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
# A2A Agent Cost Tracking
|
||||
|
||||
LiteLLM supports adding custom cost tracking for A2A agents. You can configure:
|
||||
|
||||
- **Flat cost per query** - A fixed cost charged for each agent request
|
||||
- **Cost by input/output tokens** - Variable cost based on token usage
|
||||
|
||||
This allows you to track and attribute costs for agent usage across your organization, making it easy to see how much each team or project is spending on agent calls.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Navigate to Agents
|
||||
|
||||
From the sidebar, click on "Agents" to open the agent management page.
|
||||
|
||||
|
||||
|
||||
### 2. Create a New Agent
|
||||
|
||||
Click "+ Add New Agent" to open the creation form. You'll need to provide a few basic details:
|
||||
|
||||
- **Agent Name** - A unique identifier for your agent (used in API calls)
|
||||
- **Display Name** - A human-readable name shown in the UI
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 3. Configure Cost Settings
|
||||
|
||||
Scroll down and click on "Cost Configuration" to expand the cost settings panel. This is where you define how much to charge for agent usage.
|
||||
|
||||
|
||||
|
||||
### 4. Set Cost Per Query
|
||||
|
||||
Enter the cost per query amount (in dollars). For example, entering `0.05` means each request to this agent will be charged $0.05.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 5. Create the Agent
|
||||
|
||||
Once you've configured everything, click "Create Agent" to save. Your agent is now ready to use with cost tracking enabled.
|
||||
|
||||
|
||||
|
||||
## Testing Cost Tracking
|
||||
|
||||
Let's verify that cost tracking is working by sending a test request through the Playground.
|
||||
|
||||
### 1. Go to Playground
|
||||
|
||||
Click "Playground" in the sidebar to open the interactive testing interface.
|
||||
|
||||
|
||||
|
||||
### 2. Select A2A Endpoint
|
||||
|
||||
By default, the Playground uses the chat completions endpoint. To test your agent, click "Endpoint Type" and select `/v1/a2a/message/send` from the dropdown.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 3. Select Your Agent
|
||||
|
||||
Now pick the agent you just created from the agent dropdown. You should see it listed by its display name.
|
||||
|
||||
|
||||
|
||||
### 4. Send a Test Message
|
||||
|
||||
Type a message and hit send. You can use the suggested prompts or write your own.
|
||||
|
||||
|
||||
|
||||
Once the agent responds, the request is logged with the cost you configured.
|
||||
|
||||
|
||||
|
||||
## Viewing Cost in Logs
|
||||
|
||||
Now let's confirm the cost was actually tracked.
|
||||
|
||||
### 1. Navigate to Logs
|
||||
|
||||
Click "Logs" in the sidebar to see all recent requests.
|
||||
|
||||
|
||||
|
||||
### 2. View Cost Attribution
|
||||
|
||||
Find your agent request in the list. You'll see the cost column showing the amount you configured. This cost is now attributed to the API key that made the request, so you can track spend per team or project.
|
||||
|
||||
|
||||
|
||||
## Cost Configuration Options
|
||||
|
||||
You can mix and match these options depending on your pricing model:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| **Cost Per Query ($)** | Fixed cost charged for each agent request |
|
||||
| **Input Cost Per Token ($)** | Cost per input token processed |
|
||||
| **Output Cost Per Token ($)** | Cost per output token generated |
|
||||
|
||||
For most use cases, a flat cost per query is simplest. Use token-based pricing if your agent costs vary significantly based on input/output length.
|
||||
|
||||
## Related
|
||||
|
||||
- [A2A Agent Gateway](./a2a.md)
|
||||
- [Spend Tracking](./proxy/cost_tracking.md)
|
||||
|
||||
|
|
@ -174,11 +174,11 @@ def completion(
|
|||
|
||||
- `seed`: *integer or null (optional)* - This feature is in Beta. If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the `system_fingerprint` response parameter to monitor changes in the backend.
|
||||
|
||||
- `tools`: *array (optional)* - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for.
|
||||
- `tools`: *array (optional)* - A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for.
|
||||
|
||||
- `type`: *string* - The type of the tool. Currently, only function is supported.
|
||||
- `type`: *string* - The type of the tool. You can set this to `"function"` or `"mcp"` (matching the `/responses` schema) to call LiteLLM-registered MCP servers directly from `/chat/completions`.
|
||||
|
||||
- `function`: *object* - Required.
|
||||
- `function`: *object* - Required for function tools.
|
||||
|
||||
- `tool_choice`: *string or object (optional)* - Controls which (if any) function is called by the model. none means the model will not call a function and instead generates a message. auto means the model can pick between generating a message or calling a function. Specifying a particular function via `{"type": "function", "function": {"name": "my_function"}}` forces the model to call that function.
|
||||
|
||||
|
|
@ -247,4 +247,3 @@ def completion(
|
|||
- `eos_token`: *string (optional)* - Initial string applied at the end of a sequence
|
||||
|
||||
- `hf_model_name`: *string (optional)* - [Sagemaker Only] The corresponding huggingface name of the model, used to pull the right chat template for the model.
|
||||
|
||||
|
|
|
|||
|
|
@ -1137,6 +1137,37 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
|
|||
}'
|
||||
```
|
||||
|
||||
## Use MCP tools with `/chat/completions`
|
||||
|
||||
:::tip Works with all providers
|
||||
This flow is **provider-agnostic**: the same MCP tool definition works for _every_ LLM backend behind LiteLLM (OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Vertex, self-hosted deployments, etc.).
|
||||
:::
|
||||
|
||||
LiteLLM Proxy also supports MCP-aware tooling on the classic `/v1/chat/completions` endpoint. Provide the MCP tool definition directly in the `tools` array and LiteLLM will fetch and transform the MCP server's tools into OpenAI-compatible function calls. When `require_approval` is set to `"never"`, the proxy automatically executes the returned tool calls and feeds the results back into the model before returning the assistant response.
|
||||
|
||||
```bash title="Chat Completions with MCP Tools" showLineNumbers
|
||||
curl --location '<your-litellm-proxy-base-url>/v1/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer $LITELLM_API_KEY" \
|
||||
--data '{
|
||||
"model": "gpt-4o-mini",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Summarize the latest open PR."}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_url": "litellm_proxy/mcp/github",
|
||||
"server_label": "github_mcp",
|
||||
"require_approval": "never"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
If you omit `require_approval` or set it to any value other than `"never"`, the MCP tool calls are returned to the client so that you can review and execute them manually, matching the upstream OpenAI behavior.
|
||||
|
||||
|
||||
## LiteLLM Proxy - Walk through MCP Gateway
|
||||
LiteLLM exposes an MCP Gateway for admins to add all their MCP servers to LiteLLM. The key benefits of using LiteLLM Proxy with MCP are:
|
||||
|
||||
|
|
|
|||
|
|
@ -159,3 +159,150 @@ print(completion.choices[0].message)
|
|||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Azure Blob Storage Integration
|
||||
|
||||
LiteLLM supports using Azure Blob Storage as a target storage backend for Gemini file uploads. This allows you to store files in Azure Data Lake Storage Gen2 instead of Google's managed storage.
|
||||
|
||||
### Step 1: Setup Azure Blob Storage
|
||||
|
||||
Configure your Azure Blob Storage account by setting the following environment variables:
|
||||
|
||||
**Required Environment Variables:**
|
||||
- `AZURE_STORAGE_ACCOUNT_NAME` - Your Azure Storage account name
|
||||
- `AZURE_STORAGE_FILE_SYSTEM` - The container/filesystem name where files will be stored
|
||||
- `AZURE_STORAGE_ACCOUNT_KEY` - Your account key
|
||||
|
||||
### Step 2: Pass Azure Blob Storage as Target Storage
|
||||
|
||||
When uploading files, specify `target_storage: "azure_storage"` to use Azure Blob Storage instead of the default storage.
|
||||
|
||||
**Supported File Types:**
|
||||
|
||||
Azure Blob Storage supports all Gemini-compatible file types:
|
||||
|
||||
- **Images**: PNG, JPEG, WEBP
|
||||
- **Audio**: AAC, FLAC, MP3, MPA, MPEG, MPGA, OPUS, PCM, WAV, WEBM
|
||||
- **Video**: FLV, MOV, MPEG, MPEGPS, MPG, MP4, WEBM, WMV, 3GPP
|
||||
- **Documents**: PDF, TXT
|
||||
|
||||
> **Note:** Only small files can be sent as inline data because the total request size limit is 20 MB.
|
||||
|
||||
|
||||
### Step 3: Upload Files with Azure Blob Storage for Gemini
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="proxy" label="PROXY">
|
||||
|
||||
1. Setup config.yaml
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: "gemini-2.5-flash"
|
||||
litellm_params:
|
||||
model: gemini/gemini-2.5-flash
|
||||
api_key: os.environ/GEMINI_API_KEY
|
||||
```
|
||||
|
||||
2. Set environment variables
|
||||
|
||||
```bash
|
||||
export AZURE_STORAGE_ACCOUNT_NAME="your-storage-account"
|
||||
export AZURE_STORAGE_FILE_SYSTEM="your-container-name"
|
||||
export AZURE_STORAGE_ACCOUNT_KEY="your-account-key"
|
||||
```
|
||||
or add them in your `.env`
|
||||
|
||||
3. Start proxy
|
||||
|
||||
```bash
|
||||
litellm --config config.yaml
|
||||
```
|
||||
|
||||
4. Upload file with Azure Blob Storage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://0.0.0.0:4000",
|
||||
api_key="sk-1234"
|
||||
)
|
||||
|
||||
# Upload file to Azure Blob Storage
|
||||
file = client.files.create(
|
||||
file=open("document.pdf", "rb"),
|
||||
purpose="user_data",
|
||||
extra_body={
|
||||
"target_model_names": "gemini-2.0-flash",
|
||||
"target_storage": "azure_storage" # 👈 Use Azure Blob Storage
|
||||
}
|
||||
)
|
||||
|
||||
print(f"File uploaded to Azure Blob Storage: {file.id}")
|
||||
|
||||
# Use the file with Gemini
|
||||
completion = client.chat.completions.create(
|
||||
model="gemini-2.0-flash",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Summarize this document"},
|
||||
{
|
||||
"type": "file",
|
||||
"file": {
|
||||
"file_id": file.id,
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
print(completion.choices[0].message.content)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="curl" label="cURL">
|
||||
|
||||
```bash
|
||||
# Upload file with Azure Blob Storage
|
||||
curl -X POST "http://0.0.0.0:4000/v1/files" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-F "file=@document.pdf" \
|
||||
-F "purpose=user_data" \
|
||||
-F "target_storage=azure_storage" \
|
||||
-F "target_model_names=gemini-2.0-flash" \
|
||||
-F "custom_llm_provider=gemini"
|
||||
|
||||
# Use the file with Gemini
|
||||
curl -X POST "http://0.0.0.0:4000/v1/chat/completions" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "gemini-2.0-flash",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Summarize this document"},
|
||||
{
|
||||
"type": "file",
|
||||
"file": {
|
||||
"file_id": "file-id-from-upload",
|
||||
"format": "application/pdf"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
:::info
|
||||
Files uploaded to Azure Blob Storage are stored in your Azure account and can be accessed via the returned file ID. The file URL format is: `https://{account}.blob.core.windows.net/{container}/{path}`
|
||||
:::
|
||||
|
||||
|
|
|
|||
|
|
@ -233,8 +233,65 @@ curl -s --request POST \
|
|||
|
||||
|
||||
|
||||
## LiteLLM A2A Gateway
|
||||
|
||||
You can also connect to LangGraph agents through LiteLLM's A2A (Agent-to-Agent) Gateway UI. This provides a visual way to register and test agents without writing code.
|
||||
|
||||
### 1. Navigate to Agents
|
||||
|
||||
From the sidebar, click "Agents" to open the agent management page, then click "+ Add New Agent".
|
||||
|
||||
|
||||
|
||||
### 2. Select LangGraph Agent Type
|
||||
|
||||
Click "A2A Standard" to see available agent types, then search for "langgraph" and select "Connect to LangGraph agents via the LangGraph Platform API".
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 3. Configure the Agent
|
||||
|
||||
Fill in the following fields:
|
||||
|
||||
- **Agent Name** - A unique identifier (e.g., `lan-agent`)
|
||||
- **LangGraph API Base** - Your LangGraph server URL, typically `http://127.0.0.1:2024/`
|
||||
- **API Key** - Optional. LangGraph doesn't require an API key by default
|
||||
- **Assistant ID** - Not used by LangGraph, you can enter any string here
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Click "Create Agent" to save.
|
||||
|
||||
|
||||
|
||||
### 4. Test in Playground
|
||||
|
||||
Go to "Playground" in the sidebar to test your agent. Change the endpoint type to `/v1/a2a/message/send`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 5. Select Your Agent and Send a Message
|
||||
|
||||
Pick your LangGraph agent from the dropdown and send a test message.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
The agent responds with its capabilities. You can now interact with your LangGraph agent through the A2A protocol.
|
||||
|
||||
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [LangGraph Platform Documentation](https://langchain-ai.github.io/langgraph/cloud/quick_start/)
|
||||
- [LangGraph GitHub](https://github.com/langchain-ai/langgraph)
|
||||
- [A2A Agent Gateway](../a2a.md)
|
||||
- [A2A Cost Tracking](../a2a_cost_tracking.md)
|
||||
|
||||
|
|
|
|||
|
|
@ -291,12 +291,265 @@ Give the key access to the virtual index and the embedding model.
|
|||
|
||||
### Developer Flow
|
||||
|
||||
#### MilvusRESTClient
|
||||
|
||||
To use the passthrough API, you need a simple REST client. Copy this `milvus_rest_client.py` file to your project:
|
||||
|
||||
<details>
|
||||
<summary>Click to expand milvus_rest_client.py</summary>
|
||||
|
||||
```python
|
||||
"""
|
||||
Simple Milvus REST API v2 Client
|
||||
Based on: https://milvus.io/api-reference/restful/v2.6.x/
|
||||
"""
|
||||
|
||||
import requests
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
|
||||
class DataType:
|
||||
"""Milvus data types"""
|
||||
|
||||
INT64 = "Int64"
|
||||
FLOAT_VECTOR = "FloatVector"
|
||||
VARCHAR = "VarChar"
|
||||
BOOL = "Bool"
|
||||
FLOAT = "Float"
|
||||
|
||||
|
||||
class CollectionSchema:
|
||||
"""Collection schema builder"""
|
||||
|
||||
def __init__(self):
|
||||
self.fields = []
|
||||
|
||||
def add_field(
|
||||
self,
|
||||
field_name: str,
|
||||
data_type: str,
|
||||
is_primary: bool = False,
|
||||
dim: Optional[int] = None,
|
||||
description: str = "",
|
||||
):
|
||||
"""Add a field to the schema"""
|
||||
field = {
|
||||
"fieldName": field_name,
|
||||
"dataType": data_type,
|
||||
"isPrimary": is_primary,
|
||||
"description": description,
|
||||
}
|
||||
if data_type == DataType.FLOAT_VECTOR and dim:
|
||||
field["elementTypeParams"] = {"dim": str(dim)}
|
||||
self.fields.append(field)
|
||||
return self
|
||||
|
||||
def to_dict(self):
|
||||
"""Convert schema to dict for API"""
|
||||
return {"fields": self.fields}
|
||||
|
||||
|
||||
class IndexParams:
|
||||
"""Index parameters builder"""
|
||||
|
||||
def __init__(self):
|
||||
self.indexes = []
|
||||
|
||||
def add_index(
|
||||
self, field_name: str, metric_type: str = "L2", index_name: Optional[str] = None
|
||||
):
|
||||
"""Add an index"""
|
||||
index = {
|
||||
"fieldName": field_name,
|
||||
"indexName": index_name or f"{field_name}_index",
|
||||
"metricType": metric_type,
|
||||
}
|
||||
self.indexes.append(index)
|
||||
return self
|
||||
|
||||
def to_list(self):
|
||||
"""Convert to list for API"""
|
||||
return self.indexes
|
||||
|
||||
|
||||
class MilvusRESTClient:
|
||||
"""
|
||||
Simple Milvus REST API v2 Client
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/
|
||||
"""
|
||||
|
||||
def __init__(self, uri: str, token: str, db_name: str = "default"):
|
||||
"""
|
||||
Initialize Milvus REST client
|
||||
|
||||
Args:
|
||||
uri: Milvus server URI (e.g., http://localhost:19530)
|
||||
token: Authentication token
|
||||
db_name: Database name
|
||||
"""
|
||||
self.base_url = uri.rstrip("/")
|
||||
self.token = token
|
||||
self.db_name = db_name
|
||||
self.headers = {
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
def _make_request(self, endpoint: str, data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Make a POST request to Milvus API"""
|
||||
url = f"{self.base_url}{endpoint}"
|
||||
|
||||
# Add dbName if not already in data and not default
|
||||
if "dbName" not in data and self.db_name != "default":
|
||||
data["dbName"] = self.db_name
|
||||
|
||||
try:
|
||||
response = requests.post(url, json=data, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
except requests.exceptions.HTTPError as e:
|
||||
print(f"e.response.text: {e.response.content}")
|
||||
raise e
|
||||
|
||||
result = response.json()
|
||||
|
||||
# Check for API errors
|
||||
if result.get("code") != 0:
|
||||
raise Exception(
|
||||
f"Milvus API Error: {result.get('message', 'Unknown error')}"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def has_collection(self, collection_name: str) -> bool:
|
||||
"""
|
||||
Check if a collection exists
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Collection%20(v2)/Has.md
|
||||
"""
|
||||
try:
|
||||
result = self._make_request(
|
||||
"/v2/vectordb/collections/has", {"collectionName": collection_name}
|
||||
)
|
||||
return result.get("data", {}).get("has", False)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def drop_collection(self, collection_name: str):
|
||||
"""
|
||||
Drop a collection
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Collection%20(v2)/Drop.md
|
||||
"""
|
||||
return self._make_request(
|
||||
"/v2/vectordb/collections/drop", {"collectionName": collection_name}
|
||||
)
|
||||
|
||||
def create_schema(self) -> CollectionSchema:
|
||||
"""Create a new collection schema"""
|
||||
return CollectionSchema()
|
||||
|
||||
def prepare_index_params(self) -> IndexParams:
|
||||
"""Create index parameters"""
|
||||
return IndexParams()
|
||||
|
||||
def create_collection(
|
||||
self,
|
||||
collection_name: str,
|
||||
schema: CollectionSchema,
|
||||
index_params: Optional[IndexParams] = None,
|
||||
):
|
||||
"""
|
||||
Create a collection
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Collection%20(v2)/Create.md
|
||||
"""
|
||||
data = {"collectionName": collection_name, "schema": schema.to_dict()}
|
||||
|
||||
if index_params:
|
||||
data["indexParams"] = index_params.to_list()
|
||||
|
||||
return self._make_request("/v2/vectordb/collections/create", data)
|
||||
|
||||
def describe_collection(self, collection_name: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Describe a collection
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Collection%20(v2)/Describe.md
|
||||
"""
|
||||
result = self._make_request(
|
||||
"/v2/vectordb/collections/describe", {"collectionName": collection_name}
|
||||
)
|
||||
return result.get("data", {})
|
||||
|
||||
def insert(
|
||||
self,
|
||||
collection_name: str,
|
||||
data: List[Dict[str, Any]],
|
||||
partition_name: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
Insert data into a collection
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Vector%20(v2)/Insert.md
|
||||
"""
|
||||
payload = {"collectionName": collection_name, "data": data}
|
||||
|
||||
if partition_name:
|
||||
payload["partitionName"] = partition_name
|
||||
|
||||
result = self._make_request("/v2/vectordb/entities/insert", payload)
|
||||
return result.get("data", {})
|
||||
|
||||
def flush(self, collection_name: str):
|
||||
"""
|
||||
Flush collection data to storage
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Collection%20(v2)/Flush.md
|
||||
"""
|
||||
return self._make_request(
|
||||
"/v2/vectordb/collections/flush", {"collectionName": collection_name}
|
||||
)
|
||||
|
||||
def search(
|
||||
self,
|
||||
collection_name: str,
|
||||
data: List[List[float]],
|
||||
anns_field: str,
|
||||
limit: int = 10,
|
||||
search_params: Optional[Dict[str, Any]] = None,
|
||||
output_fields: Optional[List[str]] = None,
|
||||
) -> List[List[Dict]]:
|
||||
"""
|
||||
Search for vectors
|
||||
|
||||
Reference: https://milvus.io/api-reference/restful/v2.6.x/v2/Vector%20(v2)/Search.md
|
||||
"""
|
||||
payload = {
|
||||
"collectionName": collection_name,
|
||||
"data": data,
|
||||
"annsField": anns_field,
|
||||
"limit": limit,
|
||||
}
|
||||
|
||||
if search_params:
|
||||
payload["searchParams"] = search_params
|
||||
|
||||
if output_fields:
|
||||
payload["outputFields"] = output_fields
|
||||
|
||||
result = self._make_request("/v2/vectordb/entities/search", payload)
|
||||
return result.get("data", [])
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
#### 1. Create a collection with schema
|
||||
|
||||
Note: Use the `/milvus` endpoint for the passthrough api that uses the `milvus` provider in your config.
|
||||
|
||||
```python
|
||||
from milvus_rest_client import MilvusRESTClient, DataType
|
||||
from milvus_rest_client import MilvusRESTClient, DataType # Use the client from above
|
||||
import random
|
||||
import time
|
||||
|
||||
|
|
@ -404,7 +657,7 @@ for i in range(5):
|
|||
Here's a full working example:
|
||||
|
||||
```python
|
||||
from milvus_rest_client import MilvusRESTClient, DataType
|
||||
from milvus_rest_client import MilvusRESTClient, DataType # Use the client from above
|
||||
import random
|
||||
import time
|
||||
|
||||
|
|
|
|||
|
|
@ -433,7 +433,7 @@ Expected Response:
|
|||
|
||||
### Advanced: Using `reasoning_effort` with `summary` field
|
||||
|
||||
By default, `reasoning_effort` accepts a string value (`"none"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`—`"xhigh"` is only supported on `gpt-5.1-codex-max`) and only sets the effort level without including a reasoning summary.
|
||||
By default, `reasoning_effort` accepts a string value (`"none"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`—`"xhigh"` is only supported on `gpt-5.1-codex-max` and `gpt-5.2` models) and only sets the effort level without including a reasoning summary.
|
||||
|
||||
To opt-in to the `summary` feature, you can pass `reasoning_effort` as a dictionary. **Note:** The `summary` field requires your OpenAI organization to have verification status. Using `summary` without verification will result in a 400 error from OpenAI.
|
||||
|
||||
|
|
@ -501,11 +501,13 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
|
|||
| `gpt-5.1-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) |
|
||||
| `gpt-5.1-codex-mini` | `adaptive` | `low`, `medium`, `high` (no `minimal`) |
|
||||
| `gpt-5.1-codex-max` | `adaptive` | `low`, `medium`, `high`, `xhigh` (no `minimal`) |
|
||||
| `gpt-5.2` | `medium` | `none`, `low`, `medium`, `high`, `xhigh` |
|
||||
| `gpt-5.2-pro` | `high` | `low`, `medium`, `high`, `xhigh` |
|
||||
| `gpt-5-pro` | `high` | `high` only |
|
||||
|
||||
**Note:**
|
||||
- GPT-5.1 introduced a new `reasoning_effort="none"` setting for faster, lower-latency responses. This replaces the `"minimal"` setting from GPT-5.
|
||||
- `gpt-5.1-codex-max` is the only model that supports `reasoning_effort="xhigh"`. All other models will reject this value.
|
||||
- `gpt-5.1-codex-max` and `gpt-5.2` models support `reasoning_effort="xhigh"`. All other models will reject this value.
|
||||
- `gpt-5-pro` only accepts `reasoning_effort="high"`. Other values will return an error.
|
||||
- When `reasoning_effort` is not set (None), OpenAI defaults to the value shown in the "Default" column.
|
||||
|
||||
|
|
|
|||
|
|
@ -233,7 +233,7 @@ curl -X POST "http://localhost:4000/v1/chat/completions" \
|
|||
}'
|
||||
```
|
||||
|
||||
This provides clear, explicit conversation tracking that works seamlessly with LiteLLM's session management.
|
||||
This provides clear, explicit conversation tracking that works seamlessly with LiteLLM's session management. When using monitor mode, the session ID is returned in the `x-pillar-session-id` response header for easy correlation and tracking.
|
||||
|
||||
### Actions on Flagged Content
|
||||
|
||||
|
|
@ -251,6 +251,73 @@ Logs the violation but allows the request to proceed:
|
|||
on_flagged_action: "monitor"
|
||||
```
|
||||
|
||||
**Response Headers:**
|
||||
|
||||
You can opt in to receiving detection details in response headers by configuring `include_scanners: true` and/or `include_evidence: true`. When enabled, these headers are included for **every request**—not just flagged ones—enabling comprehensive metrics, false positive analysis, and threat investigation.
|
||||
|
||||
- **`x-pillar-flagged`**: Boolean string indicating Pillar's blocking recommendation (`"true"` or `"false"`)
|
||||
- **`x-pillar-scanners`**: URL-encoded JSON object showing scanner categories (e.g., `%7B%22jailbreak%22%3Atrue%7D`) — requires `include_scanners: true`
|
||||
- **`x-pillar-evidence`**: URL-encoded JSON array of detection evidence (may contain items even when `flagged` is `false`) — requires `include_evidence: true`
|
||||
- **`x-pillar-session-id`**: URL-encoded session ID for correlation and investigation
|
||||
|
||||
:::info Understanding `flagged` vs Scanner Results
|
||||
The `flagged` field is Pillar's **policy-level blocking recommendation**, which may differ from individual scanner results:
|
||||
|
||||
- **`flagged: true`** → Pillar recommends blocking based on your configured policies
|
||||
- **`flagged: false`** → Pillar does not recommend blocking, but individual scanners may still detect content
|
||||
|
||||
For example, the `toxic_language` scanner might detect profanity (`scanners.toxic_language: true`) while `flagged` remains `false` if your Pillar policy doesn't block on toxic language alone. This allows you to:
|
||||
- Monitor threats without blocking users
|
||||
- Build metrics on detection rates vs block rates
|
||||
- Analyze false positive rates by comparing scanner results to user feedback
|
||||
:::
|
||||
|
||||
The `x-pillar-scanners`, `x-pillar-evidence`, and `x-pillar-session-id` headers use URL encoding (percent-encoding) to convert JSON data into an ASCII-safe format. This is necessary because HTTP headers only support ISO-8859-1 characters and cannot contain raw JSON special characters (`{`, `"`, `:`) or Unicode text. To read these headers, first URL-decode the value, then parse it as JSON.
|
||||
|
||||
LiteLLM truncates the `x-pillar-evidence` header to a maximum of 8 KB per header to avoid proxy limits. Note that most proxies and servers also enforce a total header size limit of approximately 32 KB across all headers combined. When truncation occurs, each affected evidence item includes an `"evidence_truncated": true` flag and the metadata contains `pillar_evidence_truncated: true`.
|
||||
|
||||
**Example Response Headers (URL-encoded):**
|
||||
```http
|
||||
x-pillar-flagged: true
|
||||
x-pillar-session-id: abc-123-def-456
|
||||
x-pillar-scanners: %7B%22jailbreak%22%3Atrue%2C%22prompt_injection%22%3Afalse%2C%22toxic_language%22%3Afalse%7D
|
||||
x-pillar-evidence: %5B%7B%22category%22%3A%22prompt_injection%22%2C%22evidence%22%3A%22Ignore%20previous%20instructions%22%7D%5D
|
||||
```
|
||||
|
||||
**After Decoding:**
|
||||
```json
|
||||
// x-pillar-scanners
|
||||
{"jailbreak": true, "prompt_injection": false, "toxic_language": false}
|
||||
|
||||
// x-pillar-evidence
|
||||
[{"category": "prompt_injection", "evidence": "Ignore previous instructions"}]
|
||||
```
|
||||
|
||||
**Decoding Example (Python):**
|
||||
|
||||
```python
|
||||
from urllib.parse import unquote
|
||||
import json
|
||||
|
||||
# Step 1: URL-decode the header value (converts %7B to {, %22 to ", etc.)
|
||||
# Step 2: Parse the resulting JSON string
|
||||
scanners = json.loads(unquote(response.headers["x-pillar-scanners"]))
|
||||
evidence = json.loads(unquote(response.headers["x-pillar-evidence"]))
|
||||
|
||||
# Session ID is a plain string, so only URL-decode is needed (no JSON parsing)
|
||||
session_id = unquote(response.headers["x-pillar-session-id"])
|
||||
```
|
||||
|
||||
:::tip
|
||||
LiteLLM mirrors the encoded values onto `metadata["pillar_response_headers"]` so you can inspect exactly what was returned. When truncation occurs, it sets `metadata["pillar_evidence_truncated"]` to `true` and marks affected evidence items with `"evidence_truncated": true`. Evidence text is shortened with a `...[truncated]` suffix, and entire evidence entries may be removed if necessary to stay under the 8 KB header limit. Check these flags to determine if full evidence details are available in your logs.
|
||||
:::
|
||||
|
||||
This allows your application to:
|
||||
- Track threats without blocking legitimate users
|
||||
- Implement custom handling logic based on threat types
|
||||
- Build analytics and alerting on security events
|
||||
- Correlate threats across requests using session IDs
|
||||
|
||||
### Resilience and Error Handling
|
||||
|
||||
#### Graceful Degradation (`fallback_on_error`)
|
||||
|
|
@ -544,6 +611,79 @@ curl -X POST "http://localhost:4000/v1/chat/completions" \
|
|||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="monitor" label="Monitor Mode with Headers">
|
||||
|
||||
**Monitor mode request with scanner detection:**
|
||||
|
||||
```bash
|
||||
# Test with content that triggers scanner detection
|
||||
curl -v -X POST "http://localhost:4000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer YOUR_LITELLM_PROXY_MASTER_KEY" \
|
||||
-d '{
|
||||
"model": "gpt-4.1-mini",
|
||||
"messages": [{"role": "user", "content": "how do I rob a bank?"}],
|
||||
"max_tokens": 50
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected response (Allowed with headers):**
|
||||
|
||||
The request succeeds and returns the LLM response. Headers are included for **all requests** when `include_scanners` and `include_evidence` are enabled—even when `flagged` is `false`:
|
||||
|
||||
```http
|
||||
HTTP/1.1 200 OK
|
||||
x-litellm-applied-guardrails: pillar-monitor-everything,pillar-monitor-everything
|
||||
x-pillar-flagged: false
|
||||
x-pillar-scanners: %7B%22jailbreak%22%3Afalse%2C%22safety%22%3Atrue%2C%22prompt_injection%22%3Afalse%2C%22pii%22%3Afalse%2C%22secret%22%3Afalse%2C%22toxic_language%22%3Afalse%7D
|
||||
x-pillar-evidence: %5B%7B%22category%22%3A%22safety%22%2C%22type%22%3A%22non_violent_crimes%22%2C%22end_idx%22%3A20%2C%22evidence%22%3A%22how%20do%20I%20rob%20a%20bank%3F%22%2C%22metadata%22%3A%7B%22start_idx%22%3A0%2C%22end_idx%22%3A20%7D%7D%5D
|
||||
x-pillar-session-id: d9433f86-b428-4ee7-93ee-e97a53f8a180
|
||||
```
|
||||
|
||||
Notice that `x-pillar-flagged: false` but `safety: true` in the scanners. This is because `flagged` represents Pillar's policy-level blocking recommendation, while individual scanners report their own detections.
|
||||
|
||||
```python
|
||||
from urllib.parse import unquote
|
||||
import json
|
||||
|
||||
scanners = json.loads(unquote(response.headers["x-pillar-scanners"]))
|
||||
evidence = json.loads(unquote(response.headers["x-pillar-evidence"]))
|
||||
session_id = unquote(response.headers["x-pillar-session-id"])
|
||||
flagged = response.headers["x-pillar-flagged"] == "true"
|
||||
|
||||
# Scanner detected safety issue, but policy didn't flag for blocking
|
||||
print(f"Flagged for blocking: {flagged}") # False
|
||||
print(f"Safety issue detected: {scanners.get('safety')}") # True
|
||||
print(f"Evidence: {evidence}")
|
||||
# [{'category': 'safety', 'type': 'non_violent_crimes', 'evidence': 'how do I rob a bank?', ...}]
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "chatcmpl-xyz123",
|
||||
"object": "chat.completion",
|
||||
"model": "gpt-4.1-mini",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "I'm sorry, but I can't assist with that request."
|
||||
},
|
||||
"finish_reason": "stop"
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 14,
|
||||
"completion_tokens": 11,
|
||||
"total_tokens": 25
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** In monitor mode, scanner results and evidence are included in response headers for every request, allowing you to build metrics and analyze detection patterns. The `flagged` field indicates whether Pillar's policy recommends blocking—your application can use the detailed scanner data for custom alerting, analytics, or false positive analysis.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="secrets" label="Secrets">
|
||||
|
||||
|
|
|
|||
|
|
@ -49,6 +49,16 @@ http://localhost:4000/metrics
|
|||
# <proxy_base_url>/metrics
|
||||
```
|
||||
|
||||
### Multiple Workers
|
||||
|
||||
When using LiteLLM with multiple workers, you need to set the `PROMETHEUS_MULTIPROC_DIR` environment variable to enable aggregated metric collection across worker processes.
|
||||
|
||||
```shell
|
||||
export PROMETHEUS_MULTIPROC_DIR="/prometheus_multiproc"
|
||||
```
|
||||
|
||||
This directory is used by the Prometheus client library to store metric files that can be shared across multiple worker processes. Make sure the directory exists and is writable by your LiteLLM process.
|
||||
|
||||
## Virtual Keys, Teams, Internal Users
|
||||
|
||||
Use this for for tracking per [user, key, team, etc.](virtual_keys)
|
||||
|
|
|
|||
|
|
@ -247,6 +247,26 @@ OIDC Auth for API: [**See Walkthrough**](https://www.loom.com/share/00fe2deab59a
|
|||
- Validate if any group has model access
|
||||
- If all checks pass, allow the request
|
||||
|
||||
### Select Team via Request Header
|
||||
|
||||
When a JWT token contains multiple teams (via `team_ids_jwt_field`), you can explicitly select which team to use for a request by passing the `x-litellm-team-id` header.
|
||||
|
||||
```bash
|
||||
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Authorization: Bearer <your-jwt-token>' \
|
||||
-H 'x-litellm-team-id: team_id_2' \
|
||||
-d '{
|
||||
"model": "gpt-4",
|
||||
"messages": [{"role": "user", "content": "Hello"}]
|
||||
}'
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- The team ID in the header must exist in the JWT's `team_ids_jwt_field` list or match `team_id_jwt_field`
|
||||
- If an invalid team is specified, a 403 error is returned
|
||||
- If no header is provided, LiteLLM auto-selects the first team with access to the requested model
|
||||
|
||||
|
||||
### Custom JWT Validate
|
||||
|
||||
|
|
|
|||
|
|
@ -411,7 +411,13 @@ const sidebars = {
|
|||
label: "/a2a - A2A Agent Gateway",
|
||||
items: [
|
||||
"a2a",
|
||||
"a2a_cost_tracking",
|
||||
"a2a_agent_permissions",
|
||||
{
|
||||
type: "link",
|
||||
label: "Adding LangGraph Agents",
|
||||
href: "/docs/providers/langgraph#litellm-a2a-gateway",
|
||||
},
|
||||
],
|
||||
},
|
||||
"assistants",
|
||||
|
|
|
|||
|
|
@ -22,7 +22,6 @@ from litellm.proxy._types import (
|
|||
)
|
||||
from litellm.proxy.openai_files_endpoints.common_utils import (
|
||||
_is_base64_encoded_unified_file_id,
|
||||
convert_b64_uid_to_unified_uid,
|
||||
get_batch_id_from_unified_batch_id,
|
||||
get_model_id_from_unified_batch_id,
|
||||
)
|
||||
|
|
@ -42,6 +41,10 @@ from litellm.types.utils import (
|
|||
LLMResponseTypes,
|
||||
SpecialEnums,
|
||||
)
|
||||
from litellm.proxy.openai_files_endpoints.common_utils import (
|
||||
get_content_type_from_file_object,
|
||||
normalize_mime_type_for_provider,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.types.llms.openai import HttpxBinaryResponseContent
|
||||
|
|
@ -108,6 +111,17 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
|||
|
||||
if file_object is not None:
|
||||
db_data["file_object"] = file_object.model_dump_json()
|
||||
# Extract storage metadata from hidden params if present
|
||||
hidden_params = getattr(file_object, "_hidden_params", {}) or {}
|
||||
if "storage_backend" in hidden_params:
|
||||
db_data["storage_backend"] = hidden_params["storage_backend"]
|
||||
if "storage_url" in hidden_params:
|
||||
db_data["storage_url"] = hidden_params["storage_url"]
|
||||
|
||||
verbose_logger.debug(
|
||||
f"Storage metadata: storage_backend={db_data.get('storage_backend')}, "
|
||||
f"storage_url={db_data.get('storage_url')}"
|
||||
)
|
||||
|
||||
result = await self.prisma_client.db.litellm_managedfiletable.create(
|
||||
data=db_data
|
||||
|
|
@ -268,7 +282,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
|||
)
|
||||
return False
|
||||
|
||||
async def async_pre_call_hook(
|
||||
async def async_pre_call_hook( # noqa: PLR0915
|
||||
self,
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
cache: DualCache,
|
||||
|
|
@ -287,15 +301,31 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
|||
await self.check_managed_file_id_access(data, user_api_key_dict)
|
||||
|
||||
### HANDLE TRANSFORMATIONS ###
|
||||
if call_type == CallTypes.completion.value:
|
||||
# Check both completion and acompletion call types
|
||||
is_completion_call = (
|
||||
call_type == CallTypes.completion.value
|
||||
or call_type == CallTypes.acompletion.value
|
||||
)
|
||||
|
||||
if is_completion_call:
|
||||
messages = data.get("messages")
|
||||
model = data.get("model", "")
|
||||
if messages:
|
||||
file_ids = self.get_file_ids_from_messages(messages)
|
||||
if file_ids:
|
||||
# Check if any files are stored in storage backends and need base64 conversion
|
||||
# This is needed for Vertex AI/Gemini which requires base64 content
|
||||
is_vertex_ai = model and ("vertex_ai" in model or "gemini" in model.lower())
|
||||
if is_vertex_ai:
|
||||
await self._convert_storage_files_to_base64(
|
||||
messages=messages,
|
||||
file_ids=file_ids,
|
||||
litellm_parent_otel_span=user_api_key_dict.parent_otel_span,
|
||||
)
|
||||
|
||||
model_file_id_mapping = await self.get_model_file_id_mapping(
|
||||
file_ids, user_api_key_dict.parent_otel_span
|
||||
)
|
||||
|
||||
data["model_file_id_mapping"] = model_file_id_mapping
|
||||
elif call_type == CallTypes.aresponses.value or call_type == CallTypes.responses.value:
|
||||
# Handle managed files in responses API input
|
||||
|
|
@ -865,3 +895,124 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
|
|||
)
|
||||
else:
|
||||
raise Exception(f"LiteLLM Managed File object with id={file_id} not found")
|
||||
|
||||
async def _convert_storage_files_to_base64(
|
||||
self,
|
||||
messages: List[AllMessageValues],
|
||||
file_ids: List[str],
|
||||
litellm_parent_otel_span: Optional[Span],
|
||||
) -> None:
|
||||
"""
|
||||
Convert files stored in storage backends to base64 format for Vertex AI/Gemini.
|
||||
|
||||
This method checks if any managed files are stored in storage backends,
|
||||
downloads them, and converts them to base64 format in the messages.
|
||||
"""
|
||||
# Check each file_id to see if it's stored in a storage backend
|
||||
for file_id in file_ids:
|
||||
# Check if this is a base64 encoded unified file ID
|
||||
decoded_unified_file_id = _is_base64_encoded_unified_file_id(file_id)
|
||||
|
||||
if not decoded_unified_file_id:
|
||||
continue
|
||||
|
||||
# Check database for storage backend info
|
||||
# IMPORTANT: The database stores the base64 encoded unified_file_id (not the decoded version)
|
||||
# So we query with the original file_id (which is base64 encoded)
|
||||
db_file = await self.prisma_client.db.litellm_managedfiletable.find_first(
|
||||
where={"unified_file_id": file_id}
|
||||
)
|
||||
|
||||
if not db_file or not db_file.storage_backend or not db_file.storage_url:
|
||||
continue
|
||||
|
||||
# File is stored in a storage backend, download and convert to base64
|
||||
try:
|
||||
from litellm.llms.base_llm.files.storage_backend_factory import get_storage_backend
|
||||
|
||||
storage_backend_name = db_file.storage_backend
|
||||
storage_url = db_file.storage_url
|
||||
|
||||
# Get storage backend (uses same env vars as callback)
|
||||
try:
|
||||
storage_backend = get_storage_backend(storage_backend_name)
|
||||
except ValueError as e:
|
||||
verbose_logger.warning(
|
||||
f"Storage backend '{storage_backend_name}' error for file {file_id}: {str(e)}"
|
||||
)
|
||||
continue
|
||||
|
||||
file_content = await storage_backend.download_file(storage_url)
|
||||
|
||||
# Determine content type from file object
|
||||
content_type = self._get_content_type_from_file_object(db_file.file_object)
|
||||
|
||||
# Convert to base64
|
||||
base64_data = base64.b64encode(file_content).decode("utf-8")
|
||||
base64_data_uri = f"data:{content_type};base64,{base64_data}"
|
||||
|
||||
# Update messages to use base64 instead of file_id
|
||||
self._update_messages_with_base64_data(messages, file_id, base64_data_uri, content_type)
|
||||
except Exception as e:
|
||||
verbose_logger.exception(
|
||||
f"Error converting file {file_id} from storage backend to base64: {str(e)}"
|
||||
)
|
||||
# Continue with other files even if one fails
|
||||
continue
|
||||
|
||||
def _get_content_type_from_file_object(self, file_object: Optional[Any]) -> str:
|
||||
"""
|
||||
Determine content type from file object.
|
||||
|
||||
Uses the MIME type utility for consistent detection and normalization.
|
||||
|
||||
Args:
|
||||
file_object: The file object from the database (can be dict, JSON string, or None)
|
||||
|
||||
Returns:
|
||||
str: MIME type (defaults to "application/octet-stream" if cannot be determined)
|
||||
"""
|
||||
# Use utility function for detection
|
||||
content_type = get_content_type_from_file_object(file_object)
|
||||
|
||||
# Normalize for Gemini/Vertex AI (requires image/jpeg, not image/jpg)
|
||||
content_type = normalize_mime_type_for_provider(content_type, provider="gemini")
|
||||
|
||||
return content_type
|
||||
|
||||
def _update_messages_with_base64_data(
|
||||
self,
|
||||
messages: List[AllMessageValues],
|
||||
file_id: str,
|
||||
base64_data_uri: str,
|
||||
content_type: str,
|
||||
) -> None:
|
||||
"""
|
||||
Update messages to replace file_id with base64 data URI.
|
||||
|
||||
Args:
|
||||
messages: List of messages to update
|
||||
file_id: The file ID to replace
|
||||
base64_data_uri: The base64 data URI to use as replacement
|
||||
content_type: The MIME type of the file (e.g., "image/jpeg", "application/pdf")
|
||||
"""
|
||||
for message in messages:
|
||||
if message.get("role") == "user":
|
||||
content = message.get("content")
|
||||
if content and isinstance(content, list):
|
||||
for element in content:
|
||||
if element.get("type") == "file":
|
||||
file_element = cast(ChatCompletionFileObject, element)
|
||||
file_element_file = file_element.get("file", {})
|
||||
|
||||
if file_element_file.get("file_id") == file_id:
|
||||
# Replace file_id with base64 data
|
||||
file_element_file["file_data"] = base64_data_uri
|
||||
# Set format to help Gemini determine mime type
|
||||
file_element_file["format"] = content_type
|
||||
# Remove file_id to ensure only file_data is used
|
||||
file_element_file.pop("file_id", None)
|
||||
|
||||
verbose_logger.debug(
|
||||
f"Converted file {file_id} from storage backend to base64 with format {content_type}"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,4 @@
|
|||
-- AlterTable
|
||||
ALTER TABLE "LiteLLM_ManagedFileTable" ADD COLUMN IF NOT EXISTS "storage_backend" TEXT;
|
||||
ALTER TABLE "LiteLLM_ManagedFileTable" ADD COLUMN IF NOT EXISTS "storage_url" TEXT;
|
||||
|
||||
|
|
@ -399,7 +399,10 @@ disable_copilot_system_to_assistant: bool = (
|
|||
public_mcp_servers: Optional[List[str]] = None
|
||||
public_model_groups: Optional[List[str]] = None
|
||||
public_agent_groups: Optional[List[str]] = None
|
||||
public_model_groups_links: Dict[str, str] = {}
|
||||
# Supports both old format (Dict[str, str]) and new format (Dict[str, Dict[str, Any]])
|
||||
# New format: { "displayName": { "url": "...", "index": 0 } }
|
||||
# Old format: { "displayName": "url" } (for backward compatibility)
|
||||
public_model_groups_links: Dict[str, Union[str, Dict[str, Any]]] = {}
|
||||
#### REQUEST PRIORITIZATION #######
|
||||
priority_reservation: Optional[Dict[str, Union[float, PriorityReservationDict]]] = None
|
||||
priority_reservation_settings: "PriorityReservationSettings" = (
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@ class A2ACompletionBridgeHandler:
|
|||
# Get completion params
|
||||
custom_llm_provider = litellm_params.get("custom_llm_provider")
|
||||
model = litellm_params.get("model", "agent")
|
||||
api_key = litellm_params.get("api_key")
|
||||
|
||||
# Build full model string if provider specified
|
||||
# Skip prepending if model already starts with the provider prefix
|
||||
|
|
@ -72,6 +73,7 @@ class A2ACompletionBridgeHandler:
|
|||
model=full_model,
|
||||
messages=openai_messages,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
stream=False,
|
||||
)
|
||||
|
||||
|
|
@ -127,6 +129,7 @@ class A2ACompletionBridgeHandler:
|
|||
# Get completion params
|
||||
custom_llm_provider = litellm_params.get("custom_llm_provider")
|
||||
model = litellm_params.get("model", "agent")
|
||||
api_key = litellm_params.get("api_key")
|
||||
|
||||
# Build full model string if provider specified
|
||||
# Skip prepending if model already starts with the provider prefix
|
||||
|
|
@ -157,6 +160,7 @@ class A2ACompletionBridgeHandler:
|
|||
model=full_model,
|
||||
messages=openai_messages,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -54,6 +54,9 @@ MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB = int(
|
|||
SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD = int(
|
||||
os.getenv("SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD", 1000)
|
||||
) # Minimum number of requests to consider "reasonable traffic". Used for single-deployment cooldown logic.
|
||||
DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS = int(
|
||||
os.getenv("DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS", 5)
|
||||
) # Minimum number of requests before applying error rate cooldown. Prevents cooldown from triggering on first failure.
|
||||
|
||||
DEFAULT_REASONING_EFFORT_DISABLE_THINKING_BUDGET = int(
|
||||
os.getenv("DEFAULT_REASONING_EFFORT_DISABLE_THINKING_BUDGET", 0)
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ from litellm.llms.vertex_ai.files.handler import VertexAIFilesHandler
|
|||
from litellm.types.llms.openai import (
|
||||
CreateFileRequest,
|
||||
FileContentRequest,
|
||||
FileExpiresAfter,
|
||||
FileTypes,
|
||||
HttpxBinaryResponseContent,
|
||||
OpenAIFileObject,
|
||||
|
|
@ -58,6 +59,7 @@ anthropic_files_instance = AnthropicFilesHandler()
|
|||
async def acreate_file(
|
||||
file: FileTypes,
|
||||
purpose: Literal["assistants", "batch", "fine-tune"],
|
||||
expires_after: Optional[FileExpiresAfter] = None,
|
||||
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
|
||||
extra_headers: Optional[Dict[str, str]] = None,
|
||||
extra_body: Optional[Dict[str, str]] = None,
|
||||
|
|
@ -75,6 +77,7 @@ async def acreate_file(
|
|||
call_args = {
|
||||
"file": file,
|
||||
"purpose": purpose,
|
||||
"expires_after": expires_after,
|
||||
"custom_llm_provider": custom_llm_provider,
|
||||
"extra_headers": extra_headers,
|
||||
"extra_body": extra_body,
|
||||
|
|
@ -83,7 +86,6 @@ async def acreate_file(
|
|||
|
||||
# Use a partial function to pass your keyword arguments
|
||||
func = partial(create_file, **call_args)
|
||||
|
||||
# Add the context to the function
|
||||
ctx = contextvars.copy_context()
|
||||
func_with_context = partial(ctx.run, func)
|
||||
|
|
@ -102,6 +104,7 @@ async def acreate_file(
|
|||
def create_file(
|
||||
file: FileTypes,
|
||||
purpose: Literal["assistants", "batch", "fine-tune"],
|
||||
expires_after: Optional[FileExpiresAfter] = None,
|
||||
custom_llm_provider: Optional[Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"]] = None,
|
||||
extra_headers: Optional[Dict[str, str]] = None,
|
||||
extra_body: Optional[Dict[str, str]] = None,
|
||||
|
|
@ -141,12 +144,21 @@ def create_file(
|
|||
elif timeout is None:
|
||||
timeout = 600.0
|
||||
|
||||
_create_file_request = CreateFileRequest(
|
||||
file=file,
|
||||
purpose=purpose,
|
||||
extra_headers=extra_headers,
|
||||
extra_body=extra_body,
|
||||
)
|
||||
if expires_after is not None:
|
||||
_create_file_request = CreateFileRequest(
|
||||
file=file,
|
||||
purpose=purpose,
|
||||
expires_after=expires_after,
|
||||
extra_headers=extra_headers,
|
||||
extra_body=extra_body,
|
||||
)
|
||||
else:
|
||||
_create_file_request = CreateFileRequest(
|
||||
file=file,
|
||||
purpose=purpose,
|
||||
extra_headers=extra_headers,
|
||||
extra_body=extra_body,
|
||||
)
|
||||
|
||||
provider_config = ProviderConfigManager.get_provider_files_config(
|
||||
model="",
|
||||
|
|
|
|||
|
|
@ -7,16 +7,18 @@ Users can define
|
|||
"""
|
||||
|
||||
import copy
|
||||
from typing import Dict, List, Optional, Tuple, Union, cast
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union, cast
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.integrations.custom_prompt_management import CustomPromptManagement
|
||||
from litellm.integrations.prompt_management_base import PromptManagementClient
|
||||
from litellm.types.integrations.anthropic_cache_control_hook import (
|
||||
CacheControlInjectionPoint,
|
||||
CacheControlMessageInjectionPoint,
|
||||
)
|
||||
from litellm.types.llms.openai import AllMessageValues, ChatCompletionCachedContent
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
|
||||
|
|
@ -29,6 +31,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
@ -141,6 +144,78 @@ class AnthropicCacheControlHook(CustomPromptManagement):
|
|||
"""Return the integration name for this hook."""
|
||||
return "anthropic_cache_control_hook"
|
||||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
"""Always return False since this is not a true prompt management system."""
|
||||
return False
|
||||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
"""Not used - this hook only modifies messages, doesn't fetch prompts."""
|
||||
return PromptManagementClient(
|
||||
prompt_id=prompt_id,
|
||||
prompt_template=[],
|
||||
prompt_template_model=None,
|
||||
prompt_template_optional_params=None,
|
||||
completed_messages=None,
|
||||
)
|
||||
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
"""Not used - this hook only modifies messages, doesn't fetch prompts."""
|
||||
return self._compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
async def async_get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: Any,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""Async version - delegates to sync since no async operations needed."""
|
||||
return self.get_chat_completion_prompt(
|
||||
model=model,
|
||||
messages=messages,
|
||||
non_default_params=non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def should_use_anthropic_cache_control_hook(non_default_params: Dict) -> bool:
|
||||
if non_default_params.get("cache_control_injection_points", None):
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ from litellm.integrations.prompt_management_base import (
|
|||
PromptManagementClient,
|
||||
)
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
from .bitbucket_client import BitBucketClient
|
||||
|
|
@ -414,7 +415,8 @@ class BitBucketPromptManager(CustomPromptManagement):
|
|||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
"""
|
||||
|
|
@ -423,11 +425,12 @@ class BitBucketPromptManager(CustomPromptManagement):
|
|||
For BitBucket, we always return True and handle the prompt loading
|
||||
in the _compile_prompt_helper method.
|
||||
"""
|
||||
return True
|
||||
return prompt_id is not None
|
||||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
|
|
@ -442,6 +445,9 @@ class BitBucketPromptManager(CustomPromptManagement):
|
|||
3. Converts the rendered text into chat messages
|
||||
4. Extracts model and optional parameters from metadata
|
||||
"""
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for BitBucket prompt manager")
|
||||
|
||||
try:
|
||||
# Load the prompt from BitBucket if not already loaded
|
||||
if prompt_id not in self.prompt_manager.prompts:
|
||||
|
|
@ -481,6 +487,31 @@ class BitBucketPromptManager(CustomPromptManagement):
|
|||
except Exception as e:
|
||||
raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
|
||||
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
"""
|
||||
Async version of compile prompt helper. Since BitBucket operations use sync client,
|
||||
this simply delegates to the sync version.
|
||||
"""
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for BitBucket prompt manager")
|
||||
|
||||
return self._compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
def get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
|
|
@ -489,6 +520,7 @@ class BitBucketPromptManager(CustomPromptManagement):
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
@ -505,6 +537,39 @@ class BitBucketPromptManager(CustomPromptManagement):
|
|||
prompt_id,
|
||||
prompt_variables,
|
||||
dynamic_callback_params,
|
||||
prompt_label,
|
||||
prompt_version,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
async def async_get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: Any,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""
|
||||
Async version - delegates to PromptManagementBase async implementation.
|
||||
"""
|
||||
return await PromptManagementBase.async_get_chat_completion_prompt(
|
||||
self,
|
||||
model,
|
||||
messages,
|
||||
non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
tools=tools,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ from litellm.caching.caching import DualCache
|
|||
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
|
||||
from litellm.types.integrations.argilla import ArgillaItem
|
||||
from litellm.types.llms.openai import AllMessageValues, ChatCompletionRequest
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import (
|
||||
AdapterCompletionStreamWrapper,
|
||||
CallTypes,
|
||||
|
|
@ -158,9 +159,12 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
|
|||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: LiteLLMLoggingObj,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
ignore_prompt_manager_optional_params: Optional[bool] = False,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""
|
||||
Returns:
|
||||
|
|
@ -178,6 +182,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ from litellm.integrations.prompt_management_base import (
|
|||
PromptManagementClient,
|
||||
)
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
|
||||
|
|
@ -29,6 +30,7 @@ class CustomPromptManagement(CustomLogger, PromptManagementBase):
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
@ -48,14 +50,16 @@ class CustomPromptManagement(CustomLogger, PromptManagementBase):
|
|||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
return True
|
||||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ from typing import Any, Dict, List, Optional, Tuple, Union
|
|||
from litellm.integrations.custom_prompt_management import CustomPromptManagement
|
||||
from litellm.integrations.prompt_management_base import PromptManagementClient
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
from .prompt_manager import PromptManager, PromptTemplate
|
||||
|
|
@ -82,7 +83,8 @@ class DotpromptManager(CustomPromptManagement):
|
|||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
"""
|
||||
|
|
@ -90,6 +92,8 @@ class DotpromptManager(CustomPromptManagement):
|
|||
|
||||
Returns True if the prompt_id exists in our prompt manager.
|
||||
"""
|
||||
if prompt_id is None:
|
||||
return False
|
||||
try:
|
||||
return prompt_id in self.prompt_manager.list_prompts()
|
||||
except Exception:
|
||||
|
|
@ -98,7 +102,8 @@ class DotpromptManager(CustomPromptManagement):
|
|||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
|
|
@ -114,6 +119,9 @@ class DotpromptManager(CustomPromptManagement):
|
|||
4. Extracts model and optional parameters from metadata
|
||||
"""
|
||||
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for dotprompt manager")
|
||||
|
||||
try:
|
||||
|
||||
# Get the prompt template (versioned or base)
|
||||
|
|
@ -153,6 +161,31 @@ class DotpromptManager(CustomPromptManagement):
|
|||
except Exception as e:
|
||||
raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
|
||||
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
"""
|
||||
Async version of compile prompt helper. Since dotprompt operations are synchronous,
|
||||
this simply delegates to the sync version.
|
||||
"""
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for dotprompt manager")
|
||||
|
||||
return self._compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
def get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
|
|
@ -161,6 +194,7 @@ class DotpromptManager(CustomPromptManagement):
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
@ -177,8 +211,43 @@ class DotpromptManager(CustomPromptManagement):
|
|||
prompt_id,
|
||||
prompt_variables,
|
||||
dynamic_callback_params,
|
||||
prompt_label,
|
||||
prompt_version,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
async def async_get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: Any,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""
|
||||
Async version - delegates to PromptManagementBase async implementation.
|
||||
"""
|
||||
from litellm.integrations.prompt_management_base import PromptManagementBase
|
||||
|
||||
return await PromptManagementBase.async_get_chat_completion_prompt(
|
||||
self,
|
||||
model,
|
||||
messages,
|
||||
non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
tools=tools,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
def _convert_to_messages(self, rendered_content: str) -> List[AllMessageValues]:
|
||||
|
|
|
|||
80
litellm/integrations/generic_prompt_management/__init__.py
Normal file
80
litellm/integrations/generic_prompt_management/__init__.py
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
"""Generic prompt management integration for LiteLLM."""
|
||||
|
||||
from typing import TYPE_CHECKING, Optional
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .generic_prompt_manager import GenericPromptManager
|
||||
from litellm.types.prompts.init_prompts import PromptLiteLLMParams, PromptSpec
|
||||
from litellm.integrations.custom_prompt_management import CustomPromptManagement
|
||||
|
||||
from litellm.types.prompts.init_prompts import SupportedPromptIntegrations
|
||||
|
||||
from .generic_prompt_manager import GenericPromptManager
|
||||
|
||||
# Global instances
|
||||
global_generic_prompt_config: Optional[dict] = None
|
||||
|
||||
|
||||
def set_global_generic_prompt_config(config: dict) -> None:
|
||||
"""
|
||||
Set the global generic prompt configuration.
|
||||
|
||||
Args:
|
||||
config: Dictionary containing generic prompt configuration
|
||||
- api_base: Base URL for the API
|
||||
- api_key: Optional API key for authentication
|
||||
- timeout: Request timeout in seconds (default: 30)
|
||||
"""
|
||||
import litellm
|
||||
|
||||
litellm.global_generic_prompt_config = config # type: ignore
|
||||
|
||||
|
||||
def prompt_initializer(
|
||||
litellm_params: "PromptLiteLLMParams", prompt_spec: "PromptSpec"
|
||||
) -> "CustomPromptManagement":
|
||||
"""
|
||||
Initialize a prompt from a generic prompt management API.
|
||||
"""
|
||||
prompt_id = getattr(litellm_params, "prompt_id", None)
|
||||
|
||||
api_base = litellm_params.api_base
|
||||
api_key = litellm_params.api_key
|
||||
if not api_base:
|
||||
raise ValueError("api_base is required in generic_prompt_config")
|
||||
|
||||
provider_specific_query_params = litellm_params.provider_specific_query_params
|
||||
|
||||
try:
|
||||
generic_prompt_manager = GenericPromptManager(
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
prompt_id=prompt_id,
|
||||
additional_provider_specific_query_params=provider_specific_query_params,
|
||||
**litellm_params.model_dump(
|
||||
exclude_none=True,
|
||||
exclude={
|
||||
"prompt_id",
|
||||
"api_key",
|
||||
"provider_specific_query_params",
|
||||
"api_base",
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
return generic_prompt_manager
|
||||
except Exception as e:
|
||||
raise e
|
||||
|
||||
|
||||
prompt_initializer_registry = {
|
||||
SupportedPromptIntegrations.GENERIC_PROMPT_MANAGEMENT.value: prompt_initializer,
|
||||
}
|
||||
|
||||
# Export public API
|
||||
__all__ = [
|
||||
"GenericPromptManager",
|
||||
"set_global_generic_prompt_config",
|
||||
"global_generic_prompt_config",
|
||||
"prompt_initializer_registry",
|
||||
]
|
||||
|
|
@ -0,0 +1,501 @@
|
|||
"""
|
||||
Generic prompt manager that integrates with LiteLLM's prompt management system.
|
||||
Fetches prompts from any API that implements the /beta/litellm_prompt_management endpoint.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
|
||||
|
||||
import httpx
|
||||
|
||||
from litellm.integrations.custom_prompt_management import CustomPromptManagement
|
||||
from litellm.integrations.prompt_management_base import (
|
||||
PromptManagementBase,
|
||||
PromptManagementClient,
|
||||
)
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
_get_httpx_client,
|
||||
get_async_httpx_client,
|
||||
)
|
||||
from litellm.types.llms.custom_http import httpxSpecialProvider
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
|
||||
|
||||
|
||||
class GenericPromptManager(CustomPromptManagement):
|
||||
"""
|
||||
Generic prompt manager that integrates with LiteLLM's prompt management system.
|
||||
|
||||
This class enables using prompts from any API that implements the
|
||||
/beta/litellm_prompt_management endpoint.
|
||||
|
||||
Usage:
|
||||
# Configure API access
|
||||
generic_config = {
|
||||
"api_base": "https://your-api.com",
|
||||
"api_key": "your-api-key", # optional
|
||||
"timeout": 30, # optional, defaults to 30
|
||||
}
|
||||
|
||||
# Use with completion
|
||||
response = litellm.completion(
|
||||
model="generic_prompt/gpt-4",
|
||||
prompt_id="my_prompt_id",
|
||||
prompt_variables={"variable": "value"},
|
||||
generic_prompt_config=generic_config,
|
||||
messages=[{"role": "user", "content": "Additional message"}]
|
||||
)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_base: str,
|
||||
api_key: Optional[str] = None,
|
||||
timeout: int = 30,
|
||||
prompt_id: Optional[str] = None,
|
||||
additional_provider_specific_query_params: Optional[Dict[str, Any]] = None,
|
||||
**kwargs,
|
||||
):
|
||||
"""
|
||||
Initialize the Generic Prompt Manager.
|
||||
|
||||
Args:
|
||||
api_base: Base URL for the API (e.g., "https://your-api.com")
|
||||
api_key: Optional API key for authentication
|
||||
timeout: Request timeout in seconds (default: 30)
|
||||
prompt_id: Optional prompt ID to pre-load
|
||||
"""
|
||||
super().__init__(**kwargs)
|
||||
self.api_base = api_base.rstrip("/")
|
||||
self.api_key = api_key
|
||||
self.timeout = timeout
|
||||
self.prompt_id = prompt_id
|
||||
self.additional_provider_specific_query_params = (
|
||||
additional_provider_specific_query_params
|
||||
)
|
||||
self._prompt_cache: Dict[str, PromptManagementClient] = {}
|
||||
|
||||
@property
|
||||
def integration_name(self) -> str:
|
||||
"""Integration name used in model names like 'generic_prompt/gpt-4'."""
|
||||
return "generic_prompt"
|
||||
|
||||
def _get_headers(self) -> Dict[str, str]:
|
||||
"""Get HTTP headers for API requests."""
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if self.api_key:
|
||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
||||
return headers
|
||||
|
||||
def _fetch_prompt_from_api(
|
||||
self, prompt_id: Optional[str], prompt_spec: Optional[PromptSpec]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch a prompt from the API.
|
||||
|
||||
Args:
|
||||
prompt_id: The ID of the prompt to fetch
|
||||
|
||||
Returns:
|
||||
The prompt data from the API
|
||||
|
||||
Raises:
|
||||
Exception: If the API request fails
|
||||
"""
|
||||
if prompt_id is None and prompt_spec is None:
|
||||
raise ValueError("prompt_id or prompt_spec is required")
|
||||
|
||||
url = f"{self.api_base}/beta/litellm_prompt_management"
|
||||
params = {
|
||||
"prompt_id": prompt_id,
|
||||
**(self.additional_provider_specific_query_params or {}),
|
||||
}
|
||||
http_client = _get_httpx_client()
|
||||
|
||||
try:
|
||||
|
||||
response = http_client.get(
|
||||
url,
|
||||
params=params,
|
||||
headers=self._get_headers(),
|
||||
)
|
||||
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPError as e:
|
||||
raise Exception(f"Failed to fetch prompt '{prompt_id}' from API: {e}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise Exception(f"Failed to parse prompt response for '{prompt_id}': {e}")
|
||||
|
||||
async def async_fetch_prompt_from_api(
|
||||
self, prompt_id: Optional[str], prompt_spec: Optional[PromptSpec]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch a prompt from the API asynchronously.
|
||||
"""
|
||||
if prompt_id is None and prompt_spec is None:
|
||||
raise ValueError("prompt_id or prompt_spec is required")
|
||||
|
||||
url = f"{self.api_base}/beta/litellm_prompt_management"
|
||||
params = {
|
||||
"prompt_id": prompt_id,
|
||||
**(
|
||||
prompt_spec.litellm_params.provider_specific_query_params
|
||||
if prompt_spec
|
||||
and prompt_spec.litellm_params.provider_specific_query_params
|
||||
else {}
|
||||
),
|
||||
}
|
||||
|
||||
http_client = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.PromptManagement,
|
||||
)
|
||||
|
||||
try:
|
||||
response = await http_client.get(
|
||||
url,
|
||||
params=params,
|
||||
headers=self._get_headers(),
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPError as e:
|
||||
raise Exception(f"Failed to fetch prompt '{prompt_id}' from API: {e}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise Exception(f"Failed to parse prompt response for '{prompt_id}': {e}")
|
||||
|
||||
def _parse_api_response(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
api_response: Dict[str, Any],
|
||||
) -> PromptManagementClient:
|
||||
"""
|
||||
Parse the API response into a PromptManagementClient structure.
|
||||
|
||||
Expected API response format:
|
||||
{
|
||||
"prompt_id": "string",
|
||||
"prompt_template": [
|
||||
{"role": "system", "content": "..."},
|
||||
{"role": "user", "content": "..."}
|
||||
],
|
||||
"prompt_template_model": "gpt-4", # optional
|
||||
"prompt_template_optional_params": { # optional
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100
|
||||
}
|
||||
}
|
||||
|
||||
Args:
|
||||
prompt_id: The ID of the prompt
|
||||
api_response: The response from the API
|
||||
|
||||
Returns:
|
||||
PromptManagementClient structure
|
||||
"""
|
||||
return PromptManagementClient(
|
||||
prompt_id=prompt_id,
|
||||
prompt_template=api_response.get("prompt_template", []),
|
||||
prompt_template_model=api_response.get("prompt_template_model"),
|
||||
prompt_template_optional_params=api_response.get(
|
||||
"prompt_template_optional_params"
|
||||
),
|
||||
completed_messages=None,
|
||||
)
|
||||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
"""
|
||||
Determine if prompt management should run based on the prompt_id.
|
||||
|
||||
For Generic Prompt Manager, we always return True and handle the prompt loading
|
||||
in the _compile_prompt_helper method.
|
||||
"""
|
||||
if prompt_id is not None or (
|
||||
prompt_spec is not None
|
||||
and prompt_spec.litellm_params.provider_specific_query_params is not None
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _get_cache_key(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> str:
|
||||
return f"{prompt_id}:{prompt_label}:{prompt_version}"
|
||||
|
||||
def _common_caching_logic(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
prompt_variables: Optional[dict] = None,
|
||||
) -> Optional[PromptManagementClient]:
|
||||
"""
|
||||
Common caching logic for the prompt manager.
|
||||
"""
|
||||
# Check cache first
|
||||
cache_key = self._get_cache_key(prompt_id, prompt_label, prompt_version)
|
||||
if cache_key in self._prompt_cache:
|
||||
cached_prompt = self._prompt_cache[cache_key]
|
||||
# Return a copy with variables applied if needed
|
||||
if prompt_variables:
|
||||
return self._apply_variables(cached_prompt, prompt_variables)
|
||||
return cached_prompt
|
||||
return None
|
||||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
"""
|
||||
Compile a prompt template into a PromptManagementClient structure.
|
||||
|
||||
This method:
|
||||
1. Fetches the prompt from the API (with caching)
|
||||
2. Applies any prompt variables (if the API supports it)
|
||||
3. Returns the structured prompt data
|
||||
|
||||
Args:
|
||||
prompt_id: The ID of the prompt
|
||||
prompt_variables: Variables to substitute in the template (optional)
|
||||
dynamic_callback_params: Dynamic callback parameters
|
||||
prompt_label: Optional label for the prompt version
|
||||
prompt_version: Optional specific version number
|
||||
|
||||
Returns:
|
||||
PromptManagementClient structure
|
||||
"""
|
||||
cached_prompt = self._common_caching_logic(
|
||||
prompt_id=prompt_id,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
prompt_variables=prompt_variables,
|
||||
)
|
||||
if cached_prompt:
|
||||
return cached_prompt
|
||||
|
||||
cache_key = self._get_cache_key(prompt_id, prompt_label, prompt_version)
|
||||
try:
|
||||
# Fetch from API
|
||||
api_response = self._fetch_prompt_from_api(prompt_id, prompt_spec)
|
||||
|
||||
# Parse the response
|
||||
prompt_client = self._parse_api_response(
|
||||
prompt_id, prompt_spec, api_response
|
||||
)
|
||||
|
||||
# Cache the result
|
||||
self._prompt_cache[cache_key] = prompt_client
|
||||
|
||||
# Apply variables if provided
|
||||
if prompt_variables:
|
||||
prompt_client = self._apply_variables(prompt_client, prompt_variables)
|
||||
|
||||
return prompt_client
|
||||
|
||||
except Exception as e:
|
||||
raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
|
||||
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
|
||||
# Check cache first
|
||||
cached_prompt = self._common_caching_logic(
|
||||
prompt_id=prompt_id,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
prompt_variables=prompt_variables,
|
||||
)
|
||||
if cached_prompt:
|
||||
return cached_prompt
|
||||
|
||||
cache_key = self._get_cache_key(prompt_id, prompt_label, prompt_version)
|
||||
|
||||
try:
|
||||
# Fetch from API
|
||||
|
||||
api_response = await self.async_fetch_prompt_from_api(
|
||||
prompt_id=prompt_id, prompt_spec=prompt_spec
|
||||
)
|
||||
|
||||
# Parse the response
|
||||
prompt_client = self._parse_api_response(
|
||||
prompt_id, prompt_spec, api_response
|
||||
)
|
||||
|
||||
# Cache the result
|
||||
self._prompt_cache[cache_key] = prompt_client
|
||||
|
||||
# Apply variables if provided
|
||||
if prompt_variables:
|
||||
prompt_client = self._apply_variables(prompt_client, prompt_variables)
|
||||
|
||||
return prompt_client
|
||||
|
||||
except Exception as e:
|
||||
raise ValueError(
|
||||
f"Error compiling prompt '{prompt_id}': {e}, prompt_spec: {prompt_spec}"
|
||||
)
|
||||
|
||||
def _apply_variables(
|
||||
self,
|
||||
prompt_client: PromptManagementClient,
|
||||
variables: Dict[str, Any],
|
||||
) -> PromptManagementClient:
|
||||
"""
|
||||
Apply variables to the prompt template.
|
||||
|
||||
This performs simple string substitution using {variable_name} syntax.
|
||||
|
||||
Args:
|
||||
prompt_client: The prompt client structure
|
||||
variables: Variables to substitute
|
||||
|
||||
Returns:
|
||||
Updated PromptManagementClient with variables applied
|
||||
"""
|
||||
# Create a copy of the prompt template with variables applied
|
||||
updated_messages: List[AllMessageValues] = []
|
||||
for message in prompt_client["prompt_template"]:
|
||||
updated_message = dict(message) # type: ignore
|
||||
if "content" in updated_message and isinstance(
|
||||
updated_message["content"], str
|
||||
):
|
||||
content = updated_message["content"]
|
||||
for key, value in variables.items():
|
||||
content = content.replace(f"{{{key}}}", str(value))
|
||||
content = content.replace(
|
||||
f"{{{{{key}}}}}", str(value)
|
||||
) # Also support {{key}}
|
||||
updated_message["content"] = content
|
||||
updated_messages.append(updated_message) # type: ignore
|
||||
|
||||
return PromptManagementClient(
|
||||
prompt_id=prompt_client["prompt_id"],
|
||||
prompt_template=updated_messages,
|
||||
prompt_template_model=prompt_client["prompt_template_model"],
|
||||
prompt_template_optional_params=prompt_client[
|
||||
"prompt_template_optional_params"
|
||||
],
|
||||
completed_messages=None,
|
||||
)
|
||||
|
||||
async def async_get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: "LiteLLMLoggingObj",
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
ignore_prompt_manager_optional_params: Optional[bool] = False,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""
|
||||
Get chat completion prompt and return processed model, messages, and parameters.
|
||||
"""
|
||||
|
||||
return await PromptManagementBase.async_get_chat_completion_prompt(
|
||||
self,
|
||||
model,
|
||||
messages,
|
||||
non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
tools=tools,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
ignore_prompt_manager_model=(
|
||||
ignore_prompt_manager_model
|
||||
or prompt_spec.litellm_params.ignore_prompt_manager_model
|
||||
if prompt_spec
|
||||
else False
|
||||
),
|
||||
ignore_prompt_manager_optional_params=(
|
||||
ignore_prompt_manager_optional_params
|
||||
or prompt_spec.litellm_params.ignore_prompt_manager_optional_params
|
||||
if prompt_spec
|
||||
else False
|
||||
),
|
||||
)
|
||||
|
||||
def get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
ignore_prompt_manager_optional_params: Optional[bool] = False,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""
|
||||
Get chat completion prompt and return processed model, messages, and parameters.
|
||||
"""
|
||||
return PromptManagementBase.get_chat_completion_prompt(
|
||||
self,
|
||||
model,
|
||||
messages,
|
||||
non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
ignore_prompt_manager_model=(
|
||||
ignore_prompt_manager_model
|
||||
or prompt_spec.litellm_params.ignore_prompt_manager_model
|
||||
if prompt_spec
|
||||
else False
|
||||
),
|
||||
ignore_prompt_manager_optional_params=(
|
||||
ignore_prompt_manager_optional_params
|
||||
or prompt_spec.litellm_params.ignore_prompt_manager_optional_params
|
||||
if prompt_spec
|
||||
else False
|
||||
),
|
||||
)
|
||||
|
||||
def clear_cache(self) -> None:
|
||||
"""Clear the prompt cache."""
|
||||
self._prompt_cache.clear()
|
||||
|
|
@ -13,6 +13,7 @@ from litellm.integrations.prompt_management_base import (
|
|||
PromptManagementClient,
|
||||
)
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
GITLAB_PREFIX = "gitlab::"
|
||||
|
|
@ -454,19 +455,24 @@ class GitLabPromptManager(CustomPromptManagement):
|
|||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
return True
|
||||
return prompt_id is not None
|
||||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for GitLab prompt manager")
|
||||
|
||||
try:
|
||||
decoded_id = decode_prompt_id(prompt_id)
|
||||
if decoded_id not in self.prompt_manager.prompts:
|
||||
|
|
@ -505,6 +511,31 @@ class GitLabPromptManager(CustomPromptManagement):
|
|||
except Exception as e:
|
||||
raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
|
||||
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
"""
|
||||
Async version of compile prompt helper. Since GitLab operations use sync client,
|
||||
this simply delegates to the sync version.
|
||||
"""
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for GitLab prompt manager")
|
||||
|
||||
return self._compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
def get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
|
|
@ -513,6 +544,7 @@ class GitLabPromptManager(CustomPromptManagement):
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
@ -526,8 +558,41 @@ class GitLabPromptManager(CustomPromptManagement):
|
|||
prompt_id,
|
||||
prompt_variables,
|
||||
dynamic_callback_params,
|
||||
prompt_label,
|
||||
prompt_version,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
async def async_get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: Any,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
"""
|
||||
Async version - delegates to PromptManagementBase async implementation.
|
||||
"""
|
||||
return await PromptManagementBase.async_get_chat_completion_prompt(
|
||||
self,
|
||||
model,
|
||||
messages,
|
||||
non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
tools=tools,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ from litellm.integrations.custom_logger import CustomLogger
|
|||
from litellm.integrations.prompt_management_base import PromptManagementClient
|
||||
from litellm.litellm_core_utils.asyncify import run_async_function
|
||||
from litellm.types.llms.openai import AllMessageValues, ChatCompletionSystemMessage
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams, StandardLoggingPayload
|
||||
|
||||
from ...litellm_core_utils.specialty_caches.dynamic_logging_cache import (
|
||||
|
|
@ -183,6 +184,7 @@ class LangfusePromptManagement(LangFuseLogger, PromptManagementBase, CustomLogge
|
|||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: LiteLLMLoggingObj,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
|
|
@ -200,9 +202,12 @@ class LangfusePromptManagement(LangFuseLogger, PromptManagementBase, CustomLogge
|
|||
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
if prompt_id is None:
|
||||
return False
|
||||
langfuse_client = langfuse_client_init(
|
||||
langfuse_public_key=dynamic_callback_params.get("langfuse_public_key"),
|
||||
langfuse_secret=dynamic_callback_params.get("langfuse_secret"),
|
||||
|
|
@ -217,12 +222,16 @@ class LangfusePromptManagement(LangFuseLogger, PromptManagementBase, CustomLogge
|
|||
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for Langfuse prompt management")
|
||||
|
||||
langfuse_client = langfuse_client_init(
|
||||
langfuse_public_key=dynamic_callback_params.get("langfuse_public_key"),
|
||||
langfuse_secret=dynamic_callback_params.get("langfuse_secret"),
|
||||
|
|
@ -257,6 +266,24 @@ class LangfusePromptManagement(LangFuseLogger, PromptManagementBase, CustomLogge
|
|||
completed_messages=None,
|
||||
)
|
||||
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
return self._compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
def log_success_event(self, kwargs, response_obj, start_time, end_time):
|
||||
return run_async_function(
|
||||
self.async_log_success_event, kwargs, response_obj, start_time, end_time
|
||||
|
|
|
|||
|
|
@ -248,6 +248,9 @@ class OpenTelemetry(CustomLogger):
|
|||
self._operation_duration_histogram = None
|
||||
self._token_usage_histogram = None
|
||||
self._cost_histogram = None
|
||||
self._time_to_first_token_histogram = None
|
||||
self._time_per_output_token_histogram = None
|
||||
self._response_duration_histogram = None
|
||||
return
|
||||
|
||||
from opentelemetry import metrics
|
||||
|
|
@ -300,6 +303,21 @@ class OpenTelemetry(CustomLogger):
|
|||
description="GenAI request cost",
|
||||
unit="USD",
|
||||
)
|
||||
self._time_to_first_token_histogram = meter.create_histogram(
|
||||
name="gen_ai.client.response.time_to_first_token",
|
||||
description="Time to first token for streaming requests",
|
||||
unit="s",
|
||||
)
|
||||
self._time_per_output_token_histogram = meter.create_histogram(
|
||||
name="gen_ai.client.response.time_per_output_token",
|
||||
description="Average time per output token (generation time / completion tokens)",
|
||||
unit="s",
|
||||
)
|
||||
self._response_duration_histogram = meter.create_histogram(
|
||||
name="gen_ai.client.response.duration",
|
||||
description="Total LLM API generation time (excludes LiteLLM overhead)",
|
||||
unit="s",
|
||||
)
|
||||
|
||||
def _init_logs(self, logger_provider):
|
||||
# nothing to do if events disabled
|
||||
|
|
@ -612,8 +630,9 @@ class OpenTelemetry(CustomLogger):
|
|||
if self.config.enable_events:
|
||||
self._emit_semantic_logs(kwargs, response_obj, span)
|
||||
|
||||
# 6. End parent span
|
||||
if parent_span is not None:
|
||||
# 6. End parent span (only if it wasn't reused as the primary span)
|
||||
# If parent_span was reused as the primary span, it was already ended in _start_primary_span
|
||||
if parent_span is not None and parent_span is not span:
|
||||
parent_span.end(end_time=self._to_ns(datetime.now()))
|
||||
|
||||
def _start_primary_span(
|
||||
|
|
@ -727,6 +746,152 @@ class OpenTelemetry(CustomLogger):
|
|||
if self._cost_histogram and cost:
|
||||
self._cost_histogram.record(cost, attributes=common_attrs)
|
||||
|
||||
# Record latency metrics (TTFT, TPOT, and Total Generation Time)
|
||||
self._record_time_to_first_token_metric(kwargs, common_attrs)
|
||||
self._record_time_per_output_token_metric(
|
||||
kwargs, response_obj, end_time, duration_s, common_attrs
|
||||
)
|
||||
self._record_response_duration_metric(kwargs, end_time, common_attrs)
|
||||
|
||||
def _record_time_to_first_token_metric(self, kwargs: dict, common_attrs: dict):
|
||||
"""Record Time to First Token (TTFT) metric for streaming requests."""
|
||||
optional_params = kwargs.get("optional_params", {})
|
||||
is_streaming = optional_params.get("stream", False)
|
||||
|
||||
if not (self._time_to_first_token_histogram and is_streaming):
|
||||
return
|
||||
|
||||
# Use api_call_start_time for precision (matches Prometheus implementation)
|
||||
# This excludes LiteLLM overhead and measures pure LLM API latency
|
||||
api_call_start_time = kwargs.get("api_call_start_time", None)
|
||||
completion_start_time = kwargs.get("completion_start_time", None)
|
||||
|
||||
if api_call_start_time is not None and completion_start_time is not None:
|
||||
# Convert to timestamps if needed (handles both datetime and float)
|
||||
if isinstance(api_call_start_time, datetime):
|
||||
api_call_start_ts = api_call_start_time.timestamp()
|
||||
else:
|
||||
api_call_start_ts = api_call_start_time
|
||||
|
||||
if isinstance(completion_start_time, datetime):
|
||||
completion_start_ts = completion_start_time.timestamp()
|
||||
else:
|
||||
completion_start_ts = completion_start_time
|
||||
|
||||
time_to_first_token_seconds = completion_start_ts - api_call_start_ts
|
||||
self._time_to_first_token_histogram.record(
|
||||
time_to_first_token_seconds, attributes=common_attrs
|
||||
)
|
||||
|
||||
def _record_time_per_output_token_metric(
|
||||
self,
|
||||
kwargs: dict,
|
||||
response_obj: Optional[Any],
|
||||
end_time: datetime,
|
||||
duration_s: float,
|
||||
common_attrs: dict,
|
||||
):
|
||||
"""Record Time Per Output Token (TPOT) metric.
|
||||
|
||||
Calculated as: generation_time / completion_tokens
|
||||
- For streaming: uses end_time - completion_start_time (time to generate all tokens after first)
|
||||
- For non-streaming: uses end_time - api_call_start_time (total generation time)
|
||||
"""
|
||||
if not self._time_per_output_token_histogram:
|
||||
return
|
||||
|
||||
# Get completion tokens from response_obj
|
||||
completion_tokens = None
|
||||
if response_obj and (usage := response_obj.get("usage")):
|
||||
completion_tokens = usage.get("completion_tokens")
|
||||
|
||||
if completion_tokens is None or completion_tokens <= 0:
|
||||
return
|
||||
|
||||
# Calculate generation time
|
||||
completion_start_time = kwargs.get("completion_start_time", None)
|
||||
api_call_start_time = kwargs.get("api_call_start_time", None)
|
||||
|
||||
# Convert end_time to timestamp
|
||||
if isinstance(end_time, datetime):
|
||||
end_time_ts = end_time.timestamp()
|
||||
else:
|
||||
end_time_ts = end_time
|
||||
|
||||
if completion_start_time is not None:
|
||||
# Streaming: use completion_start_time (when first token arrived)
|
||||
# This measures time to generate all tokens after the first one
|
||||
if isinstance(completion_start_time, datetime):
|
||||
completion_start_ts = completion_start_time.timestamp()
|
||||
else:
|
||||
completion_start_ts = completion_start_time
|
||||
|
||||
generation_time_seconds = end_time_ts - completion_start_ts
|
||||
elif api_call_start_time is not None:
|
||||
# Non-streaming: use api_call_start_time (total generation time)
|
||||
if isinstance(api_call_start_time, datetime):
|
||||
api_call_start_ts = api_call_start_time.timestamp()
|
||||
else:
|
||||
api_call_start_ts = api_call_start_time
|
||||
|
||||
generation_time_seconds = end_time_ts - api_call_start_ts
|
||||
else:
|
||||
# Fallback: use duration_s (already calculated as (end_time - start_time).total_seconds())
|
||||
generation_time_seconds = duration_s
|
||||
|
||||
if generation_time_seconds > 0:
|
||||
time_per_output_token_seconds = generation_time_seconds / completion_tokens
|
||||
self._time_per_output_token_histogram.record(
|
||||
time_per_output_token_seconds, attributes=common_attrs
|
||||
)
|
||||
|
||||
def _record_response_duration_metric(
|
||||
self,
|
||||
kwargs: dict,
|
||||
end_time: Union[datetime, float],
|
||||
common_attrs: dict,
|
||||
):
|
||||
"""Record Total Generation Time (response duration) metric.
|
||||
|
||||
Measures pure LLM API generation time: end_time - api_call_start_time
|
||||
This excludes LiteLLM overhead and measures only the LLM provider's response time.
|
||||
Works for both streaming and non-streaming requests.
|
||||
|
||||
Mirrors Prometheus's litellm_llm_api_latency_metric.
|
||||
Uses kwargs.get("end_time") with fallback to parameter for consistency with Prometheus.
|
||||
"""
|
||||
if not self._response_duration_histogram:
|
||||
return
|
||||
|
||||
api_call_start_time = kwargs.get("api_call_start_time", None)
|
||||
if api_call_start_time is None:
|
||||
return
|
||||
|
||||
# Use end_time from kwargs if available (matches Prometheus), otherwise use parameter
|
||||
# For streaming: end_time is when the stream completes (final chunk received)
|
||||
# For non-streaming: end_time is when the response is received
|
||||
_end_time = kwargs.get("end_time") or end_time
|
||||
if _end_time is None:
|
||||
_end_time = datetime.now()
|
||||
|
||||
# Convert to timestamps if needed (handles both datetime and float)
|
||||
if isinstance(api_call_start_time, datetime):
|
||||
api_call_start_ts = api_call_start_time.timestamp()
|
||||
else:
|
||||
api_call_start_ts = api_call_start_time
|
||||
|
||||
if isinstance(_end_time, datetime):
|
||||
end_time_ts = _end_time.timestamp()
|
||||
else:
|
||||
end_time_ts = _end_time
|
||||
|
||||
response_duration_seconds = end_time_ts - api_call_start_ts
|
||||
|
||||
if response_duration_seconds > 0:
|
||||
self._response_duration_histogram.record(
|
||||
response_duration_seconds, attributes=common_attrs
|
||||
)
|
||||
|
||||
def _emit_semantic_logs(self, kwargs, response_obj, span: Span):
|
||||
if not self.config.enable_events:
|
||||
return
|
||||
|
|
@ -1226,7 +1391,7 @@ class OpenTelemetry(CustomLogger):
|
|||
value=usage.get("prompt_tokens"),
|
||||
)
|
||||
|
||||
########################################################################
|
||||
########################################################################
|
||||
########## LLM Request Medssages / tools / content Attributes ###########
|
||||
#########################################################################
|
||||
|
||||
|
|
|
|||
|
|
@ -1,14 +1,18 @@
|
|||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from typing_extensions import TypedDict
|
||||
from typing_extensions import TYPE_CHECKING, TypedDict
|
||||
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.utils import StandardCallbackDynamicParams
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
|
||||
|
||||
|
||||
class PromptManagementClient(TypedDict):
|
||||
prompt_id: str
|
||||
prompt_id: Optional[str]
|
||||
prompt_template: List[AllMessageValues]
|
||||
prompt_template_model: Optional[str]
|
||||
prompt_template_optional_params: Optional[Dict[str, Any]]
|
||||
|
|
@ -24,7 +28,8 @@ class PromptManagementBase(ABC):
|
|||
@abstractmethod
|
||||
def should_run_prompt_management(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> bool:
|
||||
pass
|
||||
|
|
@ -32,7 +37,8 @@ class PromptManagementBase(ABC):
|
|||
@abstractmethod
|
||||
def _compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_id: Optional[str],
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
|
|
@ -40,6 +46,18 @@ class PromptManagementBase(ABC):
|
|||
) -> PromptManagementClient:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def async_compile_prompt_helper(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
pass
|
||||
|
||||
def merge_messages(
|
||||
self,
|
||||
prompt_template: List[AllMessageValues],
|
||||
|
|
@ -55,10 +73,41 @@ class PromptManagementBase(ABC):
|
|||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
) -> PromptManagementClient:
|
||||
|
||||
compiled_prompt_client = self._compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
try:
|
||||
messages = compiled_prompt_client["prompt_template"] + client_messages
|
||||
except Exception as e:
|
||||
raise ValueError(
|
||||
f"Error compiling prompt: {e}. Prompt id={prompt_id}, prompt_variables={prompt_variables}, client_messages={client_messages}, dynamic_callback_params={dynamic_callback_params}"
|
||||
)
|
||||
|
||||
compiled_prompt_client["completed_messages"] = messages
|
||||
return compiled_prompt_client
|
||||
|
||||
async def async_compile_prompt(
|
||||
self,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
client_messages: List[AllMessageValues],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
) -> PromptManagementClient:
|
||||
compiled_prompt_client = await self.async_compile_prompt_helper(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_label=prompt_label,
|
||||
|
|
@ -83,6 +132,39 @@ class PromptManagementBase(ABC):
|
|||
else:
|
||||
return model.replace("{}/".format(self.integration_name), "")
|
||||
|
||||
def post_compile_prompt_processing(
|
||||
self,
|
||||
prompt_template: PromptManagementClient,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
model: str,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
ignore_prompt_manager_optional_params: Optional[bool] = False,
|
||||
):
|
||||
completed_messages = prompt_template["completed_messages"] or messages
|
||||
|
||||
prompt_template_optional_params = (
|
||||
prompt_template["prompt_template_optional_params"] or {}
|
||||
)
|
||||
|
||||
updated_non_default_params = {
|
||||
**non_default_params,
|
||||
**(
|
||||
prompt_template_optional_params
|
||||
if not ignore_prompt_manager_optional_params
|
||||
else {}
|
||||
),
|
||||
}
|
||||
|
||||
if not ignore_prompt_manager_model:
|
||||
model = self._get_model_from_prompt(
|
||||
prompt_management_client=prompt_template, model=model
|
||||
)
|
||||
else:
|
||||
model = model
|
||||
|
||||
return model, completed_messages, updated_non_default_params
|
||||
|
||||
def get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
|
|
@ -91,6 +173,7 @@ class PromptManagementBase(ABC):
|
|||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
|
|
@ -100,7 +183,9 @@ class PromptManagementBase(ABC):
|
|||
if prompt_id is None:
|
||||
raise ValueError("prompt_id is required for Prompt Management Base class")
|
||||
if not self.should_run_prompt_management(
|
||||
prompt_id=prompt_id, dynamic_callback_params=dynamic_callback_params
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
):
|
||||
return model, messages, non_default_params
|
||||
|
||||
|
|
@ -113,26 +198,53 @@ class PromptManagementBase(ABC):
|
|||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
completed_messages = prompt_template["completed_messages"] or messages
|
||||
|
||||
prompt_template_optional_params = (
|
||||
prompt_template["prompt_template_optional_params"] or {}
|
||||
return self.post_compile_prompt_processing(
|
||||
prompt_template=prompt_template,
|
||||
messages=messages,
|
||||
non_default_params=non_default_params,
|
||||
model=model,
|
||||
ignore_prompt_manager_model=ignore_prompt_manager_model,
|
||||
ignore_prompt_manager_optional_params=ignore_prompt_manager_optional_params,
|
||||
)
|
||||
|
||||
if not ignore_prompt_manager_optional_params:
|
||||
updated_non_default_params = {
|
||||
**non_default_params,
|
||||
**prompt_template_optional_params,
|
||||
}
|
||||
else:
|
||||
updated_non_default_params = non_default_params
|
||||
async def async_get_chat_completion_prompt(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
litellm_logging_obj: "LiteLLMLoggingObj",
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
ignore_prompt_manager_model: Optional[bool] = False,
|
||||
ignore_prompt_manager_optional_params: Optional[bool] = False,
|
||||
) -> Tuple[str, List[AllMessageValues], dict]:
|
||||
if not self.should_run_prompt_management(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
):
|
||||
return model, messages, non_default_params
|
||||
|
||||
if not ignore_prompt_manager_model:
|
||||
model = self._get_model_from_prompt(
|
||||
prompt_management_client=prompt_template, model=model
|
||||
)
|
||||
else:
|
||||
model = model
|
||||
prompt_template = await self.async_compile_prompt(
|
||||
prompt_id=prompt_id,
|
||||
prompt_variables=prompt_variables,
|
||||
client_messages=messages,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_label=prompt_label,
|
||||
prompt_version=prompt_version,
|
||||
)
|
||||
|
||||
|
||||
return model, completed_messages, updated_non_default_params
|
||||
return self.post_compile_prompt_processing(
|
||||
prompt_template=prompt_template,
|
||||
messages=messages,
|
||||
non_default_params=non_default_params,
|
||||
model=model,
|
||||
ignore_prompt_manager_model=ignore_prompt_manager_model,
|
||||
ignore_prompt_manager_optional_params=ignore_prompt_manager_optional_params,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -85,6 +85,7 @@ from litellm.types.llms.openai import (
|
|||
ResponsesAPIResponse,
|
||||
)
|
||||
from litellm.types.mcp import MCPPostCallResponseObject
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
from litellm.types.rerank import RerankResponse
|
||||
from litellm.types.utils import (
|
||||
CachingDetails,
|
||||
|
|
@ -265,6 +266,7 @@ def _get_cached_prometheus_logger():
|
|||
global _PrometheusLogger
|
||||
if _PrometheusLogger is None:
|
||||
from litellm.integrations.prometheus import PrometheusLogger
|
||||
|
||||
_PrometheusLogger = PrometheusLogger
|
||||
return _PrometheusLogger
|
||||
|
||||
|
|
@ -601,8 +603,9 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: Dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
prompt_id: Optional[str] = None,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_management_logger: Optional[CustomLogger] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
prompt_version: Optional[int] = None,
|
||||
|
|
@ -613,6 +616,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
model=model,
|
||||
non_default_params=non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
dynamic_callback_params=self.standard_callback_dynamic_params,
|
||||
)
|
||||
)
|
||||
|
|
@ -627,6 +631,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
messages=messages,
|
||||
non_default_params=non_default_params or {},
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=self.standard_callback_dynamic_params,
|
||||
prompt_label=prompt_label,
|
||||
|
|
@ -640,8 +645,9 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
non_default_params: Dict,
|
||||
prompt_id: Optional[str],
|
||||
prompt_variables: Optional[dict],
|
||||
prompt_id: Optional[str] = None,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
prompt_management_logger: Optional[CustomLogger] = None,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_label: Optional[str] = None,
|
||||
|
|
@ -654,6 +660,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
tools=tools,
|
||||
non_default_params=non_default_params,
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
dynamic_callback_params=self.standard_callback_dynamic_params,
|
||||
)
|
||||
)
|
||||
|
|
@ -668,6 +675,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
messages=messages,
|
||||
non_default_params=non_default_params or {},
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
prompt_variables=prompt_variables,
|
||||
dynamic_callback_params=self.standard_callback_dynamic_params,
|
||||
litellm_logging_obj=self,
|
||||
|
|
@ -681,6 +689,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
def _auto_detect_prompt_management_logger(
|
||||
self,
|
||||
prompt_id: str,
|
||||
prompt_spec: Optional[PromptSpec],
|
||||
dynamic_callback_params: StandardCallbackDynamicParams,
|
||||
) -> Optional[CustomLogger]:
|
||||
"""
|
||||
|
|
@ -706,6 +715,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
try:
|
||||
if logger.should_run_prompt_management(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
):
|
||||
self.model_call_details["prompt_integration"] = (
|
||||
|
|
@ -724,6 +734,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
non_default_params: Dict,
|
||||
tools: Optional[List[Dict]] = None,
|
||||
prompt_id: Optional[str] = None,
|
||||
prompt_spec: Optional[PromptSpec] = None,
|
||||
dynamic_callback_params: Optional[StandardCallbackDynamicParams] = None,
|
||||
) -> Optional[CustomLogger]:
|
||||
"""
|
||||
|
|
@ -756,6 +767,7 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
if prompt_id and dynamic_callback_params is not None:
|
||||
auto_detected_logger = self._auto_detect_prompt_management_logger(
|
||||
prompt_id=prompt_id,
|
||||
prompt_spec=prompt_spec,
|
||||
dynamic_callback_params=dynamic_callback_params,
|
||||
)
|
||||
if auto_detected_logger is not None:
|
||||
|
|
@ -3516,7 +3528,7 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
|
|||
return _literalai_logger # type: ignore
|
||||
elif logging_integration == "prometheus":
|
||||
PrometheusLogger = _get_cached_prometheus_logger()
|
||||
|
||||
|
||||
for callback in _in_memory_loggers:
|
||||
if isinstance(callback, PrometheusLogger):
|
||||
return callback # type: ignore
|
||||
|
|
@ -3835,9 +3847,7 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
|
|||
_in_memory_loggers.append(_otel_logger)
|
||||
return _otel_logger # type: ignore
|
||||
elif logging_integration == "weave_otel":
|
||||
from litellm.integrations.opentelemetry import (
|
||||
OpenTelemetryConfig,
|
||||
)
|
||||
from litellm.integrations.opentelemetry import OpenTelemetryConfig
|
||||
from litellm.integrations.weave.weave_otel import (
|
||||
WeaveOtelLogger,
|
||||
get_weave_otel_config,
|
||||
|
|
|
|||
312
litellm/llms/base_llm/files/azure_blob_storage_backend.py
Normal file
312
litellm/llms/base_llm/files/azure_blob_storage_backend.py
Normal file
|
|
@ -0,0 +1,312 @@
|
|||
"""
|
||||
Azure Blob Storage backend implementation for file storage.
|
||||
|
||||
This module implements the Azure Blob Storage backend for storing files
|
||||
in Azure Data Lake Storage Gen2. It inherits from AzureBlobStorageLogger
|
||||
to reuse all authentication and Azure Storage operations.
|
||||
"""
|
||||
|
||||
import time
|
||||
from typing import Optional
|
||||
from urllib.parse import quote
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm._uuid import uuid
|
||||
|
||||
from .storage_backend import BaseFileStorageBackend
|
||||
from litellm.integrations.azure_storage.azure_storage import AzureBlobStorageLogger
|
||||
|
||||
|
||||
class AzureBlobStorageBackend(BaseFileStorageBackend, AzureBlobStorageLogger):
|
||||
"""
|
||||
Azure Blob Storage backend implementation.
|
||||
|
||||
Inherits from AzureBlobStorageLogger to reuse:
|
||||
- Authentication (account key and Azure AD)
|
||||
- Service client management
|
||||
- Token management
|
||||
- All Azure Storage helper methods
|
||||
|
||||
Reads configuration from the same environment variables as AzureBlobStorageLogger.
|
||||
"""
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
"""
|
||||
Initialize Azure Blob Storage backend.
|
||||
|
||||
Inherits all functionality from AzureBlobStorageLogger which handles:
|
||||
- Reading environment variables
|
||||
- Authentication (account key and Azure AD)
|
||||
- Service client management
|
||||
- Token management
|
||||
|
||||
Environment variables (same as AzureBlobStorageLogger):
|
||||
- AZURE_STORAGE_ACCOUNT_NAME (required)
|
||||
- AZURE_STORAGE_FILE_SYSTEM (required)
|
||||
- AZURE_STORAGE_ACCOUNT_KEY (optional, if using account key auth)
|
||||
- AZURE_STORAGE_TENANT_ID (optional, if using Azure AD)
|
||||
- AZURE_STORAGE_CLIENT_ID (optional, if using Azure AD)
|
||||
- AZURE_STORAGE_CLIENT_SECRET (optional, if using Azure AD)
|
||||
|
||||
Note: We skip periodic_flush since we're not using this as a logger.
|
||||
"""
|
||||
# Initialize AzureBlobStorageLogger (handles all auth and config)
|
||||
AzureBlobStorageLogger.__init__(self, **kwargs)
|
||||
|
||||
# Disable logging functionality - we're only using this for file storage
|
||||
# The periodic_flush task will be created but will do nothing since we override it
|
||||
|
||||
async def periodic_flush(self):
|
||||
"""
|
||||
Override to do nothing - we're not using this as a logger.
|
||||
This prevents the periodic flush task from doing any work.
|
||||
"""
|
||||
# Do nothing - this class is used for file storage, not logging
|
||||
return
|
||||
|
||||
async def async_log_success_event(self, *args, **kwargs):
|
||||
"""
|
||||
Override to do nothing - we're not using this as a logger.
|
||||
"""
|
||||
# Do nothing - this class is used for file storage, not logging
|
||||
pass
|
||||
|
||||
async def async_log_failure_event(self, *args, **kwargs):
|
||||
"""
|
||||
Override to do nothing - we're not using this as a logger.
|
||||
"""
|
||||
# Do nothing - this class is used for file storage, not logging
|
||||
pass
|
||||
|
||||
def _generate_file_name(
|
||||
self, original_filename: str, file_naming_strategy: str
|
||||
) -> str:
|
||||
"""Generate file name based on naming strategy."""
|
||||
if file_naming_strategy == "original_filename":
|
||||
# Use original filename, but sanitize it
|
||||
return quote(original_filename, safe="")
|
||||
elif file_naming_strategy == "timestamp":
|
||||
# Use timestamp
|
||||
extension = original_filename.split(".")[-1] if "." in original_filename else ""
|
||||
timestamp = int(time.time() * 1000) # milliseconds
|
||||
return f"{timestamp}.{extension}" if extension else str(timestamp)
|
||||
else: # default to "uuid"
|
||||
# Use UUID
|
||||
extension = original_filename.split(".")[-1] if "." in original_filename else ""
|
||||
file_uuid = str(uuid.uuid4())
|
||||
return f"{file_uuid}.{extension}" if extension else file_uuid
|
||||
|
||||
async def upload_file(
|
||||
self,
|
||||
file_content: bytes,
|
||||
filename: str,
|
||||
content_type: str,
|
||||
path_prefix: Optional[str] = None,
|
||||
file_naming_strategy: str = "uuid",
|
||||
) -> str:
|
||||
"""
|
||||
Upload a file to Azure Blob Storage.
|
||||
|
||||
Returns the blob URL in format: https://{account}.blob.core.windows.net/{container}/{path}
|
||||
"""
|
||||
try:
|
||||
# Generate file name
|
||||
file_name = self._generate_file_name(filename, file_naming_strategy)
|
||||
|
||||
# Build full path
|
||||
if path_prefix:
|
||||
# Remove leading/trailing slashes and normalize
|
||||
prefix = path_prefix.strip("/")
|
||||
full_path = f"{prefix}/{file_name}"
|
||||
else:
|
||||
full_path = file_name
|
||||
|
||||
if self.azure_storage_account_key:
|
||||
# Use Azure SDK with account key (reuse logger's method)
|
||||
storage_url = await self._upload_file_with_account_key(
|
||||
file_content=file_content,
|
||||
full_path=full_path,
|
||||
)
|
||||
else:
|
||||
# Use REST API with Azure AD token (reuse logger's methods)
|
||||
storage_url = await self._upload_file_with_azure_ad(
|
||||
file_content=file_content,
|
||||
full_path=full_path,
|
||||
)
|
||||
|
||||
verbose_logger.debug(
|
||||
f"Successfully uploaded file to Azure Blob Storage: {storage_url}"
|
||||
)
|
||||
return storage_url
|
||||
|
||||
except Exception as e:
|
||||
verbose_logger.exception(f"Error uploading file to Azure Blob Storage: {str(e)}")
|
||||
raise
|
||||
|
||||
async def _upload_file_with_account_key(
|
||||
self, file_content: bytes, full_path: str
|
||||
) -> str:
|
||||
"""Upload file using Azure SDK with account key authentication."""
|
||||
# Reuse the logger's service client method
|
||||
service_client = await self.get_service_client()
|
||||
file_system_client = service_client.get_file_system_client(
|
||||
file_system=self.azure_storage_file_system
|
||||
)
|
||||
|
||||
# Create filesystem (container) if it doesn't exist
|
||||
if not await file_system_client.exists():
|
||||
await file_system_client.create_file_system()
|
||||
verbose_logger.debug(f"Created filesystem: {self.azure_storage_file_system}")
|
||||
|
||||
# Extract directory and filename (similar to logger's pattern)
|
||||
path_parts = full_path.split("/")
|
||||
if len(path_parts) > 1:
|
||||
directory_path = "/".join(path_parts[:-1])
|
||||
file_name = path_parts[-1]
|
||||
|
||||
# Create directory if needed (like logger does)
|
||||
directory_client = file_system_client.get_directory_client(directory_path)
|
||||
if not await directory_client.exists():
|
||||
await directory_client.create_directory()
|
||||
verbose_logger.debug(f"Created directory: {directory_path}")
|
||||
|
||||
# Get file client from directory (same pattern as logger)
|
||||
file_client = directory_client.get_file_client(file_name)
|
||||
else:
|
||||
# No directory, create file directly in root
|
||||
file_client = file_system_client.get_file_client(full_path)
|
||||
|
||||
# Create, append, and flush (same pattern as logger's upload_to_azure_data_lake_with_azure_account_key)
|
||||
await file_client.create_file()
|
||||
await file_client.append_data(data=file_content, offset=0, length=len(file_content))
|
||||
await file_client.flush_data(position=len(file_content), offset=0)
|
||||
|
||||
# Return blob URL (not DFS URL)
|
||||
blob_url = f"https://{self.azure_storage_account_name}.blob.core.windows.net/{self.azure_storage_file_system}/{full_path}"
|
||||
return blob_url
|
||||
|
||||
async def _upload_file_with_azure_ad(
|
||||
self, file_content: bytes, full_path: str
|
||||
) -> str:
|
||||
"""Upload file using REST API with Azure AD authentication."""
|
||||
# Reuse the logger's token management
|
||||
await self.set_valid_azure_ad_token()
|
||||
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
get_async_httpx_client,
|
||||
httpxSpecialProvider,
|
||||
)
|
||||
|
||||
async_client = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.LoggingCallback
|
||||
)
|
||||
|
||||
# Use DFS endpoint for upload
|
||||
base_url = f"https://{self.azure_storage_account_name}.dfs.core.windows.net/{self.azure_storage_file_system}/{full_path}"
|
||||
|
||||
# Execute 3-step upload process: create, append, flush
|
||||
# Reuse the logger's helper methods
|
||||
await self._create_file(async_client, base_url)
|
||||
# Append data - logger's _append_data expects string, so we create our own for bytes
|
||||
await self._append_data_bytes(async_client, base_url, file_content)
|
||||
await self._flush_data(async_client, base_url, len(file_content))
|
||||
|
||||
# Return blob URL (not DFS URL)
|
||||
blob_url = f"https://{self.azure_storage_account_name}.blob.core.windows.net/{self.azure_storage_file_system}/{full_path}"
|
||||
return blob_url
|
||||
|
||||
async def _append_data_bytes(
|
||||
self, client, base_url: str, file_content: bytes
|
||||
):
|
||||
"""Append binary data to file using REST API."""
|
||||
from litellm.constants import AZURE_STORAGE_MSFT_VERSION
|
||||
|
||||
headers = {
|
||||
"x-ms-version": AZURE_STORAGE_MSFT_VERSION,
|
||||
"Content-Type": "application/octet-stream",
|
||||
"Authorization": f"Bearer {self.azure_auth_token}",
|
||||
}
|
||||
response = await client.patch(
|
||||
f"{base_url}?action=append&position=0",
|
||||
headers=headers,
|
||||
content=file_content,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def download_file(self, storage_url: str) -> bytes:
|
||||
"""
|
||||
Download a file from Azure Blob Storage.
|
||||
|
||||
Args:
|
||||
storage_url: Blob URL in format: https://{account}.blob.core.windows.net/{container}/{path}
|
||||
|
||||
Returns:
|
||||
bytes: File content
|
||||
"""
|
||||
try:
|
||||
# Parse blob URL to extract path
|
||||
# URL format: https://{account}.blob.core.windows.net/{container}/{path}
|
||||
if ".blob.core.windows.net/" not in storage_url:
|
||||
raise ValueError(f"Invalid Azure Blob Storage URL: {storage_url}")
|
||||
|
||||
# Extract path after container name
|
||||
container_and_path = storage_url.split(".blob.core.windows.net/", 1)[1]
|
||||
path_parts = container_and_path.split("/", 1)
|
||||
if len(path_parts) < 2:
|
||||
raise ValueError(f"Invalid Azure Blob Storage URL format: {storage_url}")
|
||||
file_path = path_parts[1] # Path after container name
|
||||
|
||||
if self.azure_storage_account_key:
|
||||
# Use Azure SDK (reuse logger's service client)
|
||||
return await self._download_file_with_account_key(file_path)
|
||||
else:
|
||||
# Use REST API (reuse logger's token management)
|
||||
return await self._download_file_with_azure_ad(file_path)
|
||||
|
||||
except Exception as e:
|
||||
verbose_logger.exception(f"Error downloading file from Azure Blob Storage: {str(e)}")
|
||||
raise
|
||||
|
||||
async def _download_file_with_account_key(self, file_path: str) -> bytes:
|
||||
"""Download file using Azure SDK with account key."""
|
||||
# Reuse the logger's service client method
|
||||
service_client = await self.get_service_client()
|
||||
file_system_client = service_client.get_file_system_client(
|
||||
file_system=self.azure_storage_file_system
|
||||
)
|
||||
# Ensure filesystem exists (should already exist, but check for safety)
|
||||
if not await file_system_client.exists():
|
||||
raise ValueError(f"Filesystem {self.azure_storage_file_system} does not exist")
|
||||
file_client = file_system_client.get_file_client(file_path)
|
||||
# Download file
|
||||
download_response = await file_client.download_file()
|
||||
file_content = await download_response.readall()
|
||||
return file_content
|
||||
|
||||
async def _download_file_with_azure_ad(self, file_path: str) -> bytes:
|
||||
"""Download file using REST API with Azure AD token."""
|
||||
# Reuse the logger's token management
|
||||
await self.set_valid_azure_ad_token()
|
||||
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
get_async_httpx_client,
|
||||
httpxSpecialProvider,
|
||||
)
|
||||
from litellm.constants import AZURE_STORAGE_MSFT_VERSION
|
||||
|
||||
async_client = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.LoggingCallback
|
||||
)
|
||||
|
||||
# Use blob endpoint for download (simpler than DFS)
|
||||
blob_url = f"https://{self.azure_storage_account_name}.blob.core.windows.net/{self.azure_storage_file_system}/{file_path}"
|
||||
|
||||
headers = {
|
||||
"x-ms-version": AZURE_STORAGE_MSFT_VERSION,
|
||||
"Authorization": f"Bearer {self.azure_auth_token}",
|
||||
}
|
||||
|
||||
response = await async_client.get(blob_url, headers=headers)
|
||||
response.raise_for_status()
|
||||
return response.content
|
||||
|
||||
79
litellm/llms/base_llm/files/storage_backend.py
Normal file
79
litellm/llms/base_llm/files/storage_backend.py
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
"""
|
||||
Base storage backend interface for file storage backends.
|
||||
|
||||
This module defines the abstract base class that all file storage backends
|
||||
(e.g., Azure Blob Storage, S3, GCS) must implement.
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class BaseFileStorageBackend(ABC):
|
||||
"""
|
||||
Abstract base class for file storage backends.
|
||||
|
||||
All storage backends (Azure Blob Storage, S3, GCS, etc.) must implement
|
||||
these methods to provide a consistent interface for file operations.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
async def upload_file(
|
||||
self,
|
||||
file_content: bytes,
|
||||
filename: str,
|
||||
content_type: str,
|
||||
path_prefix: Optional[str] = None,
|
||||
file_naming_strategy: str = "uuid",
|
||||
) -> str:
|
||||
"""
|
||||
Upload a file to the storage backend.
|
||||
|
||||
Args:
|
||||
file_content: The file content as bytes
|
||||
filename: Original filename (may be used for naming strategy)
|
||||
content_type: MIME type of the file
|
||||
path_prefix: Optional path prefix for organizing files
|
||||
file_naming_strategy: Strategy for naming files ("uuid", "timestamp", "original_filename")
|
||||
|
||||
Returns:
|
||||
str: The storage URL where the file can be accessed/downloaded
|
||||
|
||||
Raises:
|
||||
Exception: If upload fails
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def download_file(self, storage_url: str) -> bytes:
|
||||
"""
|
||||
Download a file from the storage backend.
|
||||
|
||||
Args:
|
||||
storage_url: The storage URL returned from upload_file
|
||||
|
||||
Returns:
|
||||
bytes: The file content
|
||||
|
||||
Raises:
|
||||
Exception: If download fails
|
||||
"""
|
||||
pass
|
||||
|
||||
async def delete_file(self, storage_url: str) -> None:
|
||||
"""
|
||||
Delete a file from the storage backend.
|
||||
|
||||
This is optional and can be overridden by backends that support deletion.
|
||||
Default implementation does nothing.
|
||||
|
||||
Args:
|
||||
storage_url: The storage URL of the file to delete
|
||||
|
||||
Raises:
|
||||
Exception: If deletion fails
|
||||
"""
|
||||
# Default implementation: no-op
|
||||
# Backends can override if they support deletion
|
||||
pass
|
||||
|
||||
41
litellm/llms/base_llm/files/storage_backend_factory.py
Normal file
41
litellm/llms/base_llm/files/storage_backend_factory.py
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
"""
|
||||
Factory for creating storage backend instances.
|
||||
|
||||
This module provides a factory function to instantiate the correct storage backend
|
||||
based on the backend type. Backends use the same configuration as their corresponding
|
||||
callbacks (e.g., azure_storage uses the same env vars as AzureBlobStorageLogger).
|
||||
"""
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
|
||||
from .azure_blob_storage_backend import AzureBlobStorageBackend
|
||||
from .storage_backend import BaseFileStorageBackend
|
||||
|
||||
|
||||
def get_storage_backend(backend_type: str) -> BaseFileStorageBackend:
|
||||
"""
|
||||
Factory function to create a storage backend instance.
|
||||
|
||||
Backends are configured using the same environment variables as their
|
||||
corresponding callbacks. For example, "azure_storage" uses the same
|
||||
env vars as AzureBlobStorageLogger.
|
||||
|
||||
Args:
|
||||
backend_type: Backend type identifier (e.g., "azure_storage")
|
||||
|
||||
Returns:
|
||||
BaseFileStorageBackend: Instance of the appropriate storage backend
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is not supported
|
||||
"""
|
||||
verbose_logger.debug(f"Creating storage backend: type={backend_type}")
|
||||
|
||||
if backend_type == "azure_storage":
|
||||
return AzureBlobStorageBackend()
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Unsupported storage backend type: {backend_type}. "
|
||||
f"Supported types: azure_storage"
|
||||
)
|
||||
|
||||
|
|
@ -5,7 +5,7 @@ Handles Server-Sent Events (SSE) streaming responses from AgentCore.
|
|||
"""
|
||||
|
||||
import json
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, Any, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
|
|
@ -19,262 +19,234 @@ if TYPE_CHECKING:
|
|||
|
||||
|
||||
class AgentCoreSSEStreamIterator:
|
||||
"""Iterator for AgentCore SSE streaming responses. Supports both sync and async iteration."""
|
||||
"""
|
||||
Iterator for AgentCore SSE streaming responses.
|
||||
Supports both sync and async iteration.
|
||||
|
||||
CRITICAL: The line iterators are created lazily on first access and reused.
|
||||
We must NOT create new iterators in __aiter__/__iter__ because
|
||||
CustomStreamWrapper calls __aiter__ on every call to its __anext__,
|
||||
which would create new iterators and cause StreamConsumed errors.
|
||||
"""
|
||||
|
||||
def __init__(self, response: httpx.Response, model: str):
|
||||
self.response = response
|
||||
self.model = model
|
||||
self.finished = False
|
||||
self.line_iterator = None
|
||||
self.async_line_iterator = None
|
||||
self._sync_iter: Any = None
|
||||
self._async_iter: Any = None
|
||||
self._sync_iter_initialized = False
|
||||
self._async_iter_initialized = False
|
||||
|
||||
def __iter__(self):
|
||||
"""Initialize sync iteration."""
|
||||
self.line_iterator = self.response.iter_lines()
|
||||
"""Initialize sync iteration - create iterator lazily on first call only."""
|
||||
if not self._sync_iter_initialized:
|
||||
self._sync_iter = iter(self.response.iter_lines())
|
||||
self._sync_iter_initialized = True
|
||||
return self
|
||||
|
||||
def __aiter__(self):
|
||||
"""Initialize async iteration."""
|
||||
self.async_line_iterator = self.response.aiter_lines()
|
||||
"""Initialize async iteration - create iterator lazily on first call only."""
|
||||
if not self._async_iter_initialized:
|
||||
self._async_iter = self.response.aiter_lines().__aiter__()
|
||||
self._async_iter_initialized = True
|
||||
return self
|
||||
|
||||
def __next__(self) -> ModelResponse:
|
||||
"""Sync iteration - parse SSE events and yield ModelResponse chunks."""
|
||||
def _parse_sse_line(self, line: str) -> Optional[ModelResponse]:
|
||||
"""
|
||||
Parse a single SSE line and return a ModelResponse chunk if applicable.
|
||||
|
||||
AgentCore SSE format:
|
||||
- data: {"event": {"contentBlockDelta": {"delta": {"text": "..."}}}}
|
||||
- data: {"event": {"metadata": {"usage": {...}}}}
|
||||
- data: {"message": {...}}
|
||||
"""
|
||||
line = line.strip()
|
||||
if not line or not line.startswith("data:"):
|
||||
return None
|
||||
|
||||
json_str = line[5:].strip()
|
||||
if not json_str:
|
||||
return None
|
||||
|
||||
try:
|
||||
if self.line_iterator is None:
|
||||
data = json.loads(json_str)
|
||||
|
||||
# Skip non-dict data (some lines contain Python repr strings)
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
|
||||
# Process content delta events
|
||||
if "event" in data and isinstance(data["event"], dict):
|
||||
event_payload = data["event"]
|
||||
content_block_delta = event_payload.get("contentBlockDelta")
|
||||
|
||||
if content_block_delta:
|
||||
delta = content_block_delta.get("delta", {})
|
||||
text = delta.get("text", "")
|
||||
|
||||
if text:
|
||||
# Return chunk with text
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason=None,
|
||||
index=0,
|
||||
delta=Delta(content=text, role="assistant"),
|
||||
)
|
||||
]
|
||||
|
||||
return chunk
|
||||
|
||||
# Check for metadata/usage - this signals the end
|
||||
metadata = event_payload.get("metadata")
|
||||
if metadata and "usage" in metadata:
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
usage_data: AgentCoreUsage = metadata["usage"] # type: ignore
|
||||
setattr(
|
||||
chunk,
|
||||
"usage",
|
||||
Usage(
|
||||
prompt_tokens=usage_data.get("inputTokens", 0),
|
||||
completion_tokens=usage_data.get("outputTokens", 0),
|
||||
total_tokens=usage_data.get("totalTokens", 0),
|
||||
),
|
||||
)
|
||||
|
||||
self.finished = True
|
||||
return chunk
|
||||
|
||||
# Check for final message (alternative finish signal)
|
||||
if "message" in data and isinstance(data["message"], dict):
|
||||
if not self.finished:
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
self.finished = True
|
||||
return chunk
|
||||
|
||||
except json.JSONDecodeError:
|
||||
verbose_logger.debug(f"Skipping non-JSON SSE line: {line[:100]}")
|
||||
|
||||
return None
|
||||
|
||||
def _create_final_chunk(self) -> ModelResponse:
|
||||
"""Create a final chunk to signal stream completion."""
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
return chunk
|
||||
|
||||
def __next__(self) -> ModelResponse:
|
||||
"""
|
||||
Sync iteration - parse SSE events and yield ModelResponse chunks.
|
||||
|
||||
Uses next() on the stored iterator to properly resume between calls.
|
||||
"""
|
||||
try:
|
||||
if self._sync_iter is None:
|
||||
raise StopIteration
|
||||
for line in self.line_iterator:
|
||||
line = line.strip()
|
||||
|
||||
if not line or not line.startswith('data:'):
|
||||
continue
|
||||
|
||||
# Extract JSON from SSE line
|
||||
json_str = line[5:].strip()
|
||||
if not json_str:
|
||||
continue
|
||||
|
||||
|
||||
# Keep getting lines until we have a result to return
|
||||
while True:
|
||||
try:
|
||||
data = json.loads(json_str)
|
||||
|
||||
# Skip non-dict data
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
|
||||
# Process content delta events
|
||||
if "event" in data and isinstance(data["event"], dict):
|
||||
event_payload = data["event"]
|
||||
content_block_delta = event_payload.get("contentBlockDelta")
|
||||
|
||||
if content_block_delta:
|
||||
delta = content_block_delta.get("delta", {})
|
||||
text = delta.get("text", "")
|
||||
|
||||
if text:
|
||||
# Yield chunk with text
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason=None,
|
||||
index=0,
|
||||
delta=Delta(content=text, role="assistant"),
|
||||
)
|
||||
]
|
||||
|
||||
return chunk
|
||||
|
||||
# Check for metadata/usage
|
||||
metadata = event_payload.get("metadata")
|
||||
if metadata and "usage" in metadata:
|
||||
# This is the final chunk with usage
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
usage_data: AgentCoreUsage = metadata["usage"] # type: ignore
|
||||
setattr(chunk, "usage", Usage(
|
||||
prompt_tokens=usage_data.get("inputTokens", 0),
|
||||
completion_tokens=usage_data.get("outputTokens", 0),
|
||||
total_tokens=usage_data.get("totalTokens", 0),
|
||||
))
|
||||
|
||||
self.finished = True
|
||||
return chunk
|
||||
|
||||
# Check for final message (alternative finish signal)
|
||||
if "message" in data and isinstance(data["message"], dict):
|
||||
if not self.finished:
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
self.finished = True
|
||||
return chunk
|
||||
|
||||
except json.JSONDecodeError:
|
||||
verbose_logger.debug(f"Skipping non-JSON SSE line: {line[:100]}")
|
||||
continue
|
||||
|
||||
# Stream ended naturally
|
||||
raise StopIteration
|
||||
line = next(self._sync_iter)
|
||||
except StopIteration:
|
||||
# Stream ended - send final chunk if not already finished
|
||||
if not self.finished:
|
||||
self.finished = True
|
||||
return self._create_final_chunk()
|
||||
raise
|
||||
|
||||
result = self._parse_sse_line(line)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
except StopIteration:
|
||||
raise
|
||||
except httpx.StreamConsumed:
|
||||
# This is expected when the stream has been fully consumed
|
||||
raise StopIteration
|
||||
except httpx.StreamClosed:
|
||||
# This is expected when the stream is closed
|
||||
raise StopIteration
|
||||
except Exception as e:
|
||||
verbose_logger.error(f"Error in AgentCore SSE stream: {str(e)}")
|
||||
raise StopIteration
|
||||
|
||||
async def __anext__(self) -> ModelResponse:
|
||||
"""Async iteration - parse SSE events and yield ModelResponse chunks."""
|
||||
"""
|
||||
Async iteration - parse SSE events and yield ModelResponse chunks.
|
||||
|
||||
Uses __anext__() on the stored iterator to properly resume between calls.
|
||||
"""
|
||||
try:
|
||||
if self.async_line_iterator is None:
|
||||
if self._async_iter is None:
|
||||
raise StopAsyncIteration
|
||||
async for line in self.async_line_iterator:
|
||||
line = line.strip()
|
||||
|
||||
if not line or not line.startswith('data:'):
|
||||
continue
|
||||
|
||||
# Extract JSON from SSE line
|
||||
json_str = line[5:].strip()
|
||||
if not json_str:
|
||||
continue
|
||||
|
||||
|
||||
# Keep getting lines until we have a result to return
|
||||
while True:
|
||||
try:
|
||||
data = json.loads(json_str)
|
||||
|
||||
# Skip non-dict data
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
|
||||
# Process content delta events
|
||||
if "event" in data and isinstance(data["event"], dict):
|
||||
event_payload = data["event"]
|
||||
content_block_delta = event_payload.get("contentBlockDelta")
|
||||
|
||||
if content_block_delta:
|
||||
delta = content_block_delta.get("delta", {})
|
||||
text = delta.get("text", "")
|
||||
|
||||
if text:
|
||||
# Yield chunk with text
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason=None,
|
||||
index=0,
|
||||
delta=Delta(content=text, role="assistant"),
|
||||
)
|
||||
]
|
||||
|
||||
return chunk
|
||||
|
||||
# Check for metadata/usage
|
||||
metadata = event_payload.get("metadata")
|
||||
if metadata and "usage" in metadata:
|
||||
# This is the final chunk with usage
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
usage_data: AgentCoreUsage = metadata["usage"] # type: ignore
|
||||
setattr(chunk, "usage", Usage(
|
||||
prompt_tokens=usage_data.get("inputTokens", 0),
|
||||
completion_tokens=usage_data.get("outputTokens", 0),
|
||||
total_tokens=usage_data.get("totalTokens", 0),
|
||||
))
|
||||
|
||||
self.finished = True
|
||||
return chunk
|
||||
|
||||
# Check for final message (alternative finish signal)
|
||||
if "message" in data and isinstance(data["message"], dict):
|
||||
if not self.finished:
|
||||
chunk = ModelResponse(
|
||||
id=f"chatcmpl-{uuid.uuid4()}",
|
||||
created=0,
|
||||
model=self.model,
|
||||
object="chat.completion.chunk",
|
||||
)
|
||||
|
||||
chunk.choices = [
|
||||
StreamingChoices(
|
||||
finish_reason="stop",
|
||||
index=0,
|
||||
delta=Delta(),
|
||||
)
|
||||
]
|
||||
|
||||
self.finished = True
|
||||
return chunk
|
||||
|
||||
except json.JSONDecodeError:
|
||||
verbose_logger.debug(f"Skipping non-JSON SSE line: {line[:100]}")
|
||||
continue
|
||||
|
||||
# Stream ended naturally
|
||||
raise StopAsyncIteration
|
||||
line = await self._async_iter.__anext__()
|
||||
except StopAsyncIteration:
|
||||
# Stream ended - send final chunk if not already finished
|
||||
if not self.finished:
|
||||
self.finished = True
|
||||
return self._create_final_chunk()
|
||||
raise
|
||||
|
||||
result = self._parse_sse_line(line)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
except StopAsyncIteration:
|
||||
raise
|
||||
except httpx.StreamConsumed:
|
||||
# This is expected when the stream has been fully consumed
|
||||
raise StopAsyncIteration
|
||||
except httpx.StreamClosed:
|
||||
# This is expected when the stream is closed
|
||||
raise StopAsyncIteration
|
||||
except Exception as e:
|
||||
verbose_logger.error(f"Error in AgentCore SSE stream: {str(e)}")
|
||||
raise StopAsyncIteration
|
||||
|
||||
|
|
|
|||
|
|
@ -286,11 +286,12 @@ class BedrockEmbedding(BaseAWSLLM):
|
|||
"headers": prepped.headers,
|
||||
},
|
||||
)
|
||||
headers_for_request = dict(prepped.headers) if hasattr(prepped, 'headers') else {}
|
||||
response = self._make_sync_call(
|
||||
client=client,
|
||||
timeout=timeout,
|
||||
api_base=prepped.url,
|
||||
headers=prepped.headers, # type: ignore
|
||||
headers=headers_for_request,
|
||||
data=data,
|
||||
)
|
||||
|
||||
|
|
@ -352,11 +353,14 @@ class BedrockEmbedding(BaseAWSLLM):
|
|||
"headers": prepped.headers,
|
||||
},
|
||||
)
|
||||
# Convert CaseInsensitiveDict to regular dict for httpx compatibility
|
||||
# This ensures custom headers are properly forwarded, especially with IAM roles and custom api_base
|
||||
headers_for_request = dict(prepped.headers) if hasattr(prepped, 'headers') else {}
|
||||
response = await self._make_async_call(
|
||||
client=client,
|
||||
timeout=timeout,
|
||||
api_base=prepped.url,
|
||||
headers=prepped.headers, # type: ignore
|
||||
headers=headers_for_request,
|
||||
data=data,
|
||||
)
|
||||
|
||||
|
|
@ -562,6 +566,8 @@ class BedrockEmbedding(BaseAWSLLM):
|
|||
)
|
||||
|
||||
## ROUTING ##
|
||||
# Convert CaseInsensitiveDict to regular dict for httpx compatibility
|
||||
headers_for_request = dict(prepped.headers) if hasattr(prepped, 'headers') else {}
|
||||
return cohere_embedding(
|
||||
model=model,
|
||||
input=input,
|
||||
|
|
@ -575,7 +581,7 @@ class BedrockEmbedding(BaseAWSLLM):
|
|||
aembedding=aembedding,
|
||||
timeout=timeout,
|
||||
client=client,
|
||||
headers=prepped.headers, # type: ignore
|
||||
headers=headers_for_request,
|
||||
)
|
||||
|
||||
async def _get_async_invoke_status(
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ class BedrockRerankHandler(BaseAWSLLM):
|
|||
if client is None:
|
||||
client = get_async_httpx_client(llm_provider=litellm.LlmProviders.BEDROCK)
|
||||
try:
|
||||
response = await client.post(url=prepared_request["endpoint_url"], headers=prepared_request["prepped"].headers, data=prepared_request["body"]) # type: ignore
|
||||
response = await client.post(url=prepared_request["endpoint_url"], headers=dict(prepared_request["prepped"].headers), data=prepared_request["body"])
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as err:
|
||||
error_code = err.response.status_code
|
||||
|
|
@ -84,7 +84,7 @@ class BedrockRerankHandler(BaseAWSLLM):
|
|||
additional_args={
|
||||
"complete_input_dict": data,
|
||||
"api_base": prepared_request["endpoint_url"],
|
||||
"headers": prepared_request["prepped"].headers,
|
||||
"headers": dict(prepared_request["prepped"].headers),
|
||||
},
|
||||
)
|
||||
|
||||
|
|
@ -94,7 +94,7 @@ class BedrockRerankHandler(BaseAWSLLM):
|
|||
if client is None or not isinstance(client, HTTPHandler):
|
||||
client = _get_httpx_client()
|
||||
try:
|
||||
response = client.post(url=prepared_request["endpoint_url"], headers=prepared_request["prepped"].headers, data=prepared_request["body"]) # type: ignore
|
||||
response = client.post(url=prepared_request["endpoint_url"], headers=dict(prepared_request["prepped"].headers), data=prepared_request["body"])
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as err:
|
||||
error_code = err.response.status_code
|
||||
|
|
|
|||
|
|
@ -51,6 +51,12 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
model_name = model.split("/")[-1]
|
||||
return model_name.startswith("gpt-5.2-pro")
|
||||
|
||||
@classmethod
|
||||
def is_model_gpt_5_2_model(cls, model: str) -> bool:
|
||||
"""Check if the model is a gpt-5.2 variant (including pro)."""
|
||||
model_name = model.split("/")[-1]
|
||||
return model_name.startswith("gpt-5.2")
|
||||
|
||||
def get_supported_openai_params(self, model: str) -> list:
|
||||
from litellm.utils import supports_tool_choice
|
||||
|
||||
|
|
@ -89,14 +95,14 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
if reasoning_effort is not None and reasoning_effort == "xhigh":
|
||||
if not (
|
||||
self.is_model_gpt_5_1_codex_max_model(model)
|
||||
or self.is_model_gpt_5_2_pro_model(model)
|
||||
or self.is_model_gpt_5_2_model(model)
|
||||
):
|
||||
if litellm.drop_params or drop_params:
|
||||
non_default_params.pop("reasoning_effort", None)
|
||||
else:
|
||||
raise litellm.utils.UnsupportedParamsError(
|
||||
message=(
|
||||
"reasoning_effort='xhigh' is only supported for gpt-5.1-codex-max."
|
||||
"reasoning_effort='xhigh' is only supported for gpt-5.1-codex-max and gpt-5.2 models."
|
||||
),
|
||||
status_code=400,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import time
|
||||
import types
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
Any,
|
||||
AsyncIterator,
|
||||
Callable,
|
||||
|
|
@ -10,7 +11,6 @@ from typing import (
|
|||
List,
|
||||
Literal,
|
||||
Optional,
|
||||
TYPE_CHECKING,
|
||||
Union,
|
||||
cast,
|
||||
)
|
||||
|
|
@ -20,6 +20,7 @@ import httpx
|
|||
|
||||
if TYPE_CHECKING:
|
||||
from aiohttp import ClientSession
|
||||
|
||||
import openai
|
||||
from openai import AsyncOpenAI, OpenAI
|
||||
from openai.types.beta.assistant_deleted import AssistantDeleted
|
||||
|
|
@ -1549,7 +1550,7 @@ class OpenAIFilesAPI(BaseLLM):
|
|||
create_file_data: CreateFileRequest,
|
||||
openai_client: AsyncOpenAI,
|
||||
) -> OpenAIFileObject:
|
||||
response = await openai_client.files.create(**create_file_data)
|
||||
response = await openai_client.files.create(**create_file_data) # type: ignore[arg-type]
|
||||
return OpenAIFileObject(**response.model_dump())
|
||||
|
||||
def create_file(
|
||||
|
|
@ -1585,7 +1586,7 @@ class OpenAIFilesAPI(BaseLLM):
|
|||
return self.acreate_file( # type: ignore
|
||||
create_file_data=create_file_data, openai_client=openai_client
|
||||
)
|
||||
response = cast(OpenAI, openai_client).files.create(**create_file_data)
|
||||
response = cast(OpenAI, openai_client).files.create(**create_file_data) # type: ignore[arg-type]
|
||||
return OpenAIFileObject(**response.model_dump())
|
||||
|
||||
async def afile_content(
|
||||
|
|
|
|||
|
|
@ -69,6 +69,7 @@ from litellm.constants import (
|
|||
)
|
||||
from litellm.exceptions import LiteLLMUnknownProvider
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.litellm_core_utils.asyncify import run_async_function
|
||||
from litellm.litellm_core_utils.audio_utils.utils import (
|
||||
calculate_request_duration,
|
||||
get_audio_file_for_health_check,
|
||||
|
|
@ -299,7 +300,6 @@ MOCK_RESPONSE_TYPE = Union[str, Exception, dict, ModelResponse, ModelResponseStr
|
|||
|
||||
|
||||
class LiteLLM:
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
|
|
@ -1091,6 +1091,22 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
tools = validate_and_fix_openai_tools(tools=tools)
|
||||
# validate tool_choice
|
||||
tool_choice = validate_chat_completion_tool_choice(tool_choice=tool_choice)
|
||||
|
||||
skip_mcp_handler = kwargs.pop("_skip_mcp_handler", False)
|
||||
if not skip_mcp_handler and tools:
|
||||
from litellm.responses.mcp.chat_completions_handler import (
|
||||
handle_chat_completion_with_mcp,
|
||||
)
|
||||
|
||||
mcp_handler_context = locals().copy()
|
||||
completion_callable = globals().get("acompletion")
|
||||
mcp_result = run_async_function(
|
||||
handle_chat_completion_with_mcp,
|
||||
mcp_handler_context,
|
||||
completion_callable,
|
||||
)
|
||||
if mcp_result is not None:
|
||||
return mcp_result
|
||||
######### unpacking kwargs #####################
|
||||
args = locals()
|
||||
api_base = kwargs.get("api_base", None)
|
||||
|
|
@ -1181,7 +1197,6 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
prompt_id=prompt_id, non_default_params=non_default_params
|
||||
)
|
||||
):
|
||||
|
||||
(
|
||||
model,
|
||||
messages,
|
||||
|
|
@ -2130,7 +2145,7 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
config = litellm.GenAIHubOrchestrationConfig.get_config()
|
||||
for k, v in config.items():
|
||||
if (
|
||||
k not in optional_params
|
||||
k not in optional_params
|
||||
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
|
||||
optional_params[k] = v
|
||||
|
||||
|
|
@ -2300,7 +2315,6 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
|
||||
try:
|
||||
if use_base_llm_http_handler:
|
||||
|
||||
response = base_llm_http_handler.completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
|
|
@ -3413,9 +3427,9 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
"aws_region_name" not in optional_params
|
||||
or optional_params["aws_region_name"] is None
|
||||
):
|
||||
optional_params["aws_region_name"] = (
|
||||
aws_bedrock_client.meta.region_name
|
||||
)
|
||||
optional_params[
|
||||
"aws_region_name"
|
||||
] = aws_bedrock_client.meta.region_name
|
||||
|
||||
bedrock_route = BedrockModelInfo.get_bedrock_route(model)
|
||||
if bedrock_route == "converse":
|
||||
|
|
@ -3633,7 +3647,6 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
if api_key is not None and "Authorization" not in headers:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
|
||||
|
||||
response = base_llm_http_handler.completion(
|
||||
model=model,
|
||||
stream=stream,
|
||||
|
|
@ -3769,7 +3782,6 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
)
|
||||
raise e
|
||||
elif custom_llm_provider == "gradient_ai":
|
||||
|
||||
api_base = litellm.api_base or api_base
|
||||
response = base_llm_http_handler.completion(
|
||||
model=model,
|
||||
|
|
@ -4420,7 +4432,7 @@ def embedding( # noqa: PLR0915
|
|||
litellm_params=litellm_params_dict,
|
||||
)
|
||||
elif custom_llm_provider == "github_copilot":
|
||||
api_key = (api_key or litellm.api_key)
|
||||
api_key = api_key or litellm.api_key
|
||||
response = base_llm_http_handler.embedding(
|
||||
model=model,
|
||||
input=input,
|
||||
|
|
@ -5585,9 +5597,9 @@ def adapter_completion(
|
|||
new_kwargs = translation_obj.translate_completion_input_params(kwargs=kwargs)
|
||||
|
||||
response: Union[ModelResponse, CustomStreamWrapper] = completion(**new_kwargs) # type: ignore
|
||||
translated_response: Optional[Union[BaseModel, AdapterCompletionStreamWrapper]] = (
|
||||
None
|
||||
)
|
||||
translated_response: Optional[
|
||||
Union[BaseModel, AdapterCompletionStreamWrapper]
|
||||
] = None
|
||||
if isinstance(response, ModelResponse):
|
||||
translated_response = translation_obj.translate_completion_output_params(
|
||||
response=response
|
||||
|
|
@ -6292,9 +6304,9 @@ def speech( # noqa: PLR0915
|
|||
ElevenLabsTextToSpeechConfig.ELEVENLABS_QUERY_PARAMS_KEY
|
||||
] = query_params
|
||||
|
||||
litellm_params_dict[ElevenLabsTextToSpeechConfig.ELEVENLABS_VOICE_ID_KEY] = (
|
||||
voice_id
|
||||
)
|
||||
litellm_params_dict[
|
||||
ElevenLabsTextToSpeechConfig.ELEVENLABS_VOICE_ID_KEY
|
||||
] = voice_id
|
||||
|
||||
if api_base is not None:
|
||||
litellm_params_dict["api_base"] = api_base
|
||||
|
|
@ -6344,16 +6356,16 @@ def speech( # noqa: PLR0915
|
|||
text_to_speech_provider_config = VertexAITextToSpeechConfig()
|
||||
|
||||
# Cast to specific Vertex AI config type to access dispatch method
|
||||
vertex_config = cast(
|
||||
VertexAITextToSpeechConfig, text_to_speech_provider_config
|
||||
)
|
||||
vertex_config = cast(VertexAITextToSpeechConfig, text_to_speech_provider_config)
|
||||
|
||||
# Store Vertex AI specific params in litellm_params_dict
|
||||
litellm_params_dict.update({
|
||||
"vertex_project": generic_optional_params.vertex_project,
|
||||
"vertex_location": generic_optional_params.vertex_location,
|
||||
"vertex_credentials": generic_optional_params.vertex_credentials,
|
||||
})
|
||||
litellm_params_dict.update(
|
||||
{
|
||||
"vertex_project": generic_optional_params.vertex_project,
|
||||
"vertex_location": generic_optional_params.vertex_location,
|
||||
"vertex_credentials": generic_optional_params.vertex_credentials,
|
||||
}
|
||||
)
|
||||
|
||||
response = vertex_config.dispatch_text_to_speech(
|
||||
model=model,
|
||||
|
|
@ -6724,9 +6736,9 @@ def stream_chunk_builder( # noqa: PLR0915
|
|||
]
|
||||
|
||||
if len(content_chunks) > 0:
|
||||
response["choices"][0]["message"]["content"] = (
|
||||
processor.get_combined_content(content_chunks)
|
||||
)
|
||||
response["choices"][0]["message"][
|
||||
"content"
|
||||
] = processor.get_combined_content(content_chunks)
|
||||
|
||||
thinking_blocks = [
|
||||
chunk
|
||||
|
|
@ -6737,9 +6749,9 @@ def stream_chunk_builder( # noqa: PLR0915
|
|||
]
|
||||
|
||||
if len(thinking_blocks) > 0:
|
||||
response["choices"][0]["message"]["thinking_blocks"] = (
|
||||
processor.get_combined_thinking_content(thinking_blocks)
|
||||
)
|
||||
response["choices"][0]["message"][
|
||||
"thinking_blocks"
|
||||
] = processor.get_combined_thinking_content(thinking_blocks)
|
||||
|
||||
reasoning_chunks = [
|
||||
chunk
|
||||
|
|
@ -6750,9 +6762,9 @@ def stream_chunk_builder( # noqa: PLR0915
|
|||
]
|
||||
|
||||
if len(reasoning_chunks) > 0:
|
||||
response["choices"][0]["message"]["reasoning_content"] = (
|
||||
processor.get_combined_reasoning_content(reasoning_chunks)
|
||||
)
|
||||
response["choices"][0]["message"][
|
||||
"reasoning_content"
|
||||
] = processor.get_combined_reasoning_content(reasoning_chunks)
|
||||
|
||||
annotation_chunks = [
|
||||
chunk
|
||||
|
|
|
|||
|
|
@ -3424,6 +3424,172 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"azure/gpt-5.2": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
"input_cost_per_token": 1.75e-06,
|
||||
"litellm_provider": "azure",
|
||||
"max_input_tokens": 400000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.4e-05,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"azure/gpt-5.2-2025-12-11": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
"cache_read_input_token_cost_priority": 3.5e-07,
|
||||
"input_cost_per_token": 1.75e-06,
|
||||
"input_cost_per_token_priority": 3.5e-06,
|
||||
"litellm_provider": "azure",
|
||||
"max_input_tokens": 400000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.4e-05,
|
||||
"output_cost_per_token_priority": 2.8e-05,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"azure/gpt-5.2-chat-2025-12-11": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
"cache_read_input_token_cost_priority": 3.5e-07,
|
||||
"input_cost_per_token": 1.75e-06,
|
||||
"input_cost_per_token_priority": 3.5e-06,
|
||||
"litellm_provider": "azure",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 16384,
|
||||
"max_tokens": 16384,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.4e-05,
|
||||
"output_cost_per_token_priority": 2.8e-05,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"azure/gpt-5.2-pro": {
|
||||
"input_cost_per_token": 2.1e-05,
|
||||
"litellm_provider": "azure",
|
||||
"max_input_tokens": 400000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.68e-04,
|
||||
"supported_endpoints": [
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"azure/gpt-5.2-pro-2025-12-11": {
|
||||
"input_cost_per_token": 2.1e-05,
|
||||
"litellm_provider": "azure",
|
||||
"max_input_tokens": 400000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.68e-04,
|
||||
"supported_endpoints": [
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"azure/gpt-image-1": {
|
||||
"input_cost_per_pixel": 4.0054321e-08,
|
||||
"litellm_provider": "azure",
|
||||
|
|
@ -15088,15 +15254,15 @@
|
|||
"tool_use_system_prompt_tokens": 159
|
||||
},
|
||||
"global.anthropic.claude-haiku-4-5-20251001-v1:0": {
|
||||
"cache_creation_input_token_cost": 1.375e-06,
|
||||
"cache_read_input_token_cost": 1.1e-07,
|
||||
"input_cost_per_token": 1.1e-06,
|
||||
"cache_creation_input_token_cost": 1.25e-06,
|
||||
"cache_read_input_token_cost": 1e-07,
|
||||
"input_cost_per_token": 1e-06,
|
||||
"litellm_provider": "bedrock_converse",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 64000,
|
||||
"max_tokens": 64000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 5.5e-06,
|
||||
"output_cost_per_token": 5e-06,
|
||||
"source": "https://aws.amazon.com/about-aws/whats-new/2025/10/claude-4-5-haiku-anthropic-amazon-bedrock",
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
|
|
@ -24668,6 +24834,32 @@
|
|||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 159
|
||||
},
|
||||
"eu.anthropic.claude-opus-4-5-20251101-v1:0": {
|
||||
"cache_creation_input_token_cost": 6.25e-06,
|
||||
"cache_read_input_token_cost": 5e-07,
|
||||
"input_cost_per_token": 5e-06,
|
||||
"litellm_provider": "bedrock_converse",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 64000,
|
||||
"max_tokens": 64000,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.5e-05,
|
||||
"search_context_cost_per_query": {
|
||||
"search_context_size_high": 0.01,
|
||||
"search_context_size_low": 0.01,
|
||||
"search_context_size_medium": 0.01
|
||||
},
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 159
|
||||
},
|
||||
"us.anthropic.claude-sonnet-4-20250514-v1:0": {
|
||||
"cache_creation_input_token_cost": 3.75e-06,
|
||||
"cache_read_input_token_cost": 3e-07,
|
||||
|
|
@ -30478,5 +30670,4 @@
|
|||
"litellm_provider": "fireworks_ai",
|
||||
"mode": "chat"
|
||||
}
|
||||
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue