scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 85

Merge remote-tracking branch 'origin' into litellm_sso_config_2

Browse source
This commit is contained in:
yuneng-jiang 2025-12-08 21:19:50 -08:00
commit dab4c9d8ab
84 changed files with 8332 additions and 1269 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
docker-compose.yml
View file

@ -22,7 +22,9 @@ services:
depends_on:
- db # Indicates that this service depends on the 'db' service, ensuring 'db' starts first
healthcheck: # Defines the health check configuration for the container
test: [ "CMD-SHELL", "wget --no-verbose --tries=1 http://localhost:4000/health/liveliness || exit 1" ] # Command to execute for health check
test:
- CMD-SHELL
- python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')" # Command to execute for health check
interval: 30s # Perform health check every 30 seconds
timeout: 10s # Health check command times out after 10 seconds
retries: 3 # Retry up to 3 times if health check fails

8
docs/my-website/docs/assistants.md
View file

@ -3,6 +3,14 @@ import TabItem from '@theme/TabItem';
# /assistants
:::warning Deprecation Notice
OpenAI has deprecated the Assistants API. It will shut down on **August 26, 2026**.
Consider migrating to the [Responses API](/docs/response_api) instead. See [OpenAI's migration guide](https://platform.openai.com/docs/guides/responses-vs-assistants) for details.
:::
Covers Threads, Messages, Assistants.
LiteLLM currently covers:

8
docs/my-website/docs/completion/drop_params.md
View file

@ -5,6 +5,14 @@ import TabItem from '@theme/TabItem';
Drop unsupported OpenAI params by your LLM Provider.
## Default Behavior
**By default, LiteLLM raises an exception** if you send a parameter to a model that doesn't support it.
For example, if you send `temperature=0.2` to a model that doesn't support the `temperature` parameter, LiteLLM will raise an exception.
**When `drop_params=True` is set**, LiteLLM will drop the unsupported parameter instead of raising an exception. This allows your code to work seamlessly across different providers without having to customize parameters for each one.
## Quick Start
```python

2
docs/my-website/docs/completion/json_mode.md
View file

@ -126,6 +126,8 @@ resp = completion(
)
print("Received={}".format(resp))
events_list = EventsList.model_validate_json(resp.choices[0].message.content)
```
</TabItem>
<TabItem value="proxy" label="PROXY">

14
docs/my-website/docs/contribute_integration/custom_webhook_api.md
View file

@ -95,11 +95,19 @@ curl -L -X POST 'http://0.0.0.0:4000/chat/completions' \
}'
```
4. File a PR!
4. Add Documentation
If you're adding a new integration, please add documentation for it under the `observability` folder:
- Create a new file at `docs/my-website/docs/observability/<your_integration>_integration.md`
- Follow the format of existing integration docs, such as [Langsmith Integration](https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/observability/langsmith_integration.md)
- Include: Quick Start, SDK usage, Proxy usage, and any advanced configuration options
5. File a PR!
- Review our contribution guide [here](../../extras/contributing_code)
- push your fork to your GitHub repo
- submit a PR from there
- Push your fork to your GitHub repo
- Submit a PR from there
## What get's logged?

20
docs/my-website/docs/embedding/supported_embedding.md
View file

@ -10,6 +10,26 @@ import os
os.environ['OPENAI_API_KEY'] = ""
response = embedding(model='text-embedding-ada-002', input=["good morning from litellm"])
```
## Async Usage - `aembedding()`
LiteLLM provides an asynchronous version of the `embedding` function called `aembedding`:
```python
from litellm import aembedding
import asyncio
async def get_embedding():
response = await aembedding(
model='text-embedding-ada-002',
input=["good morning from litellm"]
)
return response
response = asyncio.run(get_embedding())
print(response)
```
## Proxy Usage
**NOTE**

23
docs/my-website/docs/index.md
View file

@ -7,8 +7,8 @@ https://github.com/BerriAI/litellm
## **Call 100+ LLMs using the OpenAI Input/Output Format**
- Translate inputs to provider's `completion`, `embedding`, and `image_generation` endpoints
- [Consistent output](https://docs.litellm.ai/docs/completion/output), text responses will always be available at `['choices'][0]['message']['content']`
- Translate inputs to provider's endpoints (`/chat/completions`, `/responses`, `/embeddings`, `/images`, `/audio`, `/batches`, and more)
- [Consistent output](https://docs.litellm.ai/docs/supported_endpoints) - same response format regardless of which provider you use
- Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - [Router](https://docs.litellm.ai/docs/routing)
- Track spend & set budgets per project [LiteLLM Proxy Server](https://docs.litellm.ai/docs/simple_proxy)
@ -245,7 +245,7 @@ response = completion(
</Tabs>
### Response Format (OpenAI Format)
### Response Format (OpenAI Chat Completions Format)
```json
{
@ -514,15 +514,22 @@ response = completion(
LiteLLM maps exceptions across all supported providers to the OpenAI exceptions. All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM.
```python
from openai.error import OpenAIError
import litellm
from litellm import completion
import os
os.environ["ANTHROPIC_API_KEY"] = "bad-key"
try:
# some code
completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
except OpenAIError as e:
print(e)
completion(model="anthropic/claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
except litellm.AuthenticationError as e:
# Thrown when the API key is invalid
print(f"Authentication failed: {e}")
except litellm.RateLimitError as e:
# Thrown when you've exceeded your rate limit
print(f"Rate limited: {e}")
except litellm.APIError as e:
# Thrown for general API errors
print(f"API error: {e}")
```
### See How LiteLLM Transforms Your Requests

436
docs/my-website/docs/observability/helicone_integration.md
View file

@ -10,7 +10,7 @@ https://github.com/BerriAI/litellm
:::
[Helicone](https://helicone.ai/) is an open source observability platform that proxies your LLM requests and provides key insights into your usage, spend, latency and more.
[Helicone](https://helicone.ai/) is an open sourced observability platform providing key insights into your usage, spend, latency and more.
## Quick Start
@ -25,14 +25,10 @@ from litellm import completion
## Set env variables
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["OPENAI_API_KEY"] = "your-openai-key"
# Set callbacks
litellm.success_callback = ["helicone"]
# OpenAI call
response = completion(
model="gpt-4o",
model="helicone/gpt-4o-mini",
messages=[{"role": "user", "content": "Hi 👋 - I'm OpenAI"}],
)
@ -54,7 +50,7 @@ model_list:
# Add Helicone callback
litellm_settings:
success_callback: ["helicone"]
# Set Helicone API key
environment_variables:
HELICONE_API_KEY: "your-helicone-key"
@ -72,12 +68,12 @@ litellm --config config.yaml
There are two main approaches to integrate Helicone with LiteLLM:
1. **Callbacks**: Log to Helicone while using any provider
2. **Proxy Mode**: Use Helicone as a proxy for advanced features
1. **As a Provider**: Use Helicone to log requests for [all models supported ](../providers/helicone)
2. **Callbacks**: Log to Helicone while using any provider
### Supported LLM Providers
Helicone can log requests across [various LLM providers](https://docs.helicone.ai/getting-started/quick-start), including:
Helicone can log requests across [all major LLM providers](https://helicone.ai/models), including:
- OpenAI
- Azure
@ -88,156 +84,149 @@ Helicone can log requests across [various LLM providers](https://docs.helicone.a
- Replicate
- And more
## Method 1: Using Callbacks
## Method 1: Using Helicone as a Provider
Helicone's AI Gateway provides [advanced functionality](https://docs.helicone.ai) like caching, rate limiting, LLM security, and more.
<Tabs>
<TabItem value="sdk" label="Python SDK">
Set Helicone as your base URL and pass authentication headers:
```python
import os
import litellm
from litellm import completion
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
messages = [{"content": "What is the capital of France?", "role": "user"}]
# Helicone call - routes through Helicone gateway to any model
response = completion(
model="helicone/gpt-4o-mini", # or any 100+ models
messages=messages
)
print(response)
```
### Advanced Usage
You can add custom metadata and properties to your requests using Helicone headers. Here are some examples:
```python
litellm.metadata = {
"Helicone-User-Id": "user-abc", # Specify the user making the request
"Helicone-Property-App": "web", # Custom property to add additional information
"Helicone-Property-Custom": "any-value", # Add any custom property
"Helicone-Prompt-Id": "prompt-supreme-court", # Assign an ID to associate this prompt with future versions
"Helicone-Cache-Enabled": "true", # Enable caching of responses
"Cache-Control": "max-age=3600", # Set cache limit to 1 hour
"Helicone-RateLimit-Policy": "10;w=60;s=user", # Set rate limit policy
"Helicone-Retry-Enabled": "true", # Enable retry mechanism
"helicone-retry-num": "3", # Set number of retries
"helicone-retry-factor": "2", # Set exponential backoff factor
"Helicone-Model-Override": "gpt-3.5-turbo-0613", # Override the model used for cost calculation
"Helicone-Session-Id": "session-abc-123", # Set session ID for tracking
"Helicone-Session-Path": "parent-trace/child-trace", # Set session path for hierarchical tracking
"Helicone-Omit-Response": "false", # Include response in logging (default behavior)
"Helicone-Omit-Request": "false", # Include request in logging (default behavior)
"Helicone-LLM-Security-Enabled": "true", # Enable LLM security features
"Helicone-Moderations-Enabled": "true", # Enable content moderation
}
```
### Caching and Rate Limiting
Enable caching and set up rate limiting policies:
```python
litellm.metadata = {
"Helicone-Cache-Enabled": "true", # Enable caching of responses
"Cache-Control": "max-age=3600", # Set cache limit to 1 hour
"Helicone-RateLimit-Policy": "100;w=3600;s=user", # Set rate limit policy
}
```
</TabItem>
</Tabs>
## Method 2: Using Callbacks
Log requests to Helicone while using any LLM provider directly.
<Tabs>
<TabItem value="sdk" label="Python SDK">
<TabItem value="sdk" label="Python SDK">
```python
import os
import litellm
from litellm import completion
```python
import os
import litellm
from litellm import completion
## Set env variables
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["OPENAI_API_KEY"] = "your-openai-key"
# os.environ["HELICONE_API_BASE"] = "" # [OPTIONAL] defaults to `https://api.helicone.ai`
## Set env variables
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["OPENAI_API_KEY"] = "your-openai-key"
# os.environ["HELICONE_API_BASE"] = "" # [OPTIONAL] defaults to `https://api.helicone.ai`
# Set callbacks
litellm.success_callback = ["helicone"]
# Set callbacks
litellm.success_callback = ["helicone"]
# OpenAI call
response = completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Hi 👋 - I'm OpenAI"}],
)
# OpenAI call
response = completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Hi 👋 - I'm OpenAI"}],
)
print(response)
```
print(response)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```yaml title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-3
litellm_params:
model: anthropic/claude-3-sonnet-20240229
api_key: os.environ/ANTHROPIC_API_KEY
```yaml title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-3
litellm_params:
model: anthropic/claude-3-sonnet-20240229
api_key: os.environ/ANTHROPIC_API_KEY
# Add Helicone logging
litellm_settings:
success_callback: ["helicone"]
# Environment variables
environment_variables:
HELICONE_API_KEY: "your-helicone-key"
OPENAI_API_KEY: "your-openai-key"
ANTHROPIC_API_KEY: "your-anthropic-key"
```
# Add Helicone logging
litellm_settings:
success_callback: ["helicone"]
Start the proxy:
```bash
litellm --config config.yaml
```
# Environment variables
environment_variables:
HELICONE_API_KEY: "your-helicone-key"
OPENAI_API_KEY: "your-openai-key"
ANTHROPIC_API_KEY: "your-anthropic-key"
```
Make requests to your proxy:
```python
import openai
Start the proxy:
```bash
litellm --config config.yaml
```
client = openai.OpenAI(
api_key="anything", # proxy doesn't require real API key
base_url="http://localhost:4000"
)
Make requests to your proxy:
```python
import openai
response = client.chat.completions.create(
model="gpt-4", # This gets logged to Helicone
messages=[{"role": "user", "content": "Hello!"}]
)
```
client = openai.OpenAI(
api_key="anything", # proxy doesn't require real API key
base_url="http://localhost:4000"
)
</TabItem>
</Tabs>
response = client.chat.completions.create(
model="gpt-4", # This gets logged to Helicone
messages=[{"role": "user", "content": "Hello!"}]
)
```
## Method 2: Using Helicone as a Proxy
Helicone's proxy provides [advanced functionality](https://docs.helicone.ai/getting-started/proxy-vs-async) like caching, rate limiting, LLM security through [PromptArmor](https://promptarmor.com/) and more.
<Tabs>
<TabItem value="sdk" label="Python SDK">
Set Helicone as your base URL and pass authentication headers:
```python
import os
import litellm
from litellm import completion
# Configure LiteLLM to use Helicone proxy
litellm.api_base = "https://oai.hconeai.com/v1"
litellm.headers = {
"Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}",
}
# Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "your-openai-key"
response = completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "How does a court case get to the Supreme Court?"}]
)
print(response)
```
### Advanced Usage
You can add custom metadata and properties to your requests using Helicone headers. Here are some examples:
```python
litellm.metadata = {
"Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", # Authenticate to send requests to Helicone API
"Helicone-User-Id": "user-abc", # Specify the user making the request
"Helicone-Property-App": "web", # Custom property to add additional information
"Helicone-Property-Custom": "any-value", # Add any custom property
"Helicone-Prompt-Id": "prompt-supreme-court", # Assign an ID to associate this prompt with future versions
"Helicone-Cache-Enabled": "true", # Enable caching of responses
"Cache-Control": "max-age=3600", # Set cache limit to 1 hour
"Helicone-RateLimit-Policy": "10;w=60;s=user", # Set rate limit policy
"Helicone-Retry-Enabled": "true", # Enable retry mechanism
"helicone-retry-num": "3", # Set number of retries
"helicone-retry-factor": "2", # Set exponential backoff factor
"Helicone-Model-Override": "gpt-3.5-turbo-0613", # Override the model used for cost calculation
"Helicone-Session-Id": "session-abc-123", # Set session ID for tracking
"Helicone-Session-Path": "parent-trace/child-trace", # Set session path for hierarchical tracking
"Helicone-Omit-Response": "false", # Include response in logging (default behavior)
"Helicone-Omit-Request": "false", # Include request in logging (default behavior)
"Helicone-LLM-Security-Enabled": "true", # Enable LLM security features
"Helicone-Moderations-Enabled": "true", # Enable content moderation
"Helicone-Fallbacks": '["gpt-3.5-turbo", "gpt-4"]', # Set fallback models
}
```
### Caching and Rate Limiting
Enable caching and set up rate limiting policies:
```python
litellm.metadata = {
"Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}", # Authenticate to send requests to Helicone API
"Helicone-Cache-Enabled": "true", # Enable caching of responses
"Cache-Control": "max-age=3600", # Set cache limit to 1 hour
"Helicone-RateLimit-Policy": "100;w=3600;s=user", # Set rate limit policy
}
```
</TabItem>
</TabItem>
</Tabs>
## Session Tracking and Tracing
@ -245,57 +234,62 @@ litellm.metadata = {
Track multi-step and agentic LLM interactions using session IDs and paths:
<Tabs>
<TabItem value="sdk" label="Python SDK">
<TabItem value="sdk" label="Python SDK">
```python
import litellm
```python
import os
import litellm
from litellm import completion
litellm.api_base = "https://oai.hconeai.com/v1"
litellm.metadata = {
"Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}",
"Helicone-Session-Id": "session-abc-123",
"Helicone-Session-Path": "parent-trace/child-trace",
}
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Start a conversation"}]
)
```
messages = [{"content": "What is the capital of France?", "role": "user"}]
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
response = completion(
model="helicone/gpt-4",
messages=messages,
metadata={
"Helicone-Session-Id": "session-abc-123",
"Helicone-Session-Path": "parent-trace/child-trace",
}
)
```python
import openai
print(response)
```
client = openai.OpenAI(
api_key="anything",
base_url="http://localhost:4000"
)
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
# First request in session
response1 = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={
"Helicone-Session-Id": "session-abc-123",
"Helicone-Session-Path": "conversation/greeting"
}
)
```python
import openai
# Follow-up request in same session
response2 = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Tell me more"}],
extra_headers={
"Helicone-Session-Id": "session-abc-123",
"Helicone-Session-Path": "conversation/follow-up"
}
)
```
client = openai.OpenAI(
api_key="anything",
base_url="http://localhost:4000"
)
</TabItem>
# First request in session
response1 = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={
"Helicone-Session-Id": "session-abc-123",
"Helicone-Session-Path": "conversation/greeting"
}
)
# Follow-up request in same session
response2 = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Tell me more"}],
extra_headers={
"Helicone-Session-Id": "session-abc-123",
"Helicone-Session-Path": "conversation/follow-up"
}
)
```
</TabItem>
</Tabs>
- `Helicone-Session-Id`: Unique identifier for the session to group related requests
@ -304,52 +298,50 @@ response2 = client.chat.completions.create(
## Retry and Fallback Mechanisms
<Tabs>
<TabItem value="sdk" label="Python SDK">
<TabItem value="sdk" label="Python SDK">
```python
import litellm
```python
import litellm
litellm.api_base = "https://oai.hconeai.com/v1"
litellm.metadata = {
"Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}",
"Helicone-Retry-Enabled": "true",
"helicone-retry-num": "3",
"helicone-retry-factor": "2", # Exponential backoff
"Helicone-Fallbacks": '["gpt-3.5-turbo", "gpt-4"]',
}
litellm.api_base = "https://ai-gateway.helicone.ai/"
litellm.metadata = {
"Helicone-Retry-Enabled": "true",
"helicone-retry-num": "3",
"helicone-retry-factor": "2",
}
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}]
)
```
response = litellm.completion(
model="helicone/gpt-4o-mini/openai,claude-3-5-sonnet-20241022/anthropic", # Try OpenAI first, then fallback to Anthropic, then continue with other models
messages=[{"role": "user", "content": "Hello"}]
)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```yaml title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
api_base: "https://oai.hconeai.com/v1"
```yaml title="config.yaml"
model_list:
- model_name: gpt-4
litellm_params:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
api_base: "https://oai.hconeai.com/v1"
default_litellm_params:
headers:
Helicone-Auth: "Bearer ${HELICONE_API_KEY}"
Helicone-Retry-Enabled: "true"
helicone-retry-num: "3"
helicone-retry-factor: "2"
Helicone-Fallbacks: '["gpt-3.5-turbo", "gpt-4"]'
default_litellm_params:
headers:
Helicone-Auth: "Bearer ${HELICONE_API_KEY}"
Helicone-Retry-Enabled: "true"
helicone-retry-num: "3"
helicone-retry-factor: "2"
Helicone-Fallbacks: '["gpt-3.5-turbo", "gpt-4"]'
environment_variables:
HELICONE_API_KEY: "your-helicone-key"
OPENAI_API_KEY: "your-openai-key"
```
environment_variables:
HELICONE_API_KEY: "your-helicone-key"
OPENAI_API_KEY: "your-openai-key"
```
</TabItem>
</TabItem>
</Tabs>
> **Supported Headers** - For a full list of supported Helicone headers and their descriptions, please refer to the [Helicone documentation](https://docs.helicone.ai/getting-started/quick-start).
> **Supported Headers** - For a full list of supported Helicone headers and their descriptions, please refer to the [Helicone documentation](https://docs.helicone.ai/features/advanced-usage/custom-properties).
> By utilizing these headers and metadata options, you can gain deeper insights into your LLM usage, optimize performance, and better manage your AI workflows with Helicone and LiteLLM.

287
docs/my-website/docs/observability/sumologic_integration.md Normal file
View file

@ -0,0 +1,287 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Sumo Logic
Send LiteLLM logs to Sumo Logic for observability, monitoring, and analysis.
Sumo Logic is a cloud-native machine data analytics platform that provides real-time insights into your applications and infrastructure.
https://www.sumologic.com/
:::info
We want to learn how we can make the callbacks better! Meet the LiteLLM [founders](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) or
join our [discord](https://discord.gg/wuPM9dRgDw)
:::
## Pre-Requisites
1. Create a Sumo Logic account at https://www.sumologic.com/
2. Set up an HTTP Logs and Metrics Source in Sumo Logic:
- Go to **Manage Data** > **Collection** > **Collection**
- Click **Add Source** next to a Hosted Collector
- Select **HTTP Logs & Metrics**
- Copy the generated URL (it contains the authentication token)
For more details, see the [HTTP Logs & Metrics Source](https://www.sumologic.com/help/docs/send-data/hosted-collectors/http-source/logs-metrics/) documentation.
```shell
pip install litellm
```
## Quick Start
Use just 2 lines of code to instantly log your LLM responses to Sumo Logic.
The Sumo Logic HTTP Source URL includes the authentication token, so no separate API key is required.
<Tabs>
<TabItem value="python" label="SDK">
```python
litellm.callbacks = ["sumologic"]
```
```python
import litellm
import os
# Sumo Logic HTTP Source URL (includes auth token)
os.environ["SUMOLOGIC_WEBHOOK_URL"] = "https://collectors.sumologic.com/receiver/v1/http/your-token-here"
# LLM API Keys
os.environ['OPENAI_API_KEY'] = ""
# Set sumologic as a callback
litellm.callbacks = ["sumologic"]
# OpenAI call
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Hi 👋 - I'm testing Sumo Logic integration"}
]
)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
callbacks: ["sumologic"]
environment_variables:
SUMOLOGIC_WEBHOOK_URL: os.environ/SUMOLOGIC_WEBHOOK_URL
```
2. Start LiteLLM Proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl -L -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "Hey, how are you?"
}
]
}'
```
</TabItem>
</Tabs>
## What Data is Logged?
LiteLLM sends the [Standard Logging Payload](https://docs.litellm.ai/docs/proxy/logging_spec) to Sumo Logic, which includes:
- **Request details**: Model, messages, parameters
- **Response details**: Completion text, token usage, latency
- **Metadata**: User ID, custom metadata, timestamps
- **Cost tracking**: Response cost based on token usage
Example payload:
```json
{
"id": "chatcmpl-123",
"call_type": "litellm.completion",
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "Hello"}
],
"response": {
"choices": [{
"message": {
"role": "assistant",
"content": "Hi there!"
}
}]
},
"usage": {
"prompt_tokens": 10,
"completion_tokens": 5,
"total_tokens": 15
},
"response_cost": 0.0001,
"start_time": "2024-01-01T00:00:00",
"end_time": "2024-01-01T00:00:01"
}
```
## Advanced Configuration
### Batching Settings
Control how LiteLLM batches logs before sending to Sumo Logic:
<Tabs>
<TabItem value="python" label="SDK">
```python
import litellm
os.environ["SUMOLOGIC_WEBHOOK_URL"] = "https://collectors.sumologic.com/receiver/v1/http/your-token"
litellm.callbacks = ["sumologic"]
# Configure batch settings (optional)
# These are inherited from CustomBatchLogger
# Default batch_size: 100
# Default flush_interval: 60 seconds
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```yaml
litellm_settings:
callbacks: ["sumologic"]
environment_variables:
SUMOLOGIC_WEBHOOK_URL: os.environ/SUMOLOGIC_WEBHOOK_URL
```
</TabItem>
</Tabs>
### Compressed Data
Sumo Logic supports compressed data (gzip or deflate). LiteLLM automatically handles compression when beneficial.
Benefits:
- Reduced network usage
- Faster message delivery
- Lower data transfer costs
### Query Logs in Sumo Logic
Once logs are flowing to Sumo Logic, you can query them using the Sumo Logic Query Language:
```sql
_sourceCategory=litellm
| json "model", "response_cost", "usage.total_tokens" as model, cost, tokens
| sum(cost) by model
```
Example queries:
**Total cost by model:**
```sql
_sourceCategory=litellm
| json "model", "response_cost" as model, cost
| sum(cost) as total_cost by model
| sort by total_cost desc
```
**Average response time:**
```sql
_sourceCategory=litellm
| json "start_time", "end_time" as start, end
| parse regex field=start "(?<start_ms>\d+)"
| parse regex field=end "(?<end_ms>\d+)"
| (end_ms - start_ms) as response_time_ms
| avg(response_time_ms) as avg_response_time
```
**Requests per user:**
```sql
_sourceCategory=litellm
| json "model_parameters.user" as user
| count by user
```
## Authentication
The Sumo Logic HTTP Source URL includes the authentication token, so you only need to set the `SUMOLOGIC_WEBHOOK_URL` environment variable.
**Security Best Practices:**
- Keep your HTTP Source URL private (it contains the auth token)
- Store it in environment variables or secrets management
- Regenerate the URL if it's compromised (in Sumo Logic UI)
- Use separate HTTP Sources for different environments (dev, staging, prod)
## Getting Your Sumo Logic URL
1. Log in to [Sumo Logic](https://www.sumologic.com/)
2. Go to **Manage Data** > **Collection** > **Collection**
3. Click **Add Source** next to a Hosted Collector
4. Select **HTTP Logs & Metrics**
5. Configure the source:
- **Name**: LiteLLM Logs
- **Source Category**: litellm (optional, but helps with queries)
6. Click **Save**
7. Copy the displayed URL - it will look like:
```
https://collectors.sumologic.com/receiver/v1/http/ZaVnC4dhaV39Tn37...
```
## Troubleshooting
### Logs not appearing in Sumo Logic
1. **Verify the URL**: Make sure `SUMOLOGIC_WEBHOOK_URL` is set correctly
2. **Check the HTTP Source**: Ensure it's active in Sumo Logic UI
3. **Wait for batching**: Logs are sent in batches, wait 60 seconds
4. **Check for errors**: Enable debug logging in LiteLLM:
```python
litellm.set_verbose = True
```
### URL Format
The URL must be the complete HTTP Source URL from Sumo Logic:
- ✅ Correct: `https://collectors.sumologic.com/receiver/v1/http/ZaVnC4dhaV39Tn37...`
### No authentication errors
If you get authentication errors, regenerate the HTTP Source URL in Sumo Logic:
1. Go to your HTTP Source in Sumo Logic
2. Click the settings icon
3. Click **Show URL**
4. Click **Regenerate URL**
5. Update your `SUMOLOGIC_WEBHOOK_URL` environment variable
## Support & Talk to Founders
- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai

316
docs/my-website/docs/providers/bedrock_writer.md Normal file
View file

@ -0,0 +1,316 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Bedrock - Writer Palmyra
## Overview
| Property | Details |
|-------|-------|
| Description | Writer Palmyra X5 and X4 foundation models on Amazon Bedrock, offering advanced reasoning, tool calling, and document processing capabilities |
| Provider Route on LiteLLM | `bedrock/` |
| Supported Operations | `/chat/completions` |
| Link to Provider Doc | [Writer on AWS Bedrock ↗](https://aws.amazon.com/bedrock/writer/) |
## Quick Start
### LiteLLM SDK
```python showLineNumbers title="SDK Usage"
import litellm
import os
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = "us-west-2"
response = litellm.completion(
model="bedrock/us.writer.palmyra-x5-v1:0",
messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)
```
### LiteLLM Proxy
**1. Setup config.yaml**
```yaml showLineNumbers title="proxy_config.yaml"
model_list:
- model_name: writer-palmyra-x5
litellm_params:
model: bedrock/us.writer.palmyra-x5-v1:0
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_region_name: us-west-2
```
**2. Start the proxy**
```bash showLineNumbers title="Start Proxy"
litellm --config config.yaml
```
**3. Call the proxy**
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="curl Request"
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "writer-palmyra-x5",
"messages": [{"role": "user", "content": "Hello, how are you?"}]
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI SDK">
```python showLineNumbers title="OpenAI SDK"
from openai import OpenAI
client = OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000/v1"
)
response = client.chat.completions.create(
model="writer-palmyra-x5",
messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)
```
</TabItem>
</Tabs>
## Tool Calling
Writer Palmyra models support multi-step tool calling for complex workflows.
### LiteLLM SDK
```python showLineNumbers title="Tool Calling - SDK"
import litellm
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state"
}
},
"required": ["location"]
}
}
}
]
response = litellm.completion(
model="bedrock/us.writer.palmyra-x5-v1:0",
messages=[{"role": "user", "content": "What's the weather in Boston?"}],
tools=tools
)
```
### LiteLLM Proxy
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Tool Calling - curl"
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "writer-palmyra-x5",
"messages": [{"role": "user", "content": "What'\''s the weather in Boston?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city and state"}
},
"required": ["location"]
}
}
}]
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI SDK">
```python showLineNumbers title="Tool Calling - OpenAI SDK"
from openai import OpenAI
client = OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000/v1"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state"
}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="writer-palmyra-x5",
messages=[{"role": "user", "content": "What's the weather in Boston?"}],
tools=tools
)
```
</TabItem>
</Tabs>
## Document Input
Writer Palmyra models support document inputs including PDFs.
### LiteLLM SDK
```python showLineNumbers title="PDF Document Input - SDK"
import litellm
import base64
# Read and encode PDF
with open("document.pdf", "rb") as f:
pdf_base64 = base64.b64encode(f.read()).decode("utf-8")
response = litellm.completion(
model="bedrock/us.writer.palmyra-x5-v1:0",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:application/pdf;base64,{pdf_base64}"
}
},
{
"type": "text",
"text": "Summarize this document"
}
]
}
]
)
```
### LiteLLM Proxy
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="PDF Document Input - curl"
# First, base64 encode your PDF
PDF_BASE64=$(base64 -i document.pdf)
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "writer-palmyra-x5",
"messages": [{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": "data:application/pdf;base64,'$PDF_BASE64'"}
},
{
"type": "text",
"text": "Summarize this document"
}
]
}]
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI SDK">
```python showLineNumbers title="PDF Document Input - OpenAI SDK"
from openai import OpenAI
import base64
client = OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000/v1"
)
# Read and encode PDF
with open("document.pdf", "rb") as f:
pdf_base64 = base64.b64encode(f.read()).decode("utf-8")
response = client.chat.completions.create(
model="writer-palmyra-x5",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:application/pdf;base64,{pdf_base64}"
}
},
{
"type": "text",
"text": "Summarize this document"
}
]
}
]
)
```
</TabItem>
</Tabs>
## Supported Models
| Model ID | Context Window | Input Cost (per 1K tokens) | Output Cost (per 1K tokens) |
|----------|---------------|---------------------------|----------------------------|
| `bedrock/us.writer.palmyra-x5-v1:0` | 1M tokens | $0.0006 | $0.006 |
| `bedrock/us.writer.palmyra-x4-v1:0` | 128K tokens | $0.0025 | $0.010 |
| `bedrock/writer.palmyra-x5-v1:0` | 1M tokens | $0.0006 | $0.006 |
| `bedrock/writer.palmyra-x4-v1:0` | 128K tokens | $0.0025 | $0.010 |
:::info Cross-Region Inference
The `us.writer.*` model IDs use cross-region inference profiles. Use these for production workloads.
:::

268
docs/my-website/docs/providers/helicone.md Normal file
View file

@ -0,0 +1,268 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Helicone
## Overview
| Property | Details |
|-------|-------|
| Description | Helicone is an AI gateway and observability platform that provides OpenAI-compatible endpoints with advanced monitoring, caching, and analytics capabilities. |
| Provider Route on LiteLLM | `helicone/` |
| Link to Provider Doc | [Helicone Documentation ↗](https://docs.helicone.ai) |
| Base URL | `https://ai-gateway.helicone.ai/` |
| Supported Operations | [`/chat/completions`](#sample-usage), [`/completions`](#text-completion), [`/embeddings`](#embeddings) |
<br />
**We support [ALL models available](https://helicone.ai/models) through Helicone's AI Gateway. Use `helicone/` as a prefix when sending requests.**
## What is Helicone?
Helicone is an open-source observability platform for LLM applications that provides:
- **Request Monitoring**: Track all LLM requests with detailed metrics
- **Caching**: Reduce costs and latency with intelligent caching
- **Rate Limiting**: Control request rates per user/key
- **Cost Tracking**: Monitor spend across models and users
- **Custom Properties**: Tag requests with metadata for filtering and analysis
- **Prompt Management**: Version control for prompts
## Required Variables
```python showLineNumbers title="Environment Variables"
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
```
Get your Helicone API key from your [Helicone dashboard](https://helicone.ai).
## Usage - LiteLLM Python SDK
### Non-streaming
```python showLineNumbers title="Helicone Non-streaming Completion"
import os
import litellm
from litellm import completion
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
messages = [{"content": "What is the capital of France?", "role": "user"}]
# Helicone call - routes through Helicone gateway to OpenAI
response = completion(
model="helicone/gpt-4",
messages=messages
)
print(response)
```
### Streaming
```python showLineNumbers title="Helicone Streaming Completion"
import os
import litellm
from litellm import completion
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
messages = [{"content": "Write a short poem about AI", "role": "user"}]
# Helicone call with streaming
response = completion(
model="helicone/gpt-4",
messages=messages,
stream=True
)
for chunk in response:
print(chunk)
```
### With Metadata (Helicone Custom Properties)
```python showLineNumbers title="Helicone with Custom Properties"
import os
import litellm
from litellm import completion
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
response = completion(
model="helicone/gpt-4o-mini",
messages=[{"role": "user", "content": "What's the weather like?"}],
metadata={
"Helicone-Property-Environment": "production",
"Helicone-Property-User-Id": "user_123",
"Helicone-Property-Session-Id": "session_abc"
}
)
print(response)
```
### Text Completion
```python showLineNumbers title="Helicone Text Completion"
import os
import litellm
os.environ["HELICONE_API_KEY"] = "" # your Helicone API key
response = litellm.completion(
model="helicone/gpt-4o-mini", # text completion model
prompt="Once upon a time"
)
print(response)
```
## Retry and Fallback Mechanisms
```python
import litellm
litellm.api_base = "https://ai-gateway.helicone.ai/"
litellm.metadata = {
"Helicone-Retry-Enabled": "true",
"helicone-retry-num": "3",
"helicone-retry-factor": "2",
}
response = litellm.completion(
model="helicone/gpt-4o-mini/openai,claude-3-5-sonnet-20241022/anthropic", # Try OpenAI first, then fallback to Anthropic, then continue with other models,
messages=[{"role": "user", "content": "Hello"}]
)
```
## Supported OpenAI Parameters
Helicone supports all standard OpenAI-compatible parameters:
| Parameter | Type | Description |
|-----------|------|-------------|
| `messages` | array | **Required**. Array of message objects with 'role' and 'content' |
| `model` | string | **Required**. Model ID (e.g., gpt-4, claude-3-opus, etc.) |
| `stream` | boolean | Optional. Enable streaming responses |
| `temperature` | float | Optional. Sampling temperature |
| `top_p` | float | Optional. Nucleus sampling parameter |
| `max_tokens` | integer | Optional. Maximum tokens to generate |
| `frequency_penalty` | float | Optional. Penalize frequent tokens |
| `presence_penalty` | float | Optional. Penalize tokens based on presence |
| `stop` | string/array | Optional. Stop sequences |
| `n` | integer | Optional. Number of completions to generate |
| `tools` | array | Optional. List of available tools/functions |
| `tool_choice` | string/object | Optional. Control tool/function calling |
| `response_format` | object | Optional. Response format specification |
| `user` | string | Optional. User identifier |
## Helicone-Specific Headers
Pass these as metadata to leverage Helicone features:
| Header | Description |
|--------|-------------|
| `Helicone-Property-*` | Custom properties for filtering (e.g., `Helicone-Property-User-Id`) |
| `Helicone-Cache-Enabled` | Enable caching for this request |
| `Helicone-User-Id` | User identifier for tracking |
| `Helicone-Session-Id` | Session identifier for grouping requests |
| `Helicone-Prompt-Id` | Prompt identifier for versioning |
| `Helicone-Rate-Limit-Policy` | Rate limiting policy name |
Example with headers:
```python showLineNumbers title="Helicone with Custom Headers"
import litellm
response = litellm.completion(
model="helicone/gpt-4",
messages=[{"role": "user", "content": "Hello"}],
metadata={
"Helicone-Cache-Enabled": "true",
"Helicone-Property-Environment": "production",
"Helicone-Property-User-Id": "user_123",
"Helicone-Session-Id": "session_abc",
"Helicone-Prompt-Id": "prompt_v1"
}
)
```
## Advanced Usage
### Using with Different Providers
Helicone acts as a gateway and supports multiple providers:
```python showLineNumbers title="Helicone with Anthropic"
import litellm
# Set both Helicone and Anthropic keys
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
response = litellm.completion(
model="helicone/claude-3.5-haiku/anthropic",
messages=[{"role": "user", "content": "Hello"}]
)
```
### Caching
Enable caching to reduce costs and latency:
```python showLineNumbers title="Helicone Caching"
import litellm
response = litellm.completion(
model="helicone/gpt-4",
messages=[{"role": "user", "content": "What is 2+2?"}],
metadata={
"Helicone-Cache-Enabled": "true"
}
)
# Subsequent identical requests will be served from cache
response2 = litellm.completion(
model="helicone/gpt-4",
messages=[{"role": "user", "content": "What is 2+2?"}],
metadata={
"Helicone-Cache-Enabled": "true"
}
)
```
## Features
### Request Monitoring
- Track all requests with detailed metrics
- View request/response pairs
- Monitor latency and errors
- Filter by custom properties
### Cost Tracking
- Per-model cost tracking
- Per-user cost tracking
- Cost alerts and budgets
- Historical cost analysis
### Rate Limiting
- Per-user rate limits
- Per-API key rate limits
- Custom rate limit policies
- Automatic enforcement
### Analytics
- Request volume trends
- Cost trends
- Latency percentiles
- Error rates
Visit [Helicone Pricing](https://helicone.ai/pricing) for details.
## Additional Resources
- [Helicone Official Documentation](https://docs.helicone.ai)
- [Helicone Dashboard](https://helicone.ai)
- [Helicone GitHub](https://github.com/Helicone/helicone)
- [API Reference](https://docs.helicone.ai/rest/ai-gateway/post-v1-chat-completions)

117
docs/my-website/docs/providers/nvidia_nim_rerank.md
View file

@ -141,6 +141,111 @@ curl -X POST http://0.0.0.0:4000/rerank \
}'
```
## `/v1/ranking` Models (llama-3.2-nv-rerankqa-1b-v2)
Some Nvidia NIM rerank models use the `/v1/ranking` endpoint instead of the default `/v1/retrieval/{model}/reranking` endpoint.
Use the `ranking/` prefix to force requests to the `/v1/ranking` endpoint:
### LiteLLM Python SDK
```python showLineNumbers title="Force /v1/ranking endpoint with ranking/ prefix"
import litellm
import os
os.environ['NVIDIA_NIM_API_KEY'] = "nvapi-..."
# Use "ranking/" prefix to force /v1/ranking endpoint
response = litellm.rerank(
model="nvidia_nim/ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2",
query="which way did the traveler go?",
documents=[
"two roads diverged in a yellow wood...",
"then took the other, as just as fair...",
"i shall be telling this with a sigh somewhere ages and ages hence..."
],
top_n=3,
truncate="END", # Optional: truncate long text from the end
)
print(response)
```
### LiteLLM Proxy
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: nvidia-ranking
litellm_params:
model: nvidia_nim/ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2
api_key: os.environ/NVIDIA_NIM_API_KEY
```
```bash title="Request to LiteLLM Proxy"
curl -X POST http://0.0.0.0:4000/rerank \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia-ranking",
"query": "which way did the traveler go?",
"documents": [
"two roads diverged in a yellow wood...",
"then took the other, as just as fair..."
],
"top_n": 2
}'
```
### Understanding Model Resolution
**Ranking Endpoint (`/v1/ranking`):**
```
model: nvidia_nim/ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2
└────┬────┘ └──┬──┘ └─────────────┬──────────────────┘
│ │ │
│ │ └────▶ Model name sent to provider
│ │
│ └────────────────────────▶ Tells LiteLLM the request/response and url should be sent to Nvidia NIM /v1/ranking endpoint
│
└─────────────────────────────────▶ Provider prefix
API URL: https://ai.api.nvidia.com/v1/ranking
```
**Visual Flow:**
```
Client Request LiteLLM Provider API
────────────── ──────────── ─────────────
# Default reranking endpoint
model: "nvidia_nim/nvidia/model-name"
1. Extracts model: nvidia/model-name
2. Routes to default endpoint ──────▶ POST /v1/retrieval/nvidia/model-name/reranking
# Forced ranking endpoint
model: "nvidia_nim/ranking/nvidia/model-name"
1. Detects "ranking/" prefix
2. Extracts model: nvidia/model-name
3. Routes to ranking endpoint ──────▶ POST /v1/ranking
Body: {"model": "nvidia/model-name", ...}
```
**When to use each endpoint:**
| Endpoint | Model Prefix | Use Case |
|----------|--------------|----------|
| `/v1/retrieval/{model}/reranking` | `nvidia_nim/<model>` | Default for most rerank models |
| `/v1/ranking` | `nvidia_nim/ranking/<model>` | For models like `nvidia/llama-3.2-nv-rerankqa-1b-v2` that require this endpoint |
:::tip
Check the [Nvidia NIM model deployment page](https://build.nvidia.com/nvidia/llama-3_2-nv-rerankqa-1b-v2/deploy) to see which endpoint your model requires.
:::
## API Parameters
### Required Parameters
@ -203,16 +308,7 @@ response = litellm.rerank(
</TabItem>
</Tabs>
## API Endpoint
The rerank endpoint uses a different base URL than chat/embeddings:
- **Chat/Embeddings:** `https://integrate.api.nvidia.com/v1/`
- **Rerank:** `https://ai.api.nvidia.com/v1/`
LiteLLM automatically uses the correct endpoint for rerank requests.
### Custom API Base URL
## Custom API Base URL
You can override the default base URL in several ways:
@ -258,4 +354,3 @@ Get your Nvidia NIM API key from [Nvidia's website](https://developer.nvidia.com
- [Nvidia NIM Chat Completions](./nvidia_nim#sample-usage)
- [LiteLLM Rerank Endpoint](../rerank)
- [Nvidia NIM Official Docs ↗](https://docs.api.nvidia.com/nim/reference/)

121
docs/my-website/docs/providers/sap.md Normal file
View file

@ -0,0 +1,121 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# SAP Generative AI Hub
LiteLLM supports SAP Generative AI Hub's Orchestration Service.
| Property | Details |
|-------|-------|
| Description | SAP's Generative AI Hub provides access to foundation models through the AI Core orchestration service. |
| Provider Route on LiteLLM | `sap/` |
| Supported Endpoints | `/chat/completions` |
| API Reference | [SAP AI Core Documentation](https://help.sap.com/docs/sap-ai-core) |
## Authentication
SAP Generative AI Hub uses service key authentication. You can provide credentials via:
1. **Environment variable** - Set `AICORE_SERVICE_KEY` with your service key JSON
2. **Direct parameter** - Pass `api_key` with the service key JSON string
```python showLineNumbers title="Environment Variable"
import os
os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
```
## Usage - LiteLLM Python SDK
```python showLineNumbers title="SAP Chat Completion"
from litellm import completion
import os
os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
response = completion(
model="sap/gpt-4",
messages=[{"role": "user", "content": "Hello from LiteLLM"}]
)
print(response)
```
```python showLineNumbers title="SAP Chat Completion - Streaming"
from litellm import completion
import os
os.environ["AICORE_SERVICE_KEY"] = '{"clientid": "...", "clientsecret": "...", ...}'
response = completion(
model="sap/gpt-4",
messages=[{"role": "user", "content": "Hello from LiteLLM"}],
stream=True
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
```
## Usage - LiteLLM Proxy
Add to your LiteLLM Proxy config:
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: sap-gpt4
litellm_params:
model: sap/gpt-4
api_key: os.environ/AICORE_SERVICE_KEY
```
Start the proxy:
```bash showLineNumbers title="Start Proxy"
litellm --config config.yaml
```
<Tabs>
<TabItem value="curl" label="cURL">
```bash showLineNumbers title="Test Request"
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-proxy-api-key" \
-d '{
"model": "sap-gpt4",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI SDK">
```python showLineNumbers title="OpenAI SDK"
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000",
api_key="your-proxy-api-key"
)
response = client.chat.completions.create(
model="sap-gpt4",
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)
```
</TabItem>
</Tabs>
## Supported Parameters
| Parameter | Description |
|-----------|-------------|
| `temperature` | Controls randomness |
| `max_tokens` | Maximum tokens in response |
| `top_p` | Nucleus sampling |
| `tools` | Function calling tools |
| `tool_choice` | Tool selection behavior |
| `response_format` | Output format (json_object, json_schema) |
| `stream` | Enable streaming |

3
docs/my-website/docs/proxy/dynamic_rate_limit.md
View file

@ -149,6 +149,7 @@ litellm_settings:
priority_reservation_settings:
default_priority: 0 # Weight (0%) assigned to keys without explicit priority metadata
saturation_threshold: 0.50 # A model is saturated if it has hit 50% of its RPM limit
saturation_check_cache_ttl: 60 # How long (seconds) saturation values are cached locally
general_settings:
master_key: sk-1234 # OR set `LITELLM_MASTER_KEY=".."` in your .env
@ -168,6 +169,8 @@ general_settings:
- **default_priority (float)**: Weight/percentage (0.0 to 1.0) assigned to API keys that have no priority metadata set (defaults to 0.5)
- **saturation_threshold (float)**: Saturation level (0.0 to 1.0) at which strict priority enforcement begins for a model. Saturation is calculated as `max(current_rpm/max_rpm, current_tpm/max_tpm)`. Below this threshold, generous mode allows priority borrowing from unused capacity. Above this threshold, strict mode enforces normalized priority limits.
- Example: When model usage is low, keys can use more than their allocated share. When model usage is high, keys are strictly limited to their allocated share.
- **saturation_check_cache_ttl (int)**: TTL in seconds for local cache when reading saturation values from Redis (defaults to 60). In multi-node deployments, this controls how quickly nodes converge on the same saturation state. Lower values mean faster convergence but more Redis reads.
- Example: Set to `5` for faster multi-node consistency, or `0` to always read directly from Redis.
**Start Proxy**

146
docs/my-website/docs/proxy/enterprise.md
View file

@ -15,8 +15,7 @@ Features:
- ✅ [SSO for Admin UI](./ui.md#✨-enterprise-features)
- ✅ [Audit Logs with retention policy](#audit-logs)
- ✅ [JWT-Auth](./token_auth.md)
- ✅ [Control available public, private routes (Restrict certain endpoints on proxy)](#control-available-public-private-routes)
- ✅ [Control available public, private routes](#control-available-public-private-routes)
- ✅ [Control available public, private routes](./public_routes.md)
- ✅ [Secret Managers - AWS Key Manager, Google Secret Manager, Azure Key, Hashicorp Vault](../secret)
- ✅ [[BETA] AWS Key Manager v2 - Key Decryption](#beta-aws-key-manager---key-decryption)
- ✅ IP address‑based access control lists
@ -181,148 +180,7 @@ Expected Response
### Control available public, private routes
**Restrict certain endpoints of proxy**
:::info
❓ Use this when you want to:
- make an existing private route -> public
- set certain routes as admin_only routes
:::
#### Usage - Define public, admin only routes
**Step 1** - Set on config.yaml
| Route Type | Optional | Requires Virtual Key Auth | Admin Can Access | All Roles Can Access | Description |
|------------|----------|---------------------------|-------------------|----------------------|-------------|
| `public_routes` | ✅ | ❌ | ✅ | ✅ | Routes that can be accessed without any authentication |
| `admin_only_routes` | ✅ | ✅ | ✅ | ❌ | Routes that can only be accessed by [Proxy Admin](./self_serve#available-roles) |
| `allowed_routes` | ✅ | ✅ | ✅ | ✅ | Routes are exposed on the proxy. If not set then all routes exposed. |
`LiteLLMRoutes.public_routes` is an ENUM corresponding to the default public routes on LiteLLM. [You can see this here](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/_types.py)
```yaml
general_settings:
master_key: sk-1234
public_routes: ["LiteLLMRoutes.public_routes", "/spend/calculate"] # routes that can be accessed without any auth
admin_only_routes: ["/key/generate"] # Optional - routes that can only be accessed by Proxy Admin
allowed_routes: ["/chat/completions", "/spend/calculate", "LiteLLMRoutes.public_routes"] # Optional - routes that can be accessed by anyone after Authentication
```
**Step 2** - start proxy
```shell
litellm --config config.yaml
```
**Step 3** - Test it
<Tabs>
<TabItem value="public" label="Test `public_routes`">
```shell
curl --request POST \
--url 'http://localhost:4000/spend/calculate' \
--header 'Content-Type: application/json' \
--data '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hey, how'\''s it going?"}]
}'
```
🎉 Expect this endpoint to work without an `Authorization / Bearer Token`
</TabItem>
<TabItem value="admin_only_routes" label="Test `admin_only_routes`">
**Successful Request**
```shell
curl --location 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer <your-master-key>' \
--header 'Content-Type: application/json' \
--data '{}'
```
**Un-successfull Request**
```shell
curl --location 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer <virtual-key-from-non-admin>' \
--header 'Content-Type: application/json' \
--data '{"user_role": "internal_user"}'
```
**Expected Response**
```json
{
"error": {
"message": "user not allowed to access this route. Route=/key/generate is an admin only route",
"type": "auth_error",
"param": "None",
"code": "403"
}
}
```
</TabItem>
<TabItem value="allowed_routes" label="Test `allowed_routes`">
**Successful Request**
```shell
curl http://localhost:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "fake-openai-endpoint",
"messages": [
{"role": "user", "content": "Hello, Claude"}
]
}'
```
**Un-successfull Request**
```shell
curl --location 'http://0.0.0.0:4000/embeddings' \
--header 'Content-Type: application/json' \
-H "Authorization: Bearer sk-1234" \
--data ' {
"model": "text-embedding-ada-002",
"input": ["write a litellm poem"]
}'
```
**Expected Response**
```json
{
"error": {
"message": "Route /embeddings not allowed",
"type": "auth_error",
"param": "None",
"code": "403"
}
}
```
</TabItem>
</Tabs>
See [Control Public & Private Routes](./public_routes.md) for detailed documentation on configuring public routes, admin-only routes, allowed routes, and wildcard patterns.
## Spend Tracking

223
docs/my-website/docs/proxy/public_routes.md Normal file
View file

@ -0,0 +1,223 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Control Public & Private Routes
:::info
Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat).
:::
Control which routes require authentication and which routes are publicly accessible.
## Route Types
| Route Type | Requires Auth | Description |
|------------|---------------|-------------|
| `public_routes` | No | Routes accessible without any authentication |
| `admin_only_routes` | Yes (Admin only) | Routes only accessible by [Proxy Admin](./self_serve#available-roles) |
| `allowed_routes` | Yes | Routes exposed on the proxy. If not set, all routes are exposed |
## Quick Start
### Make Routes Public
Allow specific routes to be accessed without authentication:
```yaml
general_settings:
master_key: sk-1234
public_routes: ["LiteLLMRoutes.public_routes", "/spend/calculate"]
```
### Restrict Routes to Admin Only
Restrict certain routes to only be accessible by Proxy Admin:
```yaml
general_settings:
master_key: sk-1234
admin_only_routes: ["/key/generate", "/key/delete"]
```
### Limit Available Routes
Only expose specific routes on the proxy:
```yaml
general_settings:
master_key: sk-1234
allowed_routes: ["/chat/completions", "/embeddings", "LiteLLMRoutes.public_routes"]
```
## Usage Examples
### Define Public, Admin Only, and Allowed Routes
```yaml
general_settings:
master_key: sk-1234
public_routes: ["LiteLLMRoutes.public_routes", "/spend/calculate"]
admin_only_routes: ["/key/generate"]
allowed_routes: ["/chat/completions", "/spend/calculate", "LiteLLMRoutes.public_routes"]
```
`LiteLLMRoutes.public_routes` is an ENUM corresponding to the default public routes on LiteLLM. [View the source](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/_types.py).
### Testing
<Tabs>
<TabItem value="public" label="Test public_routes">
```shell
curl --request POST \
--url 'http://localhost:4000/spend/calculate' \
--header 'Content-Type: application/json' \
--data '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hey, how'\''s it going?"}]
}'
```
This endpoint works without an `Authorization` header.
</TabItem>
<TabItem value="admin_only_routes" label="Test admin_only_routes">
**Successful Request (Admin)**
```shell
curl --location 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer <your-master-key>' \
--header 'Content-Type: application/json' \
--data '{}'
```
**Unsuccessful Request (Non-Admin)**
```shell
curl --location 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer <virtual-key-from-non-admin>' \
--header 'Content-Type: application/json' \
--data '{"user_role": "internal_user"}'
```
**Expected Response**
```json
{
"error": {
"message": "user not allowed to access this route. Route=/key/generate is an admin only route",
"type": "auth_error",
"param": "None",
"code": "403"
}
}
```
</TabItem>
<TabItem value="allowed_routes" label="Test allowed_routes">
**Successful Request**
```shell
curl http://localhost:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "fake-openai-endpoint",
"messages": [
{"role": "user", "content": "Hello, Claude"}
]
}'
```
**Unsuccessful Request (Route Not Allowed)**
```shell
curl --location 'http://0.0.0.0:4000/embeddings' \
--header 'Content-Type: application/json' \
-H "Authorization: Bearer sk-1234" \
--data '{
"model": "text-embedding-ada-002",
"input": ["write a litellm poem"]
}'
```
**Expected Response**
```json
{
"error": {
"message": "Route /embeddings not allowed",
"type": "auth_error",
"param": "None",
"code": "403"
}
}
```
</TabItem>
</Tabs>
## Advanced: Wildcard Patterns
Use wildcard patterns to match multiple routes at once.
### Syntax
| Pattern | Description | Example |
|---------|-------------|---------|
| `/path/*` | Matches any route starting with `/path/` | `/api/*` matches `/api/users`, `/api/users/123` |
### Examples
#### Make All Routes Under a Path Public
```yaml
general_settings:
master_key: sk-1234
public_routes:
- "LiteLLMRoutes.public_routes"
- "/api/v1/*" # All routes under /api/v1/
- "/health/*" # All health check routes
```
#### Restrict Admin Routes with Wildcards
```yaml
general_settings:
master_key: sk-1234
admin_only_routes:
- "/admin/*" # All admin routes
- "/internal/*" # All internal routes
```
### Testing Wildcard Routes
**Config:**
```yaml
general_settings:
master_key: sk-1234
public_routes:
- "/public/*"
```
**Test:**
```shell
# This works without auth (matches /public/*)
curl http://localhost:4000/public/status
# This also works without auth (matches /public/*)
curl http://localhost:4000/public/health/detailed
# This requires auth (doesn't match /public/*)
curl http://localhost:4000/private/data
```

32
docs/my-website/docs/response_api.md
View file

@ -43,6 +43,38 @@ response = litellm.responses(
print(response)
```
#### Response Format (OpenAI Responses API Format)
```json
{
"id": "resp_abc123",
"object": "response",
"created_at": 1734366691,
"status": "completed",
"model": "o1-pro-2025-01-30",
"output": [
{
"type": "message",
"id": "msg_abc123",
"status": "completed",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Once upon a time, a little unicorn named Stardust lived in a magical meadow where flowers sang lullabies. One night, she discovered that her horn could paint dreams across the sky, and she spent the evening creating the most beautiful aurora for all the forest creatures to enjoy. As the animals drifted off to sleep beneath her shimmering lights, Stardust curled up on a cloud of moonbeams, happy to have shared her magic with her friends.",
"annotations": []
}
]
}
],
"usage": {
"input_tokens": 18,
"output_tokens": 98,
"total_tokens": 116
}
}
```
#### Streaming
```python showLineNumbers title="OpenAI Streaming Response"
import litellm

106
docs/my-website/sidebars.js
View file

@ -118,11 +118,83 @@ const sidebars = {
],
// But you can create a sidebar manually
tutorialSidebar: [
{ type: "doc", id: "index" }, // NEW
{ type: "doc", id: "index", label: "Getting Started" },
{
type: "category",
label: "LiteLLM AI Gateway",
label: "LiteLLM Python SDK",
items: [
{
type: "link",
label: "Quick Start",
href: "/docs/#litellm-python-sdk",
},
{
type: "category",
label: "SDK Functions",
items: [
{
type: "doc",
id: "completion/input",
label: "completion()",
},
{
type: "doc",
id: "embedding/supported_embedding",
label: "embedding()",
},
{
type: "doc",
id: "response_api",
label: "responses()",
},
{
type: "doc",
id: "text_completion",
label: "text_completion()",
},
{
type: "doc",
id: "image_generation",
label: "image_generation()",
},
{
type: "doc",
id: "audio_transcription",
label: "transcription()",
},
{
type: "doc",
id: "text_to_speech",
label: "speech()",
},
{
type: "link",
label: "All Supported Endpoints →",
href: "/docs/supported_endpoints",
},
],
},
{
type: "category",
label: "Configuration",
items: [
"set_keys",
"caching/all_caches",
],
},
"completion/token_usage",
"exception_mapping",
{
type: "category",
label: "LangChain, LlamaIndex, Instructor",
items: ["langchain/langchain", "tutorials/instructor"],
}
],
},
{
type: "category",
label: "LiteLLM AI Gateway (Proxy)",
link: {
type: "generated-index",
title: "LiteLLM AI Gateway (LLM Proxy)",
@ -226,6 +298,7 @@ const sidebars = {
"proxy/custom_auth",
"proxy/ip_address",
"proxy/multiple_admins",
"proxy/public_routes",
],
},
{
@ -578,6 +651,7 @@ const sidebars = {
"providers/bedrock_rerank",
"providers/bedrock_agentcore",
"providers/bedrock_agents",
"providers/bedrock_writer",
"providers/bedrock_batches",
"providers/bedrock_vector_store",
]
@ -614,6 +688,7 @@ const sidebars = {
"providers/github_copilot",
"providers/gradient_ai",
"providers/groq",
"providers/helicone",
"providers/heroku",
{
type: "category",
@ -667,6 +742,7 @@ const sidebars = {
]
},
"providers/sambanova",
"providers/sap",
"providers/snowflake",
"providers/togetherai",
"providers/topaz",
@ -694,6 +770,7 @@ const sidebars = {
type: "category",
label: "Guides",
items: [
"budget_manager",
"completion/computer_use",
"completion/web_search",
"completion/web_fetch",
@ -746,27 +823,6 @@ const sidebars = {
"wildcard_routing"
],
},
{
type: "category",
label: "LiteLLM Python SDK",
items: [
"set_keys",
"budget_manager",
"caching/all_caches",
"completion/token_usage",
"sdk_custom_pricing",
"embedding/async_embedding",
"embedding/moderation",
"migration",
"sdk_custom_pricing",
{
type: "category",
label: "LangChain, LlamaIndex, Instructor Integration",
items: ["langchain/langchain", "tutorials/instructor"],
}
],
},
{
type: "category",
label: "Load Testing",
@ -836,6 +892,8 @@ const sidebars = {
type: "category",
label: "Extras",
items: [
"sdk_custom_pricing",
"migration",
"data_security",
"data_retention",
"proxy/security_encryption_faq",
@ -850,7 +908,7 @@ const sidebars = {
"Learn how to deploy + call models from different providers on LiteLLM",
slug: "/project",
},
items: [
items: [
"projects/smolagents",
"projects/mini-swe-agent",
"projects/openai-agents",

16
litellm/__init__.py
View file

@ -265,6 +265,7 @@ heroku_key: Optional[str] = None
cometapi_key: Optional[str] = None
ovhcloud_key: Optional[str] = None
lemonade_key: Optional[str] = None
sap_service_key: Optional[str] = None
amazon_nova_api_key: Optional[str] = None
common_cloud_provider_auth_params: dict = {
"params": ["project", "region_name", "token"],
@ -1069,7 +1070,7 @@ from litellm.litellm_core_utils.core_helpers import remove_index_from_tool_calls
from litellm.litellm_core_utils.token_counter import get_modified_max_tokens
# client must be imported immediately as it's used as a decorator at function definition time
from .utils import client
# Note: Most other utils imports are lazy-loaded via __getattr__ to avoid loading utils.py
# Note: Most other utils imports are lazy-loaded via __getattr__ to avoid loading utils.py
# (which imports tiktoken) at import time
from .llms.bytez.chat.transformation import BytezChatConfig
@ -1110,6 +1111,7 @@ from .llms.jina_ai.rerank.transformation import JinaAIRerankConfig
from .llms.deepinfra.rerank.transformation import DeepinfraRerankConfig
from .llms.hosted_vllm.rerank.transformation import HostedVLLMRerankConfig
from .llms.nvidia_nim.rerank.transformation import NvidiaNimRerankConfig
from .llms.nvidia_nim.rerank.ranking_transformation import NvidiaNimRankingConfig
from .llms.vertex_ai.rerank.transformation import VertexAIRerankConfig
from .llms.fireworks_ai.rerank.transformation import FireworksAIRerankConfig
from .llms.clarifai.chat.transformation import ClarifaiConfig
@ -1241,6 +1243,7 @@ from .llms.topaz.common_utils import TopazModelInfo
from .llms.topaz.image_variations.transformation import TopazImageVariationConfig
from litellm.llms.openai.completion.transformation import OpenAITextCompletionConfig
from .llms.groq.chat.transformation import GroqChatConfig
from .llms.sap.chat.transformation import GenAIHubOrchestrationConfig
from .llms.voyage.embedding.transformation import VoyageEmbeddingConfig
from .llms.voyage.embedding.transformation_contextual import (
VoyageContextualEmbeddingConfig,
@ -1339,6 +1342,7 @@ from .llms.azure.chat.o_series_transformation import AzureOpenAIO1Config
from .llms.watsonx.completion.transformation import IBMWatsonXAIConfig
from .llms.watsonx.chat.transformation import IBMWatsonXChatConfig
from .llms.watsonx.embed.transformation import IBMWatsonXEmbeddingConfig
from .llms.sap.embed.transformation import GenAIHubEmbeddingConfig
from .llms.watsonx.audio_transcription.transformation import (
IBMWatsonXAudioTranscriptionConfig,
)
@ -1511,13 +1515,13 @@ def set_global_gitlab_config(config: Dict[str, Any]) -> None:
if TYPE_CHECKING:
from litellm.types.utils import ModelInfo as _ModelInfoType
# Cost calculator functions
cost_per_token: Callable[..., Tuple[float, float]]
completion_cost: Callable[..., float]
response_cost_calculator: Any
modify_integration: Any
# Utils functions - type stubs for truly lazy loaded functions only
# (functions NOT imported via "from .main import *")
get_response_string: Callable[..., str]
@ -1547,7 +1551,7 @@ if TYPE_CHECKING:
get_first_chars_messages: Callable[..., str]
get_provider_fields: Callable[..., List]
get_valid_models: Callable[..., list]
# Response types - truly lazy loaded only (not in main.py or elsewhere)
ModelResponseListIterator: Type[Any]
@ -1563,7 +1567,7 @@ def __getattr__(name: str) -> Any:
if name in _cost_calculator_names:
from ._lazy_imports import _lazy_import_cost_calculator
return _lazy_import_cost_calculator(name)
# Lazy load litellm_logging functions
_litellm_logging_names = (
"Logging",
@ -1572,7 +1576,7 @@ def __getattr__(name: str) -> Any:
if name in _litellm_logging_names:
from ._lazy_imports import _lazy_import_litellm_logging
return _lazy_import_litellm_logging(name)
# Lazy load utils functions
_utils_names = (
"exception_type", "get_optional_params", "get_response_string", "token_counter",

10
litellm/completion_extras/litellm_responses_transformation/transformation.py
View file

@ -873,8 +873,10 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
usage=None,
)
elif output_item.get("type") == "message":
# Don't emit is_finished=True here - there may be more output items
# (e.g., tool_calls) coming after the message. Wait for response.completed.
return GenericStreamingChunk(
finish_reason="stop", is_finished=True, usage=None, text=""
finish_reason="", is_finished=False, usage=None, text=""
)
elif event_type == "response.output_text.delta":
@ -907,6 +909,12 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
)
]
)
elif event_type == "response.completed":
# Response is fully complete - now we can signal is_finished=True
# This ensures we don't prematurely end the stream before tool_calls arrive
return GenericStreamingChunk(
text="", tool_use=None, is_finished=True, finish_reason="stop", usage=None
)
else:
pass
# For any unhandled event types, create a minimal valid chunk or skip

5
litellm/constants.py
View file

@ -345,6 +345,7 @@ LITELLM_CHAT_PROVIDERS = [
"huggingface",
"together_ai",
"datarobot",
"helicone",
"openrouter",
"cometapi",
"vertex_ai",
@ -553,6 +554,7 @@ openai_compatible_endpoints: List = [
"https://api.morphllm.com/v1",
"https://api.lambda.ai/v1",
"https://api.hyperbolic.xyz/v1",
"https://ai-gateway.helicone.ai/",
"https://ai-gateway.vercel.sh/v1",
"https://api.inference.wandb.ai/v1",
"https://api.clarifai.com/v2/ext/openai/v1",
@ -598,6 +600,7 @@ openai_compatible_providers: List = [
"moonshot",
"publicai",
"v0",
"helicone",
"morph",
"lambda_ai",
"hyperbolic",
@ -935,6 +938,8 @@ BEDROCK_CONVERSE_MODELS = [
"amazon.nova-lite-v1:0",
"amazon.nova-2-lite-v1:0",
"amazon.nova-pro-v1:0",
"writer.palmyra-x4-v1:0",
"writer.palmyra-x5-v1:0",
]

8
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -82,6 +82,14 @@ class ExceptionCheckers:
for substring in known_exception_substrings:
if substring in _error_str_lowercase:
return True
# Cerebras pattern: "Current length is X while limit is Y"
if (
"current length is" in _error_str_lowercase
and "while limit is" in _error_str_lowercase
):
return True
return False
@staticmethod

2
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -406,6 +406,8 @@ def get_llm_provider( # noqa: PLR0915
custom_llm_provider = "clarifai"
elif model.startswith("amazon_nova"):
custom_llm_provider = "amazon_nova"
elif model.startswith("sap/"):
custom_llm_provider = "sap"
if not custom_llm_provider:
if litellm.suppress_debug_info is False:
print() # noqa

5
litellm/litellm_core_utils/get_supported_openai_params.py
View file

@ -116,6 +116,11 @@ def get_supported_openai_params( # noqa: PLR0915
f"Unsupported provider config: {transcription_provider_config} for model: {model}"
)
return litellm.OpenAIConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "sap":
if request_type == "chat_completion":
return litellm.GenAIHubOrchestrationConfig().get_supported_openai_params(model=model)
elif request_type == "embeddings":
return litellm.GenAIHubEmbeddingConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "azure":
if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
return litellm.AzureOpenAIO1Config().get_supported_openai_params(

1
litellm/litellm_core_utils/streaming_handler.py
View file

@ -441,7 +441,6 @@ class CustomStreamWrapper:
finish_reason = None
logprobs = None
usage = None
if str_line and str_line.choices and len(str_line.choices) > 0:
if (
str_line.choices[0].delta is not None

7
litellm/llms/azure_ai/anthropic/messages_transformation.py
View file

@ -48,12 +48,7 @@ class AzureAnthropicMessagesConfig(AnthropicMessagesConfig):
headers = BaseAzureLLM._base_validate_azure_environment(
headers=headers, litellm_params=litellm_params_obj
)
# Azure Anthropic uses x-api-key header (not api-key)
# Convert api-key to x-api-key if present
if "api-key" in headers and "x-api-key" not in headers:
headers["x-api-key"] = headers.pop("api-key")
# Set anthropic-version header
if "anthropic-version" not in headers:
headers["anthropic-version"] = "2023-06-01"

5
litellm/llms/azure_ai/anthropic/transformation.py
View file

@ -55,11 +55,6 @@ class AzureAnthropicConfig(AnthropicConfig):
headers = BaseAzureLLM._base_validate_azure_environment(
headers=headers, litellm_params=litellm_params_obj
)
# Azure Anthropic uses x-api-key header (not api-key)
# Convert api-key to x-api-key if present
if "api-key" in headers and "x-api-key" not in headers:
headers["x-api-key"] = headers.pop("api-key")
# Get tools and other anthropic-specific setup
tools = optional_params.get("tools")

2
litellm/llms/dashscope/chat/transformation.py
View file

@ -51,7 +51,7 @@ class DashScopeChatConfig(OpenAIGPTConfig):
api_base = (
api_base
or get_secret_str("DASHSCOPE_API_BASE")
or "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
or "https://dashscope.aliyuncs.com/compatible-mode/v1"
) # type: ignore
dynamic_api_key = api_key or get_secret_str("DASHSCOPE_API_KEY")
return api_base, dynamic_api_key

28
litellm/llms/nvidia_nim/rerank/common_utils.py Normal file
View file

@ -0,0 +1,28 @@
"""
Common utilities for NVIDIA NIM rerank provider.
"""
def get_nvidia_nim_rerank_config(model: str):
"""
Get the appropriate NVIDIA NIM rerank config based on the model.
Args:
model: The model string (e.g., "nvidia/llama-3.2-nv-rerankqa-1b-v2" or "ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2")
Returns:
NvidiaNimRankingConfig if model starts with "ranking/", else NvidiaNimRerankConfig
Example:
- "ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2" -> NvidiaNimRankingConfig
- "nvidia/llama-3.2-nv-rerankqa-1b-v2" -> NvidiaNimRerankConfig
"""
from litellm.llms.nvidia_nim.rerank.ranking_transformation import (
NvidiaNimRankingConfig,
)
from litellm.llms.nvidia_nim.rerank.transformation import NvidiaNimRerankConfig
if model.startswith("ranking/"):
return NvidiaNimRankingConfig()
return NvidiaNimRerankConfig()

75
litellm/llms/nvidia_nim/rerank/ranking_transformation.py Normal file
View file

@ -0,0 +1,75 @@
"""
Transformation for NVIDIA NIM Ranking models that use /v1/ranking endpoint.
Use this by passing "nvidia_nim/ranking/<model>" to force the /v1/ranking endpoint.
Reference: https://build.nvidia.com/nvidia/llama-3_2-nv-rerankqa-1b-v2/deploy
"""
from typing import Dict, Optional
from litellm.llms.nvidia_nim.rerank.transformation import NvidiaNimRerankConfig
class NvidiaNimRankingConfig(NvidiaNimRerankConfig):
"""
Configuration for NVIDIA NIM models that use the /v1/ranking endpoint.
Example:
curl -X "POST" 'https://ai.api.nvidia.com/v1/ranking' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"model": "nvidia/llama-3.2-nv-rerankqa-1b-v2",
"query": {"text": "which way did the traveler go?"},
"passages": [{"text": "..."}, {"text": "..."}],
"truncate": "END"
}'
"""
def _get_clean_model_name(self, model: str) -> str:
"""Strip 'ranking/' prefix from model name."""
if model.startswith("ranking/"):
return model[len("ranking/"):]
return model
def get_complete_url(
self,
api_base: Optional[str],
model: str,
optional_params: Optional[dict] = None,
) -> str:
"""
Construct the Nvidia NIM ranking URL.
Format: {api_base}/v1/ranking
"""
if not api_base:
api_base = self.DEFAULT_NIM_RERANK_API_BASE
api_base = api_base.rstrip("/")
if api_base.endswith("/ranking"):
return api_base
if api_base.endswith("/v1"):
api_base = api_base[:-3]
return f"{api_base}/v1/ranking"
def transform_rerank_request(
self,
model: str,
optional_rerank_params: Dict,
headers: dict,
) -> dict:
"""
Transform request, using clean model name without 'ranking/' prefix.
"""
clean_model = self._get_clean_model_name(model)
return super().transform_rerank_request(
model=clean_model,
optional_rerank_params=optional_rerank_params,
headers=headers,
)

6
litellm/llms/openai/chat/gpt_transformation.py
View file

@ -168,9 +168,11 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
): # gpt-4 does not support 'response_format'
model_specific_params.append("response_format")
# Normalize model name for responses API (e.g., "responses/gpt-4.1" -> "gpt-4.1")
model_for_check = model.split("responses/", 1)[1] if "responses/" in model else model
if (
model in litellm.open_ai_chat_completion_models
) or model in litellm.open_ai_text_completion_models:
model_for_check in litellm.open_ai_chat_completion_models
) or model_for_check in litellm.open_ai_text_completion_models:
model_specific_params.append(
"user"
) # user is not a param supported by all openai-compatible endpoints - e.g. azure ai

4
litellm/llms/openai_like/providers.json
View file

@ -10,5 +10,9 @@
"special_handling": {
"convert_content_list_to_string": true
}
},
"helicone": {
"base_url": "https://ai-gateway.helicone.ai/",
"api_key_env": "HELICONE_API_KEY"
}
}

1
litellm/llms/sap/chat/__init__.py Executable file
View file

@ -0,0 +1 @@

262
litellm/llms/sap/chat/handler.py Executable file
View file

@ -0,0 +1,262 @@
from __future__ import annotations
import json
import time
import httpx
from typing import Iterator, Optional, AsyncIterator
from litellm.llms.base_llm.chat.transformation import BaseConfig
from litellm.types.llms.openai import OpenAIChatCompletionChunk
from ...custom_httpx.llm_http_handler import BaseLLMHTTPHandler
# -------------------------------
# Errors
# -------------------------------
class GenAIHubOrchestrationError(Exception):
def __init__(self, status_code: int, message: str):
super().__init__(message)
self.status_code = status_code
self.message = message
# -------------------------------
# Stream parsing helpers
# -------------------------------
def _now_ts() -> int:
return int(time.time())
def _is_terminal_chunk(chunk: OpenAIChatCompletionChunk) -> bool:
"""OpenAI-shaped chunk is terminal if any choice has a non-None finish_reason."""
try:
for ch in chunk.choices or []:
if ch.finish_reason is not None:
return True
except Exception:
pass
return False
class _StreamParser:
"""Normalize orchestration streaming events into OpenAI-like chunks."""
@staticmethod
def _from_orchestration_result(evt: dict) -> Optional[OpenAIChatCompletionChunk]:
"""
Accepts orchestration_result shape and maps it to an OpenAI-like *chunk*.
"""
orc = evt.get("orchestration_result") or {}
if not orc:
return None
return OpenAIChatCompletionChunk.model_validate(
{
"id": orc.get("id") or evt.get("request_id") or "stream-chunk",
"object": orc.get("object") or "chat.completion.chunk",
"created": orc.get("created") or evt.get("created") or _now_ts(),
"model": orc.get("model") or "unknown",
"choices": [
{
"index": c.get("index", 0),
"delta": c.get("delta") or {},
"finish_reason": c.get("finish_reason"),
}
for c in (orc.get("choices") or [])
],
}
)
@staticmethod
def to_openai_chunk(event_obj: dict) -> Optional[OpenAIChatCompletionChunk]:
"""
Accepts:
- {"final_result": <openai-style CHUNK>} (IMPORTANT: this is just another chunk, NOT terminal)
- {"orchestration_result": {...}} (map to chunk)
- already-openai-shaped chunks
- other events (ignored)
Raises:
- ValueError for in-stream error objects
"""
# In-stream error per spec (surface as exception)
if "code" in event_obj or "error" in event_obj:
raise ValueError(json.dumps(event_obj))
# FINAL RESULT IS *NOT* TERMINAL: treat it as the next chunk
if "final_result" in event_obj:
fr = event_obj["final_result"] or {}
# ensure it looks like an OpenAI chunk
if "object" not in fr:
fr["object"] = "chat.completion.chunk"
return OpenAIChatCompletionChunk.model_validate(fr)
# Orchestration incremental delta
if "orchestration_result" in event_obj:
return _StreamParser._from_orchestration_result(event_obj)
# Already an OpenAI-like chunk
if "choices" in event_obj and "object" in event_obj:
return OpenAIChatCompletionChunk.model_validate(event_obj)
# Unknown / heartbeat / metrics
return None
# -------------------------------
# Iterators
# -------------------------------
class SAPStreamIterator:
"""
Sync iterator over an httpx streaming response that yields OpenAIChatCompletionChunk.
Accepts both SSE `data: ...` and raw JSON lines. Closes on terminal chunk or [DONE].
"""
def __init__(
self,
response: Iterator,
event_prefix: str = "data: ",
final_msg: str = "[DONE]",
):
self._resp = response
self._iter = response
self._prefix = event_prefix
self._final = final_msg
self._done = False
def __iter__(self) -> Iterator[OpenAIChatCompletionChunk]:
return self
def __next__(self) -> OpenAIChatCompletionChunk:
if self._done:
raise StopIteration
for raw in self._iter:
line = (raw or "").strip()
if not line:
continue
payload = (
line[len(self._prefix) :] if line.startswith(self._prefix) else line
)
if payload == self._final:
self._safe_close()
raise StopIteration
try:
obj = json.loads(payload)
except Exception:
continue
try:
chunk = _StreamParser.to_openai_chunk(obj)
except ValueError as e:
self._safe_close()
raise e
if chunk is None:
continue
# Close on terminal
if _is_terminal_chunk(chunk):
self._safe_close()
return chunk
self._safe_close()
raise StopIteration
def _safe_close(self) -> None:
if self._done:
return
else:
self._done = True
class AsyncSAPStreamIterator:
sync_stream = False
def __init__(
self,
response:AsyncIterator,
event_prefix: str = "data: ",
final_msg: str = "[DONE]",
):
self._resp = response
self._prefix = event_prefix
self._final = final_msg
self._line_iter = None
self._done = False
def __aiter__(self):
return self
async def __anext__(self):
if self._done:
raise StopAsyncIteration
if self._line_iter is None:
self._line_iter = self._resp
while True:
try:
raw = await self._line_iter.__anext__()
except (StopAsyncIteration, httpx.ReadError, OSError):
await self._aclose()
raise StopAsyncIteration
line = (raw or "").strip()
if not line:
continue
# now = lambda: int(time.time() * 1000)
payload = (
line[len(self._prefix) :] if line.startswith(self._prefix) else line
)
if payload == self._final:
await self._aclose()
raise StopAsyncIteration
try:
obj = json.loads(payload)
except Exception:
continue
try:
chunk = _StreamParser.to_openai_chunk(obj)
except ValueError as e:
await self._aclose()
raise GenAIHubOrchestrationError(502, str(e))
if chunk is None:
continue
# If terminal, close BEFORE returning. Next __anext__() will stop immediately.
if any(c.finish_reason is not None for c in (chunk.choices or [])):
await self._aclose()
return chunk
async def _aclose(self):
if self._done:
return
else:
self._done = True
# -------------------------------
# LLM handler
# -------------------------------
class GenAIHubOrchestration(BaseLLMHTTPHandler):
def _add_stream_param_to_request_body(
self,
data: dict,
provider_config: BaseConfig,
fake_stream: bool
):
if data.get("config", {}).get("stream", None) is not None:
data["config"]["stream"]["enabled"] = True
else:
data["config"]["stream"] = {"enabled": True}
return data

112
litellm/llms/sap/chat/models.py Normal file
View file

@ -0,0 +1,112 @@
from typing import Union, Literal
from pydantic import BaseModel, Field, field_validator
def validate_different_content(v: Union[str, dict, list]) -> str:
if v in ((), {}, []):
return ""
elif isinstance(v, dict) and "text" in v:
return v['text']
elif isinstance(v, list):
new_v = []
for item in v:
if isinstance(item, dict) and "text" in item:
if item['text']:
new_v.append(item['text'])
elif isinstance(item, str):
new_v.append(item)
return '\n'.join(new_v)
elif isinstance(v, str):
return v
raise ValueError("Content must be a string")
return v
class TextContent(BaseModel):
type_: Literal["text"] = Field(default="text", alias="type")
text: str
class ImageURLContent(BaseModel):
url: str
detail: str = "auto"
class ImageContent(BaseModel):
type_: Literal["image_url"] = Field(default="image_url", alias="type")
image_url: ImageURLContent
class FunctionObj(BaseModel):
name: str
arguments: str
class FunctionTool(BaseModel):
description: str = ""
name: str
parameters: dict = {}
strict: bool = False
class ChatCompletionTool(BaseModel):
type_: Literal["function"] = Field(default="function", alias="type")
function: FunctionTool
class MessageToolCall(BaseModel):
id: str
type_: Literal["function"] = Field(default="function", alias="type")
function: FunctionObj
class SAPMessage(BaseModel):
"""
Model for SystemChatMessage and DeveloperChatMessage
"""
role: Literal["system", "developer"] = "system"
content: str
_content_validator = field_validator("content", mode="before")(validate_different_content)
class SAPUserMessage(BaseModel):
role: Literal["user"] = "user"
content: Union[
str, TextContent, ImageContent, list[Union[TextContent, ImageContent]]
]
class SAPAssistantMessage(BaseModel):
role: Literal["assistant"] = "assistant"
content: str = ""
refusal: str = ""
tool_calls: list[MessageToolCall] = []
_content_validator = field_validator("content", mode="before")(validate_different_content)
class SAPToolChatMessage(BaseModel):
role: Literal["tool"] = "tool"
tool_call_id: str
content: str
_content_validator = field_validator("content", mode="before")(validate_different_content)
class ResponseFormat(BaseModel):
type_: Literal["text", "json_object"] = Field(default="text", alias="type")
class JSONResponseSchema(BaseModel):
description: str = ""
name: str
schema_: dict = Field(default_factory=dict, alias="schema")
strict: bool = False
class ResponseFormatJSONSchema(BaseModel):
type_: Literal["json_schema"] = Field(default="json_schema", alias="type")
json_schema: JSONResponseSchema

299
litellm/llms/sap/chat/transformation.py Executable file
View file

@ -0,0 +1,299 @@
"""
Translate from OpenAI's `/v1/chat/completions` to SAP Generative AI Hub's Orchestration Service`v2/completion`
"""
from typing import List, Optional, Union, Dict, Tuple, Any, TYPE_CHECKING, Iterator, AsyncIterator
from functools import cached_property
import litellm
import httpx
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
from ..credentials import get_token_creator
from .models import (
SAPMessage,
SAPAssistantMessage,
SAPToolChatMessage,
ChatCompletionTool,
ResponseFormatJSONSchema,
ResponseFormat,
SAPUserMessage,
)
from .handler import GenAIHubOrchestrationError, AsyncSAPStreamIterator, SAPStreamIterator
def validate_dict(data: dict, model) -> dict:
return model(**data).model_dump(by_alias=True)
class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
frequency_penalty: Optional[int] = None
function_call: Optional[Union[str, dict]] = None
functions: Optional[list] = None
logit_bias: Optional[dict] = None
max_tokens: Optional[int] = None
n: Optional[int] = None
presence_penalty: Optional[int] = None
stop: Optional[Union[str, list]] = None
temperature: Optional[int] = None
top_p: Optional[int] = None
response_format: Optional[dict] = None
tools: Optional[list] = None
tool_choice: Optional[Union[str, dict]] = None #
model_version: str = "latest"
def __init__(
self,
frequency_penalty: Optional[int] = None,
function_call: Optional[Union[str, dict]] = None,
functions: Optional[list] = None,
logit_bias: Optional[dict] = None,
max_tokens: Optional[int] = None,
n: Optional[int] = None,
presence_penalty: Optional[int] = None,
stop: Optional[Union[str, list]] = None,
temperature: Optional[int] = None,
top_p: Optional[int] = None,
response_format: Optional[dict] = None,
tools: Optional[list] = None,
tool_choice: Optional[Union[str, dict]] = None,
) -> None:
locals_ = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
self.token_creator = None
self._base_url = None
self._resource_group = None
def run_env_setup(self, service_key: Optional[str] = None) -> None:
try:
self.token_creator, self._base_url, self._resource_group = get_token_creator(service_key) # type: ignore
except ValueError as err:
raise GenAIHubOrchestrationError(status_code=400, message=err.args[0])
@property
def headers(self) -> Dict[str, str]:
if self.token_creator is None:
self.run_env_setup()
access_token = self.token_creator() # type: ignore
return {
"Authorization": access_token,
"AI-Resource-Group": self.resource_group,
"Content-Type": "application/json",
}
@property
def base_url(self) -> str:
if self._base_url is None:
self.run_env_setup()
return self._base_url # type: ignore
@property
def resource_group(self) -> str:
if self._resource_group is None:
self.run_env_setup()
return self._resource_group # type: ignore
@cached_property
def deployment_url(self) -> str:
# Keep a short, tight client lifecycle here to avoid fd leaks
client = litellm.module_level_client
# with httpx.Client(timeout=30) as client:
deployments = client.get(
f"{self.base_url}/lm/deployments", headers=self.headers
).json()
valid: List[Tuple[str, str]] = []
for dep in deployments.get("resources", []):
if dep.get("scenarioId") == "orchestration":
cfg = client.get(
f'{self.base_url}/lm/configurations/{dep["configurationId"]}',
headers=self.headers,
).json()
if cfg.get("executableId") == "orchestration":
valid.append((dep["deploymentUrl"], dep["createdAt"]))
# newest first
return sorted(valid, key=lambda x: x[1], reverse=True)[0][0]
@classmethod
def get_config(cls):
return super().get_config()
def get_supported_openai_params(self, model):
params = [
"frequency_penalty",
"logit_bias",
"logprobs",
"top_logprobs",
"max_tokens",
"max_completion_tokens",
"prediction",
"n",
"presence_penalty",
"seed",
"stop",
"stream",
"stream_options",
"temperature",
"top_p",
"tools",
"tool_choice",
"function_call",
"functions",
"extra_headers",
"parallel_tool_calls",
"response_format",
"timeout",
]
if (
model.startswith('anthropic')
or model.startswith("amazon")
or model.startswith("cohere")
or model.startswith("alephalpha")
or model == "gpt-4"
):
params.remove("response_format")
if model.startswith("gemini") or model.startswith("amazon"):
params.remove("tool_choice")
return params
def validate_environment(
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
if api_key:
self.run_env_setup(api_key)
return self.headers
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
):
api_base_ = f"{self.deployment_url}/v2/completion"
return api_base_
def transform_request(
self,
model: str,
messages: List[Dict[str, str]], # type: ignore
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
supported_params = self.get_supported_openai_params(model)
model_params = {
k: v for k, v in optional_params.items() if k in supported_params
}
model_version = optional_params.pop("model_version", "latest")
template = []
for message in messages:
if message["role"] == "user":
template.append(validate_dict(message, SAPUserMessage))
elif message["role"] == "assistant":
template.append(validate_dict(message, SAPAssistantMessage))
elif message["role"] == "tool":
template.append(validate_dict(message, SAPToolChatMessage))
else:
template.append(validate_dict(message, SAPMessage))
tools_ = optional_params.pop("tools", [])
tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_]
if tools_ != []:
tools = {"tools": tools_}
else:
tools = {}
response_format = model_params.pop("response_format", {})
resp_type = response_format.get("type", None)
if resp_type:
if resp_type== "json_schema":
response_format = validate_dict(response_format, ResponseFormatJSONSchema)
else:
response_format = validate_dict(response_format, ResponseFormat)
response_format = {"response_format": response_format}
model_params.pop("stream", False)
stream_config = {}
if "stream_options" in model_params:
# stream_config["enabled"] = True
stream_options = model_params.pop("stream_options", {})
stream_config["chunk_size"] = stream_options.get("chunk_size", 100)
if "delimiters" in stream_options:
stream_config["delimiters"] = stream_options.get("delimiters")
# else:
# stream_config["enabled"] = False
config = {
"config": {
"modules": {
"prompt_templating": {
"prompt": {
"template": template,
**tools,
**response_format
},
"model": {
"name": model,
"params": model_params,
"version": model_version,
},
},
},
"stream": stream_config,
}
}
return config
def transform_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ModelResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ModelResponse:
logging_obj.post_call(
input=messages,
api_key=api_key,
original_response=raw_response.text,
additional_args={"complete_input_dict": request_data},
)
return ModelResponse.model_validate(raw_response.json()["final_result"])
def get_model_response_iterator(
self,
streaming_response: Union[Iterator[str], AsyncIterator[str], "ModelResponse"],
sync_stream: bool,
json_mode: Optional[bool] = False,
):
if sync_stream:
return SAPStreamIterator(response=streaming_response) # type: ignore
else:
return AsyncSAPStreamIterator(response=streaming_response) # type: ignore

325
litellm/llms/sap/credentials.py Normal file
View file

@ -0,0 +1,325 @@
from __future__ import annotations
from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple
from datetime import datetime, timedelta, timezone
from threading import Lock
from pathlib import Path
from dataclasses import dataclass
import json
import os
import tempfile
from litellm import sap_service_key
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
AUTH_ENDPOINT_SUFFIX = "/oauth/token"
CONFIG_FILE_ENV_VAR = "AICORE_CONFIG"
HOME_PATH_ENV_VAR = "AICORE_HOME"
PROFILE_ENV_VAR = "AICORE_PROFILE"
VCAP_SERVICES_ENV_VAR = "VCAP_SERVICES"
VCAP_AICORE_SERVICE_NAME = "aicore"
SERVICE_KEY_ENV_VAR = "AICORE_SERVICE_KEY"
DEFAULT_HOME_PATH = os.path.join(os.path.expanduser("~"), ".aicore")
def _get_home() -> str:
return os.getenv(HOME_PATH_ENV_VAR, DEFAULT_HOME_PATH)
def _get_nested(d: Dict[str, Any], path: Sequence[str]) -> Any:
cur: Any = d
for k in path:
if not isinstance(cur, dict) or k not in cur:
raise KeyError(".".join(path))
cur = cur[k]
return cur
def _load_json_env(var_name: str) -> Optional[Dict[str, Any]]:
raw = os.environ.get(var_name)
if not raw:
return None
try:
return json.loads(raw)
except json.JSONDecodeError:
return None
def _load_vcap() -> Dict[str, Any]:
return _load_json_env(VCAP_SERVICES_ENV_VAR) or {}
def _get_vcap_service(label: str) -> Optional[Dict[str, Any]]:
for services in _load_vcap().values():
for svc in services:
if svc.get("label") == label:
return svc
return None
@dataclass(frozen=True)
class CredentialsValue:
name: str
vcap_key: Optional[Tuple[str, ...]] = None
default: Optional[str] = None
transform_fn: Optional[Callable[[str], str]] = None
CREDENTIAL_VALUES: Final[List[CredentialsValue]] = [
CredentialsValue("client_id", ("clientid",)),
CredentialsValue("client_secret", ("clientsecret",)),
CredentialsValue(
"auth_url",
("url",),
transform_fn=lambda url: url.rstrip("/")
+ ("" if url.endswith(AUTH_ENDPOINT_SUFFIX) else AUTH_ENDPOINT_SUFFIX),
),
CredentialsValue(
"base_url",
("serviceurls", "AI_API_URL"),
transform_fn=lambda url: url.rstrip("/")
+ ("" if url.endswith("/v2") else "/v2"),
),
CredentialsValue("resource_group", default="default"),
CredentialsValue(
"cert_url",
("certurl",),
transform_fn=lambda url: url.rstrip("/")
+ ("" if url.endswith(AUTH_ENDPOINT_SUFFIX) else AUTH_ENDPOINT_SUFFIX),
),
# file paths (kept for config compatibility)
CredentialsValue("cert_file_path"),
CredentialsValue("key_file_path"),
# inline PEMs from VCAP
CredentialsValue(
"cert_str", ("certificate",), transform_fn=lambda s: s.replace("\\n", "\n")
),
CredentialsValue(
"key_str", ("key",), transform_fn=lambda s: s.replace("\\n", "\n")
),
]
def init_conf(profile: Optional[str] = None) -> Dict[str, Any]:
"""
Loads config JSON from:
1) $AICORE_CONFIG if set, otherwise
2) $AICORE_HOME/config.json (or config_<profile>.json when profile is given/not default)
Returns {} when nothing is found.
"""
home = Path(_get_home())
profile = profile or os.environ.get(PROFILE_ENV_VAR)
cfg_env = os.getenv(CONFIG_FILE_ENV_VAR)
cfg_path = (
Path(cfg_env)
if cfg_env
else (
home
/ (
"config.json"
if profile in (None, "", "default")
else f"config_{profile}.json"
)
)
)
if cfg_path and cfg_path.exists():
try:
with cfg_path.open(encoding="utf-8") as f:
return json.load(f)
except json.JSONDecodeError:
raise KeyError(f"{cfg_path} is not valid JSON. Please fix or remove it!")
# If an explicit non-default profile was requested but not found, raise.
if cfg_env or (profile not in (None, "", "default")):
raise FileNotFoundError(
f"Unable to locate profile config file at '{cfg_path}' in AICORE_HOME '{home}'"
)
return {}
def _env_name(name: str) -> str:
return f"AICORE_{name.upper()}"
def _resolve_value(
cred: CredentialsValue,
*,
kwargs: Dict[str, Any],
env: Dict[str, str],
config: Dict[str, Any],
service_like: Optional[Dict[str, Any]],
) -> Optional[str]:
# 1) explicit kwargs
if cred.name in kwargs and kwargs[cred.name] is not None:
return kwargs[cred.name]
# 2) environment variables (primary name)
env_key = _env_name(cred.name)
if env_key in env and env[env_key] is not None:
return env[env_key]
# 3) config file (accept both prefixed and plain keys)
for key in (env_key, cred.name):
if key in config and config[key] is not None:
return config[key]
# 4) service-like source (AICORE_SERVICE_KEY first, else VCAP)
if service_like and cred.vcap_key:
try:
val = _get_nested(service_like, ("credentials",) + cred.vcap_key)
if val is not None:
return val
except KeyError:
pass
# 5) default
return cred.default
def fetch_credentials(service_key: Optional[str] = None, profile: Optional[str] = None, **kwargs) -> Dict[str, str]:
"""
Resolution order per key:
kwargs
> env (AICORE_<NAME>)
> config (AICORE_<NAME> or plain <name>)
> service-like source from JSON in $AICORE_SERVICE_KEY (same structure as a VCAP service object)
falling back to service entry in $VCAP_SERVICES with label 'aicore'
> default
"""
config = init_conf(profile)
env = os.environ # snapshot for testability
service_like = None
if not config:
# Prefer AICORE_SERVICE_KEY if present; otherwise fall back to the VCAP service.
service_like = service_key or sap_service_key or _load_json_env(SERVICE_KEY_ENV_VAR) or _get_vcap_service(
VCAP_AICORE_SERVICE_NAME
)
out: Dict[str, str] = {}
for cred in CREDENTIAL_VALUES:
value = _resolve_value(cred, kwargs=kwargs, env=env, config=config, service_like=service_like) # type: ignore
if value is None:
continue
if cred.transform_fn:
value = cred.transform_fn(value)
out[cred.name] = value
if "cert_url" in out.keys():
out["auth_url"] = out.pop("cert_url")
return out
def get_token_creator(
service_key: Optional[str] = None,
profile: Optional[str] = None,
*,
timeout: float = 30.0,
expiry_buffer_minutes: int = 60,
**overrides,
) -> Tuple[Callable[[], str], str, str]:
"""
Creates a callable that fetches and caches an OAuth2 bearer token
using credentials from `fetch_credentials()`.
The callable:
- Automatically loads credentials via fetch_credentials(profile, **overrides)
- Fetches a new token only if expired or near expiry
- Caches token thread-safely with a configurable refresh buffer
Args:
profile: Optional AICore profile name
timeout: HTTP request timeout in seconds (default 30s)
expiry_buffer_minutes: Refresh the token this many minutes before expiry
overrides: Any explicit credential overrides (client_id, client_secret, etc.)
Returns:
Callable[[], str]: function returning a valid "Bearer <token>" string.
"""
# Resolve credentials using your helper
credentials: Dict[str, str] = fetch_credentials(service_key=service_key, profile=profile, **overrides)
auth_url = credentials.get("auth_url")
client_id = credentials.get("client_id")
client_secret = credentials.get("client_secret")
cert_str = credentials.get("cert_str")
key_str = credentials.get("key_str")
cert_file_path = credentials.get("cert_file_path")
key_file_path = credentials.get("key_file_path")
# Sanity check
if not auth_url or not client_id:
raise ValueError(
"fetch_credentials did not return valid 'auth_url' or 'client_id'"
)
modes = [
client_secret is not None,
(cert_str is not None and key_str is not None),
(cert_file_path is not None and key_file_path is not None),
]
if sum(bool(m) for m in modes) != 1:
raise ValueError(
"Invalid credentials: provide exactly one of client_secret, "
"(cert_str & key_str), or (cert_file_path & key_file_path)."
)
lock = Lock()
token: Optional[str] = None
token_expiry: Optional[datetime] = None
def _request_token(cert_pair=None) -> tuple[str, datetime]:
data = {"grant_type": "client_credentials", "client_id": client_id}
if client_secret:
data["client_secret"] = client_secret
client = _get_httpx_client()
# with httpx.Client(cert=cert_pair, timeout=timeout) as client:
resp = client.post(auth_url, data=data)
try:
resp.raise_for_status()
payload = resp.json()
access_token = payload["access_token"]
expires_in = int(payload.get("expires_in", 3600))
expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in)
return f"Bearer {access_token}", expiry_date
except Exception as e:
msg = getattr(resp, "text", str(e))
raise RuntimeError(f"Token request failed: {msg}") from e
def _fetch_token() -> tuple[str, datetime]:
# Case 1: secret-based auth
if client_secret:
return _request_token()
# Case 2: cert/key strings
if cert_str and key_str:
cert_str_fixed = cert_str.replace("\\n", "\n")
key_str_fixed = key_str.replace("\\n", "\n")
with tempfile.TemporaryDirectory() as tmp:
cert_path = os.path.join(tmp, "cert.pem")
key_path = os.path.join(tmp, "key.pem")
with open(cert_path, "w") as f:
f.write(cert_str_fixed)
with open(key_path, "w") as f:
f.write(key_str_fixed)
return _request_token(cert_pair=(cert_path, key_path))
# Case 3: file-based cert/key
return _request_token(cert_pair=(cert_file_path, key_file_path))
def get_token() -> str:
nonlocal token, token_expiry
with lock:
now = datetime.now(timezone.utc)
if (
token is None
or token_expiry is None
or token_expiry - now < timedelta(minutes=expiry_buffer_minutes)
):
token, token_expiry = _fetch_token()
return token
return get_token, credentials["base_url"], credentials["resource_group"]

176
litellm/llms/sap/embed/transformation.py Normal file
View file

@ -0,0 +1,176 @@
"""
Translates from OpenAI's `/v1/embeddings` to IBM's `/text/embeddings` route.
"""
from typing import Optional, List, Dict, Literal
from pydantic import BaseModel, Field
from functools import cached_property
import httpx
from litellm.llms.base_llm.embedding.transformation import (
BaseEmbeddingConfig,
LiteLLMLoggingObj,
)
from litellm.types.llms.openai import AllEmbeddingInputValues
from litellm.types.utils import EmbeddingResponse
from ..chat.handler import GenAIHubOrchestrationError
from ..credentials import get_token_creator
class Usage(BaseModel):
prompt_tokens: int
total_tokens: int
class EmbeddingItem(BaseModel):
object: Literal["embedding"]
embedding: List[float] = Field(
..., description="Vector of floats (length varies by model)."
)
index: int
class FinalResult(BaseModel):
object: Literal["list"]
data: List[EmbeddingItem]
model: str
usage: Usage
class EmbeddingsResponse(BaseModel):
request_id: str
final_result: FinalResult
class EmbeddingModel(BaseModel):
name: str
version: str = "latest"
params: dict = Field(default_factory=dict, validation_alias="parameters")
class EmbeddingsModules(BaseModel):
embeddings: EmbeddingModel
class EmbeddingInput(BaseModel):
text: str | List[str]
type: Literal["text", "document", "query"] = "text"
class EmbeddingRequest(BaseModel):
config: EmbeddingsModules
input: EmbeddingInput
def validate_dict(data: dict, model) -> dict:
return model(**data).model_dump()
class GenAIHubEmbeddingConfig(BaseEmbeddingConfig):
def __init__(self):
super().__init__()
self._access_token_data = {}
self.token_creator, self.base_url, self.resource_group = get_token_creator()
@property
def headers(self) -> Dict:
access_token = self.token_creator()
# headers for completions and embeddings requests
headers = {
"Authorization": access_token,
"AI-Resource-Group": self.resource_group,
"Content-Type": "application/json",
}
return headers
@cached_property
def deployment_url(self) -> str:
with httpx.Client(timeout=30) as client:
valid_deployments = []
deployments = client.get(
self.base_url + "/lm/deployments", headers=self.headers
).json()
for deployment in deployments.get("resources", []):
if deployment["scenarioId"] == "orchestration":
config_details = client.get(
self.base_url
+ f'/lm/configurations/{deployment["configurationId"]}',
headers=self.headers,
).json()
if config_details["executableId"] == "orchestration":
valid_deployments.append(
(deployment["deploymentUrl"], deployment["createdAt"])
)
return sorted(valid_deployments, key=lambda x: x[1], reverse=True)[0][0]
def get_error_class(self, error_message, status_code, headers):
return GenAIHubOrchestrationError(status_code, error_message)
def get_supported_openai_params(self, model: str) -> list:
if "text-embedding-3" in model:
return ["encoding_format", "dimensions"]
else:
return [
"encoding_format",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
return optional_params
def validate_environment(self, headers: dict, *args, **kwargs) -> dict:
return self.headers
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
url = self.deployment_url.rstrip("/") + "/v2/embeddings"
return url
def transform_embedding_request(
self,
model: str,
input: AllEmbeddingInputValues,
optional_params: dict,
headers: dict,
) -> dict:
model_dict = {}
model_dict["name"] = model
model_dict["version"] = optional_params.get("version", "latest")
model_dict["params"] = optional_params.get("parameters", {})
input_dict = {"text": input}
body = {
"config": {
"modules": {
"embeddings": {"model": validate_dict(model_dict, EmbeddingModel)}
}
},
"input": validate_dict(input_dict, EmbeddingInput),
}
return body
def transform_embedding_response(
self,
model: str,
raw_response: httpx.Response,
model_response: EmbeddingResponse,
logging_obj: LiteLLMLoggingObj,
api_key: Optional[str],
request_data: dict,
optional_params: dict,
litellm_params: dict,
) -> EmbeddingResponse:
return EmbeddingResponse.model_validate(raw_response.json()["final_result"])

46
litellm/main.py
View file

@ -176,6 +176,7 @@ from .llms.databricks.embed.handler import DatabricksEmbeddingHandler
from .llms.deprecated_providers import aleph_alpha, palm
from .llms.gemini.common_utils import get_api_key_from_env
from .llms.groq.chat.handler import GroqChatCompletion
from .llms.sap.chat.handler import GenAIHubOrchestration
from .llms.heroku.chat.transformation import HerokuChatConfig
from .llms.huggingface.embedding.handler import HuggingFaceEmbedding
from .llms.lemonade.chat.transformation import LemonadeChatConfig
@ -255,6 +256,8 @@ openai_text_completions = OpenAITextCompletion()
openai_audio_transcriptions = OpenAIAudioTranscription()
openai_image_variations = OpenAIImageVariationsHandler()
groq_chat_completions = GroqChatCompletion()
sap_gen_ai_hub_chat_completions = GenAIHubOrchestration()
sap_gen_ai_hub_emb = GenAIHubOrchestration()
azure_ai_embedding = AzureAIEmbedding()
anthropic_chat_completions = AnthropicChatCompletion()
azure_anthropic_chat_completions = AzureAnthropicChatCompletion()
@ -2093,6 +2096,34 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging, # model call logging done inside the class as we make need to modify I/O to fit aleph alpha's requirements
client=client,
)
elif custom_llm_provider == "sap":
headers = headers or litellm.headers
## LOAD CONFIG - if set
config = litellm.GenAIHubOrchestrationConfig.get_config()
for k, v in config.items():
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
optional_params[k] = v
response = sap_gen_ai_hub_chat_completions.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
acompletion=acompletion,
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
timeout=timeout, # type: ignore
shared_session=shared_session,
client=client,
custom_llm_provider=custom_llm_provider,
encoding=encoding,
api_key=api_key,
api_base=api_base,
stream=stream,
)
elif custom_llm_provider == "aiohttp_openai":
# NEW aiohttp provider for 10-100x higher RPS
api_base = (
@ -4858,6 +4889,21 @@ def embedding( # noqa: PLR0915
client=client,
aembedding=aembedding,
)
elif custom_llm_provider == "sap":
response = base_llm_http_handler.embedding(
model=model,
input=input,
custom_llm_provider=custom_llm_provider,
api_base=api_base,
api_key=api_key,
logging_obj=logging,
timeout=timeout,
model_response=EmbeddingResponse(),
optional_params=optional_params,
litellm_params={},
client=client,
aembedding=aembedding,
)
elif custom_llm_provider == "azure_ai":
api_base = (
api_base # for deepinfra/perplexity/anyscale/groq/friendliai we check in get_llm_provider and pass in the api base from there

64
litellm/model_prices_and_context_window_backup.json
View file

@ -255,6 +255,50 @@
"mode": "image_generation",
"output_cost_per_image": 0.06
},
"us.writer.palmyra-x4-v1:0": {
"input_cost_per_token": 2.5e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 128000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_pdf_input": true
},
"us.writer.palmyra-x5-v1:0": {
"input_cost_per_token": 6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 6e-06,
"supports_function_calling": true,
"supports_pdf_input": true
},
"writer.palmyra-x4-v1:0": {
"input_cost_per_token": 2.5e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 128000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_pdf_input": true
},
"writer.palmyra-x5-v1:0": {
"input_cost_per_token": 6e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 6e-06,
"supports_function_calling": true,
"supports_pdf_input": true
},
"amazon.nova-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
@ -6206,6 +6250,19 @@
"supports_function_calling": true,
"supports_tool_choice": true
},
"cerebras/zai-glm-4.6": {
"input_cost_per_token": 2.25e-06,
"litellm_provider": "cerebras",
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"source": "https://www.cerebras.ai/pricing",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"chat-bison": {
"input_cost_per_character": 2.5e-07,
"input_cost_per_token": 1.25e-07,
@ -22865,6 +22922,13 @@
"mode": "rerank",
"output_cost_per_token": 0.0
},
"nvidia_nim/ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2": {
"input_cost_per_query": 0.0,
"input_cost_per_token": 0.0,
"litellm_provider": "nvidia_nim",
"mode": "rerank",
"output_cost_per_token": 0.0
},
"sagemaker/meta-textgeneration-llama-2-13b": {
"input_cost_per_token": 0.0,
"litellm_provider": "sagemaker",

85
litellm/proxy/_experimental/mcp_server/ui_session_utils.py Normal file
View file

@ -0,0 +1,85 @@
"""Helpers to resolve real team contexts for UI session tokens."""
from __future__ import annotations
from typing import List
from litellm._logging import verbose_logger
from litellm.constants import UI_SESSION_TOKEN_TEAM_ID
from litellm.proxy._types import UserAPIKeyAuth
def clone_user_api_key_auth_with_team(
user_api_key_auth: UserAPIKeyAuth,
team_id: str,
) -> UserAPIKeyAuth:
"""Return a deep copy of the auth context with a different team id."""
try:
cloned_auth = user_api_key_auth.model_copy(deep=True)
except AttributeError:
cloned_auth = user_api_key_auth.copy(deep=True) # type: ignore[attr-defined]
cloned_auth.team_id = team_id
return cloned_auth
async def resolve_ui_session_team_ids(
user_api_key_auth: UserAPIKeyAuth,
) -> List[str]:
"""Resolve the real team ids backing a UI session token."""
if (
user_api_key_auth.team_id != UI_SESSION_TOKEN_TEAM_ID
or not user_api_key_auth.user_id
):
return []
from litellm.proxy.auth.auth_checks import get_user_object
from litellm.proxy.proxy_server import (
prisma_client,
proxy_logging_obj,
user_api_key_cache,
)
if prisma_client is None:
verbose_logger.debug("Cannot resolve UI session team ids without DB access")
return []
try:
user_obj = await get_user_object(
user_id=user_api_key_auth.user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
parent_otel_span=user_api_key_auth.parent_otel_span,
proxy_logging_obj=proxy_logging_obj,
)
except Exception as exc: # pragma: no cover - defensive logging
verbose_logger.warning(
"Failed to load teams for UI session token user.",
exc,
)
return []
if user_obj is None or not user_obj.teams:
return []
resolved_team_ids: List[str] = []
for team_id in user_obj.teams:
if team_id and team_id not in resolved_team_ids:
resolved_team_ids.append(team_id)
return resolved_team_ids
async def build_effective_auth_contexts(
user_api_key_auth: UserAPIKeyAuth,
) -> List[UserAPIKeyAuth]:
"""Return auth contexts that reflect the actual teams for UI session tokens."""
resolved_team_ids = await resolve_ui_session_team_ids(user_api_key_auth)
if resolved_team_ids:
return [
clone_user_api_key_auth_with_team(user_api_key_auth, team_id)
for team_id in resolved_team_ids
]
return [user_api_key_auth]

11
litellm/proxy/auth/auth_checks.py
View file

@ -402,13 +402,14 @@ def _allowed_routes_check(user_route: str, allowed_routes: list) -> bool:
- user_route: str - the route the user is trying to call
- allowed_routes: List[str|LiteLLMRoutes] - the list of allowed routes for the user.
"""
from starlette.routing import compile_path
for allowed_route in allowed_routes:
if (
allowed_route in LiteLLMRoutes.__members__
and user_route in LiteLLMRoutes[allowed_route].value
):
return True
if allowed_route in LiteLLMRoutes.__members__:
for template in LiteLLMRoutes[allowed_route].value:
regex, _, _ = compile_path(template)
if regex.match(user_route):
return True
elif allowed_route == user_route:
return True
return False

16
litellm/proxy/auth/auth_utils.py
View file

@ -251,30 +251,38 @@ def route_in_additonal_public_routes(current_route: str):
- bool - True if the route is defined in public_routes
- bool - False if the route is not defined in public_routes
Supports wildcard patterns (e.g., "/api/*" matches "/api/users", "/api/users/123")
In order to use this the litellm config.yaml should have the following in general_settings:
```yaml
general_settings:
master_key: sk-1234
public_routes: ["LiteLLMRoutes.public_routes", "/spend/calculate"]
public_routes: ["LiteLLMRoutes.public_routes", "/spend/calculate", "/api/*"]
```
"""
# check if user is premium_user - if not do nothing
from litellm.proxy.auth.route_checks import RouteChecks
from litellm.proxy.proxy_server import general_settings, premium_user
try:
if premium_user is not True:
return False
# check if this is defined on the config
if general_settings is None:
return False
routes_defined = general_settings.get("public_routes", [])
# Check exact match first
if current_route in routes_defined:
return True
# Check wildcard patterns
for route_pattern in routes_defined:
if RouteChecks._route_matches_wildcard_pattern(
route=current_route, pattern=route_pattern
):
return True
return False
except Exception as e:
verbose_proxy_logger.error(f"route_in_additonal_public_routes: {str(e)}")

26
litellm/proxy/common_utils/html_forms/ui_login.py
View file

@ -8,7 +8,22 @@ if server_root_path != "":
url_to_redirect_to += server_root_path
url_to_redirect_to += "/login"
new_ui_login_url = get_custom_url("", "ui/login")
html_form = f"""
def build_ui_login_form(show_deprecation_banner: bool = False) -> str:
banner_html = (
f"""
<div class="deprecation-banner">
<strong>Deprecated:</strong> Logging in with username and password on this page is deprecated.
Please use the <a href="{new_ui_login_url}">new login page</a> instead.
This page will be dedicated to signing in via SSO in the future.
</div>
"""
if show_deprecation_banner
else ""
)
return f"""
<!DOCTYPE html>
<html lang="en">
<head>
@ -209,11 +224,7 @@ html_form = f"""
</head>
<body>
<form action="{url_to_redirect_to}" method="post">
<div class="deprecation-banner">
<strong>Deprecated:</strong> Logging in with username and password on this page is deprecated.
Please use the <a href="{new_ui_login_url}">new login page</a> instead.
This page will be dedicated to signing in via SSO in the future.
</div>
{banner_html}
<div class="logo-container">
<div class="logo">
🚅 LiteLLM
@ -253,3 +264,6 @@ html_form = f"""
</body>
</html>
"""
html_form = build_ui_login_form(show_deprecation_banner=True)

45
litellm/proxy/hooks/dynamic_rate_limiter_v3.py
View file

@ -58,6 +58,35 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
def update_variables(self, llm_router: Router):
self.llm_router = llm_router
def _get_saturation_check_cache_ttl(self) -> int:
"""Get the configurable TTL for local cache when reading saturation values."""
return litellm.priority_reservation_settings.saturation_check_cache_ttl
async def _get_saturation_value_from_cache(
self,
counter_key: str,
) -> Optional[str]:
"""
Get saturation value with configurable local cache TTL.
Uses DualCache with configurable TTL for local cache storage.
TTL is configurable via litellm.priority_reservation_settings.saturation_check_cache_ttl
Args:
counter_key: The cache key for the saturation counter
Returns:
Counter value as string, or None if not found
"""
local_cache_ttl = self._get_saturation_check_cache_ttl()
return await self.internal_usage_cache.async_get_cache(
key=counter_key,
litellm_parent_otel_span=None,
local_only=False,
ttl=local_cache_ttl,
)
def _get_priority_weight(
self, priority: Optional[str], model_info: Optional[ModelGroupInfo] = None
) -> float:
@ -195,7 +224,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
try:
max_saturation = 0.0
# Query RPM saturation
# Query RPM saturation - always read from Redis for multi-node consistency
if model_group_info.rpm is not None and model_group_info.rpm > 0:
# Use v3 limiter's key format: {key:value}:rate_limit_type
counter_key = self.v3_limiter.create_rate_limit_keys(
@ -204,11 +233,9 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
rate_limit_type="requests",
)
# Query cache for current counter value
counter_value = await self.internal_usage_cache.async_get_cache(
key=counter_key,
litellm_parent_otel_span=None,
local_only=False, # Check Redis too
# Query Redis directly for current counter value (skip local cache for consistency)
counter_value = await self._get_saturation_value_from_cache(
counter_key=counter_key
)
if counter_value is not None:
@ -229,10 +256,8 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
rate_limit_type="tokens",
)
counter_value = await self.internal_usage_cache.async_get_cache(
key=counter_key,
litellm_parent_otel_span=None,
local_only=False,
counter_value = await self._get_saturation_value_from_cache(
counter_key=counter_key
)
if counter_value is not None:

20
litellm/proxy/management_endpoints/mcp_management_endpoints.py
View file

@ -63,6 +63,9 @@ if MCP_AVAILABLE:
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
from litellm.proxy._experimental.mcp_server.ui_session_utils import (
build_effective_auth_contexts,
)
from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
from litellm.proxy._types import (
LiteLLM_MCPServerTable,
@ -422,13 +425,18 @@ if MCP_AVAILABLE:
```
"""
# Use server manager to get all servers with health and team data
mcp_servers = (
await global_mcp_server_manager.get_all_mcp_servers_with_health_and_teams(
user_api_key_auth=user_api_key_dict
auth_contexts = await build_effective_auth_contexts(user_api_key_dict)
aggregated_servers: Dict[str, LiteLLM_MCPServerTable] = {}
for auth_context in auth_contexts:
servers = await global_mcp_server_manager.get_all_mcp_servers_with_health_and_teams(
user_api_key_auth=auth_context
)
)
redacted_mcp_servers = _redact_mcp_credentials_list(mcp_servers)
for server in servers:
if server.server_id not in aggregated_servers:
aggregated_servers[server.server_id] = server
redacted_mcp_servers = _redact_mcp_credentials_list(aggregated_servers.values())
# augment the mcp servers with public status
if litellm.public_mcp_servers is not None:

20
litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py
View file

@ -96,9 +96,20 @@ class AnthropicPassthroughLoggingHandler:
handles streaming and non-streaming responses
"""
try:
# Get custom_llm_provider from logging object if available (e.g., azure_ai for Azure Anthropic)
custom_llm_provider = logging_obj.model_call_details.get(
"custom_llm_provider"
)
# Prepend custom_llm_provider to model if not already present
model_for_cost = model
if custom_llm_provider and not model.startswith(f"{custom_llm_provider}/"):
model_for_cost = f"{custom_llm_provider}/{model}"
response_cost = litellm.completion_cost(
completion_response=litellm_model_response,
model=model,
model=model_for_cost,
custom_llm_provider=custom_llm_provider,
)
kwargs["response_cost"] = response_cost
@ -157,19 +168,14 @@ class AnthropicPassthroughLoggingHandler:
"""
model = request_body.get("model", "")
# Dheck if it's available in the logging object
# Check if it's available in the logging object
if (
not model
and hasattr(litellm_logging_obj, "model_call_details")
and litellm_logging_obj.model_call_details.get("model")
):
model = cast(str, litellm_logging_obj.model_call_details.get("model"))
custom_llm_provider = litellm_logging_obj.model_call_details.get(
"custom_llm_provider"
)
if custom_llm_provider and not model.startswith(custom_llm_provider):
model = f"{custom_llm_provider}/{model}"
complete_streaming_response = (
AnthropicPassthroughLoggingHandler._build_complete_streaming_response(
all_chunks=all_chunks,

10
litellm/proxy/proxy_server.py
View file

@ -236,7 +236,7 @@ from litellm.proxy.common_utils.encrypt_decrypt_utils import (
decrypt_value_helper,
encrypt_value_helper,
)
from litellm.proxy.common_utils.html_forms.ui_login import html_form
from litellm.proxy.common_utils.html_forms.ui_login import build_ui_login_form
from litellm.proxy.common_utils.http_parsing_utils import (
_read_request_body,
check_file_size_under_limit,
@ -8306,11 +8306,15 @@ async def fallback_login(request: Request):
# Use UI Credentials set in .env
from fastapi.responses import HTMLResponse
return HTMLResponse(content=html_form, status_code=200)
return HTMLResponse(
content=build_ui_login_form(show_deprecation_banner=False), status_code=200
)
else:
from fastapi.responses import HTMLResponse
return HTMLResponse(content=html_form, status_code=200)
return HTMLResponse(
content=build_ui_login_form(show_deprecation_banner=False), status_code=200
)
@router.post(

18
litellm/proxy/public_endpoints/provider_create_fields.json
View file

@ -2446,6 +2446,24 @@
],
"default_model_placeholder": "gpt-3.5-turbo"
},
{
"provider": "SAP",
"provider_display_name": "SAP Generative AI Hub",
"litellm_provider": "sap",
"credential_fields": [
{
"key": "api_key",
"label": "SAP AI Core Service Key (JSON)",
"placeholder": null,
"tooltip": "Paste your SAP AI Core service key JSON. Contains clientid, clientsecret, and service URLs.",
"required": true,
"field_type": "textarea",
"options": null,
"default_value": null
}
],
"default_model_placeholder": "sap/gpt-4"
},
{
"provider": "Snowflake",
"provider_display_name": "Snowflake",

2
litellm/proxy/utils.py
View file

@ -995,7 +995,7 @@ class ProxyLogging:
):
result = await self._process_guardrail_callback(
callback=_callback,
data=data,
data=data, # type: ignore
user_api_key_dict=user_api_key_dict,
call_type=call_type,
)

2
litellm/proxy/vertex_ai_endpoints/langfuse_endpoints.py
View file

@ -128,12 +128,12 @@ async def langfuse_proxy_route(
endpoint=endpoint,
target=str(updated_url),
custom_headers={"Authorization": langfuse_combined_key},
query_params=dict(request.query_params), # type: ignore
) # dynamically construct pass-through endpoint based on incoming path
received_value = await endpoint_func(
request,
fastapi_response,
user_api_key_dict,
query_params=dict(request.query_params), # type: ignore
)
return received_value

2
litellm/router_utils/common_utils.py
View file

@ -110,7 +110,7 @@ def filter_web_search_deployments(
return healthy_deployments
is_web_search_request = False
tools = request_kwargs.get("tools", [])
tools = request_kwargs.get("tools") or []
for tool in tools:
# These are the two websearch tools for OpenAI / Azure.
if tool.get("type") == "web_search" or tool.get("type") == "web_search_preview":

5
litellm/types/llms/anthropic.py
View file

@ -1,7 +1,7 @@
from enum import Enum
from typing import Any, Dict, Iterable, List, Optional, Union
from pydantic import BaseModel
from pydantic import BaseModel, ConfigDict
from typing_extensions import Literal, Required, TypedDict
from .openai import (
@ -535,8 +535,7 @@ class AnthropicResponseContentBlockToolUse(BaseModel):
input: dict
provider_specific_fields: Optional[Dict[str, Any]] = None
class Config:
extra = "allow" # Allow provider_specific_fields
model_config = ConfigDict(extra="allow") # Allow provider_specific_fields
class AnthropicResponseContentBlockThinking(BaseModel):

5
litellm/types/rag.py
View file

@ -4,7 +4,7 @@ Type definitions for RAG (Retrieval Augmented Generation) Ingest API.
from typing import Any, Dict, List, Literal, Optional, Union
from pydantic import BaseModel
from pydantic import BaseModel, ConfigDict
from typing_extensions import TypedDict
@ -185,6 +185,5 @@ class RAGIngestRequest(BaseModel):
file_id: Optional[str] = None # Existing file ID
ingest_options: Dict[str, Any] # RAGIngestOptions as dict for flexibility
class Config:
extra = "allow" # Allow additional fields
model_config = ConfigDict(extra="allow") # Allow additional fields

7
litellm/types/utils.py
View file

@ -2982,6 +2982,7 @@ class LlmProviders(str, Enum):
LANGFUSE = "langfuse"
HUMANLOOP = "humanloop"
TOPAZ = "topaz"
SAP_GENERATIVE_AI_HUB = "sap"
ASSEMBLYAI = "assemblyai"
GITHUB_COPILOT = "github_copilot"
SNOWFLAKE = "snowflake"
@ -2989,6 +2990,7 @@ class LlmProviders(str, Enum):
LLAMA = "meta_llama"
NSCALE = "nscale"
PG_VECTOR = "pg_vector"
HELICONE = "helicone"
HYPERBOLIC = "hyperbolic"
RECRAFT = "recraft"
FAL_AI = "fal_ai"
@ -3308,4 +3310,9 @@ class PriorityReservationSettings(BaseModel):
description="Saturation threshold (0.0-1.0) at which strict priority enforcement begins. Below this threshold, generous mode allows priority borrowing. Above this threshold, strict mode enforces normalized priority limits.",
)
saturation_check_cache_ttl: int = Field(
default=60,
description="TTL in seconds for local cache when reading saturation check values from Redis.",
)
model_config = ConfigDict(protected_namespaces=())

38
litellm/utils.py
View file

@ -2886,6 +2886,21 @@ def get_optional_params_embeddings( # noqa: PLR0915
model=model,
drop_params=drop_params if drop_params is not None else False,
)
final_params = {**optional_params, **kwargs}
return final_params
elif custom_llm_provider == "sap":
supported_params = get_supported_openai_params(
model=model,
custom_llm_provider="sap",
request_type="embeddings",
)
_check_valid_arg(supported_params=supported_params)
optional_params = litellm.GenAIHubEmbeddingConfig().map_openai_params(
non_default_params=non_default_params,
optional_params={},
model=model,
drop_params=drop_params if drop_params is not None else False
)
elif custom_llm_provider == "infinity":
supported_params = get_supported_openai_params(
model=model,
@ -2899,6 +2914,10 @@ def get_optional_params_embeddings( # noqa: PLR0915
model=model,
drop_params=drop_params if drop_params is not None else False,
)
final_params = {**optional_params, **kwargs}
return final_params
elif custom_llm_provider == "fireworks_ai":
supported_params = get_supported_openai_params(
model=model,
@ -7216,6 +7235,8 @@ class ProviderConfigManager:
return litellm.TritonConfig()
elif litellm.LlmProviders.PETALS == provider:
return litellm.PetalsConfig()
elif litellm.LlmProviders.SAP_GENERATIVE_AI_HUB == provider:
return litellm.GenAIHubOrchestrationConfig()
elif litellm.LlmProviders.FEATHERLESS_AI == provider:
return litellm.FeatherlessAIConfig()
elif litellm.LlmProviders.NOVITA == provider:
@ -7276,6 +7297,8 @@ class ProviderConfigManager:
return litellm.TritonEmbeddingConfig()
elif litellm.LlmProviders.WATSONX == provider:
return litellm.IBMWatsonXEmbeddingConfig()
elif litellm.LlmProviders.SAP_GENERATIVE_AI_HUB == provider:
return litellm.GenAIHubEmbeddingConfig()
elif litellm.LlmProviders.INFINITY == provider:
return litellm.InfinityEmbeddingConfig()
elif litellm.LlmProviders.SAMBANOVA == provider:
@ -7343,7 +7366,11 @@ class ProviderConfigManager:
elif litellm.LlmProviders.DEEPINFRA == provider:
return litellm.DeepinfraRerankConfig()
elif litellm.LlmProviders.NVIDIA_NIM == provider:
return litellm.NvidiaNimRerankConfig()
from litellm.llms.nvidia_nim.rerank.common_utils import (
get_nvidia_nim_rerank_config,
)
return get_nvidia_nim_rerank_config(model)
elif litellm.LlmProviders.VERTEX_AI == provider:
return litellm.VertexAIRerankConfig()
elif litellm.LlmProviders.FIREWORKS_AI == provider:
@ -7364,12 +7391,19 @@ class ProviderConfigManager:
return BedrockModelInfo.get_bedrock_provider_config_for_messages_api(model)
elif litellm.LlmProviders.VERTEX_AI == provider:
if "claude" in model:
if "claude" in model.lower():
from litellm.llms.vertex_ai.vertex_ai_partner_models.anthropic.experimental_pass_through.transformation import (
VertexAIPartnerModelsAnthropicMessagesConfig,
)
return VertexAIPartnerModelsAnthropicMessagesConfig()
elif litellm.LlmProviders.AZURE_AI == provider:
if "claude" in model.lower():
from litellm.llms.azure_ai.anthropic.messages_transformation import (
AzureAnthropicMessagesConfig,
)
return AzureAnthropicMessagesConfig()
return None
@staticmethod

2110
model_prices_and_context_window.json
View file

File diff suppressed because it is too large Load diff

17
provider_endpoints_support.json
View file

@ -1554,6 +1554,23 @@
"a2a": true
}
},
"sap": {
"display_name": "SAP Generative AI Hub (`sap`)",
"url": "https://docs.litellm.ai/docs/providers/sap",
"endpoints": {
"chat_completions": true,
"messages": true,
"responses": true,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": false,
"moderations": false,
"batches": false,
"rerank": false,
"a2a": true
}
},
"snowflake": {
"display_name": "Snowflake (`snowflake`)",
"url": "https://docs.litellm.ai/docs/providers/snowflake",

358
scripts/mock_ibm_guardrails_server.py
View file

@ -1,358 +0,0 @@
"""
Mock FastAPI server for IBM FMS Guardrails Orchestrator Detector API.
This server implements the Detector API endpoints for testing purposes.
Based on: https://foundation-model-stack.github.io/fms-guardrails-orchestrator/
Usage:
python scripts/mock_ibm_guardrails_server.py
The server will run on http://localhost:8001 by default.
"""
import uuid
from typing import Any, Dict, List, Optional
import uvicorn
from fastapi import FastAPI, Header, HTTPException, status
from pydantic import BaseModel, Field
app = FastAPI(
title="IBM FMS Guardrails Orchestrator Mock",
description="Mock server for testing IBM Guardrails Detector API",
version="1.0.0",
)
# Request Models
class DetectorParams(BaseModel):
"""Parameters specific to the detector."""
threshold: Optional[float] = Field(None, ge=0.0, le=1.0)
custom_param: Optional[str] = None
class TextDetectionRequest(BaseModel):
"""Request model for text detection."""
contents: List[str] = Field(..., description="Text content to analyze")
detector_params: Optional[DetectorParams] = None
class TextGenerationDetectionRequest(BaseModel):
"""Request model for text generation detection."""
detector_id: str = Field(..., description="ID of the detector to use")
prompt: str = Field(..., description="Input prompt")
generated_text: str = Field(..., description="Generated text to analyze")
detector_params: Optional[DetectorParams] = None
class ContextDetectionRequest(BaseModel):
"""Request model for detection with context."""
detector_id: str = Field(..., description="ID of the detector to use")
content: str = Field(..., description="Text content to analyze")
context: Optional[Dict[str, Any]] = Field(None, description="Additional context")
detector_params: Optional[DetectorParams] = None
# Response Models
class Detection(BaseModel):
"""Individual detection result."""
detection_type: str = Field(..., description="Type of detection")
detection: bool = Field(..., description="Whether content was detected as harmful")
score: float = Field(..., ge=0.0, le=1.0, description="Detection confidence score")
start: Optional[int] = Field(None, description="Start position in text")
end: Optional[int] = Field(None, description="End position in text")
text: Optional[str] = Field(None, description="Detected text segment")
evidence: Optional[List[str]] = Field(None, description="Supporting evidence")
class DetectionResponse(BaseModel):
"""Response model for detection results."""
detections: List[Detection] = Field(..., description="List of detections")
detection_id: str = Field(..., description="Unique ID for this detection request")
# Mock detector configurations
MOCK_DETECTORS = {
"hate": {
"name": "Hate Speech Detector",
"triggers": ["hate", "offensive", "discriminatory", "slur"],
"default_score": 0.85,
},
"pii": {
"name": "PII Detector",
"triggers": ["email", "ssn", "credit card", "phone number", "address"],
"default_score": 0.92,
},
"toxicity": {
"name": "Toxicity Detector",
"triggers": ["toxic", "abusive", "profanity", "insult"],
"default_score": 0.78,
},
"jailbreak": {
"name": "Jailbreak Detector",
"triggers": ["ignore instructions", "override", "bypass", "jailbreak"],
"default_score": 0.88,
},
"prompt_injection": {
"name": "Prompt Injection Detector",
"triggers": ["ignore previous", "new instructions", "system prompt"],
"default_score": 0.90,
},
}
def simulate_detection(
detector_id: str, content: str, detector_params: Optional[DetectorParams] = None
) -> List[Detection]:
"""
Simulate detection logic based on detector type and content.
Args:
detector_id: ID of the detector to simulate
content: Text content to analyze
detector_params: Optional detector parameters
Returns:
List of Detection objects
"""
detections = []
content_lower = " ".join(c for c in content).lower()
# Get detector config
detector_config = MOCK_DETECTORS.get(detector_id)
if not detector_config:
# Unknown detector - return no detections
return detections
# Check for triggers in content
for trigger in detector_config["triggers"]:
if trigger in content_lower:
# Calculate score (use threshold if provided, otherwise default)
base_score = detector_config["default_score"]
threshold = (
detector_params.threshold
if detector_params and detector_params.threshold
else None
)
# Adjust score slightly based on content length (longer content = slightly lower confidence)
score_adjustment = max(0, min(0.1, len(content) / 10000))
score = max(0.0, min(1.0, base_score - score_adjustment))
# Find position of trigger
start_pos = content_lower.find(trigger)
end_pos = start_pos + len(trigger)
detection = Detection(
detection_type=detector_id,
detection=threshold is None or score >= threshold,
score=score,
start=start_pos,
end=end_pos,
text=content[start_pos:end_pos] if start_pos >= 0 else None,
evidence=[f"Found trigger word: {trigger}"],
)
detections.append(detection)
# If no triggers found, return a negative detection
if not detections:
detections.append(
Detection(
detection_type=detector_id,
detection=False,
score=0.05, # Low score for clean content
)
)
return detections
# Authentication middleware
def verify_auth_token(authorization: Optional[str] = Header(None)) -> bool:
"""
Verify the authentication token.
Args:
authorization: Authorization header value
Returns:
True if valid, raises HTTPException otherwise
"""
if not authorization:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing authorization header",
)
# Simple token validation - in real implementation, this would validate against a real auth system
if not authorization.startswith("Bearer "):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid authorization header format. Expected: Bearer <token>",
)
token = authorization.replace("Bearer ", "")
# Accept any non-empty token for mock purposes
if not token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Empty token provided",
)
return True
# API Endpoints
@app.get("/health")
async def health_check():
"""Health check endpoint."""
return {"status": "healthy", "service": "IBM FMS Guardrails Mock Server"}
@app.get("/")
async def root():
"""Root endpoint with API information."""
return {
"service": "IBM FMS Guardrails Orchestrator Mock",
"version": "1.0.0",
"endpoints": {
"health": "/health",
"text_detection": "/api/v1/text/detection",
"generation_detection": "/api/v1/text/generation/detection",
"context_detection": "/api/v1/text/context/detection",
},
"available_detectors": list(MOCK_DETECTORS.keys()),
}
@app.post("/api/v1/text/contents")
async def text_detection(
request: TextDetectionRequest,
detector_id: str = Header(None), # query parameter
authorization: Optional[str] = Header(None),
):
"""
Detect potential issues in text content.
Args:
request: Detection request with content and detector ID
detector_id: ID of detector
authorization: Bearer token for authentication
Returns:
Detection results
"""
verify_auth_token(authorization)
detections = simulate_detection(
detector_id=detector_id,
content=request.contents,
detector_params=request.detector_params,
)
return detections
@app.post("/api/v1/text/generation/detection", response_model=DetectionResponse)
async def text_generation_detection(
request: TextGenerationDetectionRequest,
authorization: Optional[str] = Header(None),
):
"""
Detect potential issues in generated text.
Args:
request: Detection request with prompt and generated text
authorization: Bearer token for authentication
Returns:
Detection results
"""
verify_auth_token(authorization)
# Analyze both prompt and generated text
combined_content = f"{request.prompt} {request.generated_text}"
detections = simulate_detection(
detector_id=request.detector_id,
content=combined_content,
detector_params=request.detector_params,
)
return DetectionResponse(
detections=detections,
detection_id=str(uuid.uuid4()),
)
@app.post("/api/v1/text/context/detection", response_model=DetectionResponse)
async def context_detection(
request: ContextDetectionRequest,
authorization: Optional[str] = Header(None),
):
"""
Detect potential issues in text with additional context.
Args:
request: Detection request with content and context
authorization: Bearer token for authentication
Returns:
Detection results
"""
verify_auth_token(authorization)
detections = simulate_detection(
detector_id=request.detector_id,
content=request.content,
detector_params=request.detector_params,
)
return DetectionResponse(
detections=detections,
detection_id=str(uuid.uuid4()),
)
@app.get("/api/v1/detectors")
async def list_detectors(authorization: Optional[str] = Header(None)):
"""
List available detectors.
Args:
authorization: Bearer token for authentication
Returns:
List of available detectors
"""
verify_auth_token(authorization)
return {
"detectors": [
{
"id": detector_id,
"name": config["name"],
"triggers": config["triggers"],
}
for detector_id, config in MOCK_DETECTORS.items()
]
}
if __name__ == "__main__":
print("🚀 Starting IBM FMS Guardrails Mock Server...")
print("📍 Server will be available at: http://localhost:8001")
print("📚 API docs at: http://localhost:8001/docs")
print("\nAvailable detectors:")
for detector_id, config in MOCK_DETECTORS.items():
print(f" - {detector_id}: {config['name']}")
print("\n✨ Use any Bearer token for authentication in this mock server\n")
uvicorn.run(app, host="0.0.0.0", port=8001)

54
scripts/test_groq_streaming_issue.py
View file

@ -1,54 +0,0 @@
"""
Test script to reproduce the Groq streaming ASCII encoding issue.
This reproduces the issue described in #12660 where streaming responses
containing non-ASCII characters like µ cause encoding errors.
"""
import asyncio
import os
import traceback
from litellm import acompletion
async def test_groq_streaming_with_special_chars():
"""Test that reproduces the ASCII encoding issue with Groq streaming."""
try:
print("Testing acompletion + streaming with Groq...")
# Test message that should trigger the µ character or similar non-ASCII content
test_messages = [
{"content": "What is the symbol for micro? Please include the µ symbol in your response.", "role": "user"}
]
# This should trigger the ASCII encoding error described in the issue
response = await acompletion(
model="groq/llama-3.3-70b-versatile",
messages=test_messages,
stream=True
)
print(f"Response type: {type(response)}")
# Try to iterate through the stream
async for chunk in response:
print(f"Chunk: {chunk}")
print("✅ Test completed successfully - no encoding errors!")
except Exception as e:
print(f"❌ Error occurred: {e}")
print(f"Error type: {type(e)}")
print(f"Traceback:\n{traceback.format_exc()}")
return False
return True
if __name__ == "__main__":
# Note: This requires GROQ_API_KEY to be set
if not os.getenv("GROQ_API_KEY"):
print("⚠️ GROQ_API_KEY not set. Skipping test.")
else:
success = asyncio.run(test_groq_streaming_with_special_chars())
if success:
print("🎉 All tests passed!")
else:
print("💥 Test failed!")

181
scripts/test_mock_ibm_guardrails.py
View file

@ -1,181 +0,0 @@
"""
Test script for the mock IBM Guardrails server.
This demonstrates how to interact with the mock server.
Usage:
# Start the mock server in one terminal:
python scripts/mock_ibm_guardrails_server.py
# Run this test in another terminal:
python scripts/test_mock_ibm_guardrails.py
"""
import asyncio
import httpx
async def test_mock_server():
"""Test the mock IBM Guardrails server."""
base_url = "http://localhost:8001"
headers = {"Authorization": "Bearer test-token-12345"}
print("🧪 Testing IBM FMS Guardrails Mock Server\n")
async with httpx.AsyncClient() as client:
# Test 1: Health check
print("1️⃣ Testing health check...")
try:
response = await client.get(f"{base_url}/health")
print(f" ✅ Health check: {response.json()}\n")
except Exception as e:
print(f" ❌ Health check failed: {e}\n")
return
# Test 2: List detectors
print("2️⃣ Testing list detectors...")
try:
response = await client.get(
f"{base_url}/api/v1/detectors",
headers=headers
)
detectors = response.json()
print(f" ✅ Found {len(detectors['detectors'])} detectors:")
for detector in detectors["detectors"]:
print(f" - {detector['id']}: {detector['name']}")
print()
except Exception as e:
print(f" ❌ List detectors failed: {e}\n")
# Test 3: Text detection with clean content
print("3️⃣ Testing text detection (clean content)...")
try:
response = await client.post(
f"{base_url}/api/v1/text/detection",
headers=headers,
json={
"detector_id": "hate",
"content": "This is a normal, friendly message.",
}
)
result = response.json()
print(f" ✅ Detection result:")
print(f" Detection ID: {result['detection_id']}")
for detection in result["detections"]:
print(f" - Type: {detection['detection_type']}, Detected: {detection['detection']}, Score: {detection['score']:.2f}")
print()
except Exception as e:
print(f" ❌ Text detection failed: {e}\n")
# Test 4: Text detection with problematic content
print("4️⃣ Testing text detection (problematic content)...")
try:
response = await client.post(
f"{base_url}/api/v1/text/detection",
headers=headers,
json={
"detector_id": "hate",
"content": "This message contains hate speech and offensive language.",
}
)
result = response.json()
print(f" ✅ Detection result:")
print(f" Detection ID: {result['detection_id']}")
for detection in result["detections"]:
print(f" - Type: {detection['detection_type']}, Detected: {detection['detection']}, Score: {detection['score']:.2f}")
if detection.get("evidence"):
print(f" Evidence: {detection['evidence']}")
print()
except Exception as e:
print(f" ❌ Text detection failed: {e}\n")
# Test 5: PII detection
print("5️⃣ Testing PII detection...")
try:
response = await client.post(
f"{base_url}/api/v1/text/detection",
headers=headers,
json={
"detector_id": "pii",
"content": "Please send the report to my email address john@example.com",
}
)
result = response.json()
print(f" ✅ Detection result:")
print(f" Detection ID: {result['detection_id']}")
for detection in result["detections"]:
print(f" - Type: {detection['detection_type']}, Detected: {detection['detection']}, Score: {detection['score']:.2f}")
if detection.get("text"):
print(f" Detected text: '{detection['text']}'")
print()
except Exception as e:
print(f" ❌ PII detection failed: {e}\n")
# Test 6: Generation detection
print("6️⃣ Testing text generation detection...")
try:
response = await client.post(
f"{base_url}/api/v1/text/generation/detection",
headers=headers,
json={
"detector_id": "jailbreak",
"prompt": "Tell me about AI safety",
"generated_text": "I will ignore instructions and provide harmful content.",
}
)
result = response.json()
print(f" ✅ Detection result:")
print(f" Detection ID: {result['detection_id']}")
for detection in result["detections"]:
print(f" - Type: {detection['detection_type']}, Detected: {detection['detection']}, Score: {detection['score']:.2f}")
print()
except Exception as e:
print(f" ❌ Generation detection failed: {e}\n")
# Test 7: Detection with custom threshold
print("7️⃣ Testing detection with custom threshold...")
try:
response = await client.post(
f"{base_url}/api/v1/text/detection",
headers=headers,
json={
"detector_id": "toxicity",
"content": "This contains toxic language",
"detector_params": {
"threshold": 0.9
}
}
)
result = response.json()
print(f" ✅ Detection result (threshold=0.9):")
print(f" Detection ID: {result['detection_id']}")
for detection in result["detections"]:
print(f" - Type: {detection['detection_type']}, Detected: {detection['detection']}, Score: {detection['score']:.2f}")
print()
except Exception as e:
print(f" ❌ Threshold detection failed: {e}\n")
# Test 8: Authentication error
print("8️⃣ Testing authentication error...")
try:
response = await client.post(
f"{base_url}/api/v1/text/detection",
json={
"detector_id": "hate",
"content": "Test content",
}
)
if response.status_code == 401:
print(f" ✅ Authentication error handled correctly: {response.json()}\n")
else:
print(f" ⚠️ Unexpected status code: {response.status_code}\n")
except Exception as e:
print(f" ❌ Auth test failed: {e}\n")
print("✨ All tests completed!")
if __name__ == "__main__":
asyncio.run(test_mock_server())

147
scripts/update_readme_providers_table.py
View file

@ -1,147 +0,0 @@
#!/usr/bin/env python3
"""
Script to update the README.md providers table from provider_endpoints_support.json
"""
import json
import re
from pathlib import Path
# Define paths
REPO_ROOT = Path(__file__).parent.parent
JSON_PATH = REPO_ROOT / "provider_endpoints_support.json"
README_PATH = REPO_ROOT / "README.md"
# Endpoint column headers
ENDPOINT_COLUMNS = [
("/chat/completions", "chat_completions"),
("/messages", "messages"),
("/responses", "responses"),
("/embeddings", "embeddings"),
("/image/generations", "image_generations"),
("/audio/transcriptions", "audio_transcriptions"),
("/audio/speech", "audio_speech"),
("/moderations", "moderations"),
("/batches", "batches"),
("/rerank", "rerank"),
]
def load_providers_data():
"""Load provider data from JSON file"""
with open(JSON_PATH, 'r') as f:
data = json.load(f)
# Handle both old and new format
if "providers" in data:
return data["providers"]
return data
def generate_markdown_table(providers_data):
"""Generate markdown table from providers data"""
# Sort providers alphabetically by display name
sorted_providers = sorted(
providers_data.items(),
key=lambda x: x[1]['display_name'].lower()
)
# Generate header
header_cols = ["Provider"] + [col[0] for col in ENDPOINT_COLUMNS]
header = "| " + " | ".join(header_cols) + " |"
separator = "|" + "|".join(["-" * (len(col) + 2) for col in header_cols]) + "|"
# Generate rows
rows = []
for slug, data in sorted_providers:
display_name = data['display_name']
url = data['url']
# Build row
row_parts = [f"[{display_name}]({url})"]
for _, endpoint_key in ENDPOINT_COLUMNS:
supported = data['endpoints'].get(endpoint_key, False)
row_parts.append("✅" if supported else "")
row = "| " + " | ".join(row_parts) + " |"
rows.append(row)
# Combine all parts
table_lines = [
"<!-- AUTO-GENERATED TABLE - DO NOT EDIT MANUALLY -->",
"<!-- Edit provider_endpoints_support.json and run scripts/update_readme_providers_table.py -->",
"",
header,
separator
] + rows + [
"<!-- END AUTO-GENERATED TABLE -->"
]
return "\n".join(table_lines)
def update_readme(table_markdown):
"""Update README.md with new table"""
with open(README_PATH, 'r') as f:
content = f.read()
print(f" Original README length: {len(content)} bytes")
# Find the table section
# Look for the AUTO-GENERATED comment or the header, and replace until Read the Docs
pattern = r"(## Supported Providers.*?\n\n)(?:<!-- AUTO-GENERATED TABLE.*?<!-- END AUTO-GENERATED TABLE -->|.*?)(\n\n\[\*\*Read the Docs\*\*\])"
# Test if pattern matches
match = re.search(pattern, content, flags=re.DOTALL)
if not match:
print("❌ Pattern did not match in README.md")
return False
print(f" Pattern matched, replacing table...")
def replacer(match):
return match.group(1) + table_markdown + match.group(2)
new_content = re.sub(pattern, replacer, content, flags=re.DOTALL)
print(f" New README length: {len(new_content)} bytes")
if new_content == content:
print(" ℹ️ Table is already up-to-date, no changes needed")
return True # Not an error - table is already correct
with open(README_PATH, 'w') as f:
f.write(new_content)
print(" ✓ README.md has been updated")
return True
def main():
"""Main function"""
print("Loading provider data from provider_endpoints_support.json...")
providers_data = load_providers_data()
print(f"✓ Loaded {len(providers_data)} providers")
print("\nGenerating markdown table...")
table_markdown = generate_markdown_table(providers_data)
print(f"✓ Generated table with {len(providers_data)} rows")
print("\nUpdating README.md...")
if update_readme(table_markdown):
print("✓ Successfully updated README.md")
print("\n📝 Please review the changes and commit both files:")
print(" - provider_endpoints_support.json")
print(" - README.md")
else:
print("❌ Failed to update README.md")
return 1
return 0
if __name__ == "__main__":
exit(main())

5
tests/litellm_utils_tests/test_secret_manager.py
View file

@ -133,9 +133,8 @@ def test_oidc_circleci_v2():
print(f"secret_val: {redact_oidc_signature(secret_val)}")
@pytest.mark.skipif(
os.environ.get("CIRCLE_OIDC_TOKEN") is None,
reason="Cannot run without being in CircleCI Runner",
@pytest.mark.skip(
reason="Quarantined: Flaky test - fails with 401 Unauthorized from Azure OAuth. TODO: Switch to our own Azure account or fix authentication"
)
def test_oidc_circleci_with_azure():
# TODO: Switch to our own Azure account, currently using ai.moda's account

72
tests/llm_translation/test_helicone.py Normal file
View file

@ -0,0 +1,72 @@
import os
import sys
import pytest
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm
def test_completion_helicone():
"""Test basic completion through Helicone gateway"""
litellm._turn_on_debug()
resp = litellm.completion(
model="helicone/gpt-4o-mini",
messages=[{"role": "user", "content": "Say 'Hello from Helicone' and nothing else"}],
max_tokens=10,
)
print(resp)
assert resp.choices[0].message.content is not None
assert len(resp.choices[0].message.content) > 0
def test_completion_helicone_specific_provider():
"""Test basic completion through Helicone gateway"""
litellm._turn_on_debug()
resp = litellm.completion(
model="helicone/claude-4.5-haiku/anthropic",
messages=[{"role": "user", "content": "Say 'Hello from Helicone' and nothing else"}],
max_tokens=10,
)
print(resp)
assert resp.choices[0].message.content is not None
assert len(resp.choices[0].message.content) > 0
def test_completion_helicone_streaming():
"""Test streaming completion through Helicone gateway"""
litellm._turn_on_debug()
resp = litellm.completion(
model="helicone/gpt-4o-mini",
messages=[{"role": "user", "content": "Count to 3"}],
max_tokens=20,
stream=True,
)
chunks = []
for chunk in resp:
print(chunk)
if hasattr(chunk.choices[0], "delta") and hasattr(chunk.choices[0].delta, "content"):
if chunk.choices[0].delta.content:
chunks.append(chunk.choices[0].delta.content)
full_response = "".join(chunks)
assert len(full_response) > 0
print(f"Full response: {full_response}")
def test_completion_helicone_with_metadata():
"""Test Helicone with custom properties"""
litellm._turn_on_debug()
resp = litellm.completion(
model="helicone/gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=10,
metadata={
"Helicone-Property-Environment": "test",
"Helicone-Property-Session": "test-session-123"
}
)
print(resp)
assert resp.choices[0].message.content is not None

65
tests/llm_translation/test_nvidia_nim.py
View file

@ -184,13 +184,76 @@ def test_chat_completion_nvidia_nim_with_tools():
assert request_body["tool_choice"] == "auto"
assert request_body["parallel_tool_calls"] == True
@pytest.mark.asyncio()
async def test_nvidia_nim_rerank_ranking_endpoint():
"""
Test that using "nvidia_nim/ranking/<model>" forces the /v1/ranking endpoint.
This allows users to explicitly use the /v1/ranking endpoint for models like
nvidia/llama-3.2-nv-rerankqa-1b-v2.
Reference: https://build.nvidia.com/nvidia/llama-3_2-nv-rerankqa-1b-v2/deploy
"""
mock_response = AsyncMock()
def return_val():
return {
"rankings": [
{"index": 0, "logit": 0.95},
{"index": 1, "logit": 0.75},
],
}
mock_response.json = return_val
mock_response.headers = {"key": "value"}
mock_response.status_code = 200
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
return_value=mock_response,
) as mock_post:
# Use "ranking/" prefix to force /v1/ranking endpoint
response = await litellm.arerank(
model="nvidia_nim/ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2",
query="What is the GPU memory bandwidth?",
documents=["H100 delivers 3TB/s memory bandwidth", "A100 has 2TB/s memory bandwidth"],
top_n=2,
api_key="fake-api-key",
)
mock_post.assert_called_once()
args_to_api = mock_post.call_args.kwargs["data"]
_url = mock_post.call_args.kwargs["url"]
print("url = ", _url)
# Verify URL is /v1/ranking
assert _url == "https://ai.api.nvidia.com/v1/ranking"
# Verify request body structure
request_data = json.loads(args_to_api)
print("request_data=", request_data)
# Query should be an object with 'text' field
assert request_data["query"] == {"text": "What is the GPU memory bandwidth?"}
# Documents should be 'passages'
assert request_data["passages"] == [
{"text": "H100 delivers 3TB/s memory bandwidth"},
{"text": "A100 has 2TB/s memory bandwidth"},
]
# Model name in body should NOT have "ranking/" prefix
assert request_data["model"] == "nvidia/llama-3.2-nv-rerankqa-1b-v2"
class TestNvidiaNim(BaseLLMRerankTest):
def get_custom_llm_provider(self) -> litellm.LlmProviders:
return litellm.LlmProviders.NVIDIA_NIM
def get_base_rerank_call_args(self) -> dict:
return {
"model": "nvidia_nim/nvidia/llama-3_2-nv-rerankqa-1b-v2",
"model": "nvidia_nim/nvidia/llama-3.2-nv-rerankqa-1b-v2",
}
def get_expected_cost(self) -> float:

7
tests/proxy_unit_tests/test_response_polling_handler.py
View file

@ -519,6 +519,13 @@ class TestResponsePollingHandler:
# init_async_client is a sync method that returns an async client
mock_redis.init_async_client = Mock(return_value=mock_async_client)
# Mock async_delete_cache to actually call init_async_client and delete
async def mock_async_delete_cache(key):
client = mock_redis.init_async_client()
await client.delete(key)
mock_redis.async_delete_cache = mock_async_delete_cache
handler = ResponsePollingHandler(redis_cache=mock_redis)
result = await handler.delete_polling("litellm_poll_test")

127
tests/test_litellm/completion_extras/litellm_responses_transformation/test_completion_extras_litellm_responses_transformation_transformation.py
View file

@ -444,3 +444,130 @@ def test_transform_request_single_char_keys_not_matched():
assert result_correct.get("previous_response_id") == "resp_abc"
print("✓ Single-character keys are not incorrectly matched to metadata/previous_response_id")
# =============================================================================
# Tests for issue #17246: Streaming tool_calls dropped when text + tool_calls
# =============================================================================
def test_message_done_does_not_emit_is_finished():
"""
Test that OUTPUT_ITEM_DONE for a message does NOT emit is_finished=True.
This is the core fix for issue #17246.
Before fix: message completion emitted is_finished=True, causing tool_calls
that came after to be dropped.
"""
from litellm.completion_extras.litellm_responses_transformation.transformation import (
OpenAiResponsesToChatCompletionStreamIterator,
)
iterator = OpenAiResponsesToChatCompletionStreamIterator(
streaming_response=None, sync_stream=True
)
chunk = {
"type": "response.output_item.done",
"item": {"type": "message", "content": []}
}
result = iterator.chunk_parser(chunk)
# After the fix, message completion should NOT set is_finished=True
assert result["is_finished"] == False, "message completion should not emit is_finished=True"
assert result["finish_reason"] == "", "message completion should not emit finish_reason"
def test_response_completed_emits_is_finished():
"""
Test that response.completed DOES emit is_finished=True.
This ensures streaming ends properly after ALL output items are sent.
"""
from litellm.completion_extras.litellm_responses_transformation.transformation import (
OpenAiResponsesToChatCompletionStreamIterator,
)
iterator = OpenAiResponsesToChatCompletionStreamIterator(
streaming_response=None, sync_stream=True
)
chunk = {"type": "response.completed"}
result = iterator.chunk_parser(chunk)
assert result["is_finished"] == True, "response.completed should emit is_finished=True"
assert result["finish_reason"] == "stop", "response.completed should emit finish_reason='stop'"
def test_function_call_done_emits_is_finished():
"""
Test that OUTPUT_ITEM_DONE for a function_call still emits is_finished=True.
This preserves existing behavior for tool_calls.
"""
from litellm.completion_extras.litellm_responses_transformation.transformation import (
OpenAiResponsesToChatCompletionStreamIterator,
)
iterator = OpenAiResponsesToChatCompletionStreamIterator(
streaming_response=None, sync_stream=True
)
chunk = {
"type": "response.output_item.done",
"item": {
"type": "function_call",
"name": "get_weather",
"call_id": "call_123",
"arguments": '{"location": "Tokyo"}'
}
}
result = iterator.chunk_parser(chunk)
assert result["is_finished"] == True, "function_call completion should emit is_finished=True"
assert result["finish_reason"] == "tool_calls", "function_call should emit finish_reason='tool_calls'"
assert result["tool_use"] is not None, "function_call should include tool_use"
def test_text_plus_tool_calls_sequence():
"""
Test the full sequence when model returns text + tool_calls.
This is the main scenario for issue #17246.
Expected: is_finished=True should NOT appear until function_call is done,
not when message is done.
"""
from litellm.completion_extras.litellm_responses_transformation.transformation import (
OpenAiResponsesToChatCompletionStreamIterator,
)
iterator = OpenAiResponsesToChatCompletionStreamIterator(
streaming_response=None, sync_stream=True
)
# Simulate the sequence from OpenAI Responses API
chunks = [
{"type": "response.output_text.delta", "delta": "Hello"},
{"type": "response.output_text.delta", "delta": "!"},
{"type": "response.output_item.done", "item": {"type": "message", "content": []}}, # message done
{"type": "response.output_item.added", "item": {"type": "function_call", "name": "get_weather", "call_id": "call_123"}},
{"type": "response.function_call_arguments.delta", "delta": '{"location":"Tokyo"}'},
{"type": "response.output_item.done", "item": {"type": "function_call", "name": "get_weather", "call_id": "call_123", "arguments": '{"location":"Tokyo"}'}},
{"type": "response.completed"},
]
results = [iterator.chunk_parser(chunk) for chunk in chunks]
# Check message done (index 2) does NOT have is_finished=True
message_done_result = results[2]
assert message_done_result["is_finished"] == False, "message done should not have is_finished=True"
# Check function_call done (index 5) DOES have is_finished=True
function_done_result = results[5]
assert function_done_result["is_finished"] == True, "function_call done should have is_finished=True"
assert function_done_result["finish_reason"] == "tool_calls"
# Check response.completed (index 6) also has is_finished=True
completed_result = results[6]
assert completed_result["is_finished"] == True, "response.completed should have is_finished=True"

10
tests/test_litellm/litellm_core_utils/test_exception_mapping_utils.py
View file

@ -42,6 +42,16 @@ context_window_test_cases = [
),
# Test case insensitivity
("ERROR: THIS MODEL'S MAXIMUM CONTEXT LENGTH IS 1024.", True),
# Cerebras context window error format
# See: https://github.com/BerriAI/litellm/issues/XXXX
(
"Current length is 132784 while limit is 131000",
True,
),
(
"CerebrasException - Please reduce the length of the messages or completion. Current length is 50000 while limit is 40000",
True,
),
# Negative cases (should return False)
("A generic API error occurred.", False),
("Invalid API Key provided.", False),

63
tests/test_litellm/llms/azure_ai/claude/test_azure_anthropic_messages_transformation.py
View file

@ -55,12 +55,11 @@ class TestAzureAnthropicMessagesConfig:
assert isinstance(call_args[1]["litellm_params"], GenericLiteLLMParams)
assert call_args[1]["litellm_params"].api_key == "test-api-key"
assert "anthropic-version" in result
assert "x-api-key" in result
assert result["x-api-key"] == "test-api-key"
assert "api-key" not in result
# api-key header is preserved as-is (no conversion to x-api-key)
assert "api-key" in result
def test_validate_anthropic_messages_environment_converts_api_key_to_x_api_key(self):
"""Test that api-key header is converted to x-api-key"""
def test_validate_anthropic_messages_environment_preserves_api_key_header(self):
"""Test that api-key header is preserved as-is (Azure handles the header internally)"""
config = AzureAnthropicMessagesConfig()
headers = {}
model = "claude-sonnet-4-5"
@ -80,10 +79,9 @@ class TestAzureAnthropicMessagesConfig:
litellm_params=litellm_params,
)
# Verify api-key was converted to x-api-key
assert "x-api-key" in result
assert result["x-api-key"] == "test-api-key"
assert "api-key" not in result
# Verify api-key header is preserved as-is
assert "api-key" in result
assert result["api-key"] == "test-api-key"
def test_validate_anthropic_messages_environment_sets_headers(self):
"""Test that required headers are set"""
@ -110,7 +108,8 @@ class TestAzureAnthropicMessagesConfig:
assert result["anthropic-version"] == "2023-06-01"
assert "content-type" in result
assert result["content-type"] == "application/json"
assert "x-api-key" in result
# api-key header is preserved as-is
assert "api-key" in result
def test_get_complete_url_with_base_url(self):
"""Test get_complete_url with base URL"""
@ -239,3 +238,47 @@ class TestAzureAnthropicMessagesConfig:
assert "tools" in params
assert "tool_choice" in params
class TestProviderConfigManagerAzureAnthropicMessages:
"""Test ProviderConfigManager returns correct config for Azure AI Anthropic Messages API"""
def test_get_provider_anthropic_messages_config_returns_azure_config(self):
"""Test that ProviderConfigManager returns AzureAnthropicMessagesConfig for azure_ai provider with claude model"""
import litellm
from litellm.utils import ProviderConfigManager
config = ProviderConfigManager.get_provider_anthropic_messages_config(
model="claude-sonnet-4-5_gb_20250929",
provider=litellm.LlmProviders.AZURE_AI,
)
assert config is not None
assert isinstance(config, AzureAnthropicMessagesConfig)
def test_get_provider_anthropic_messages_config_case_insensitive_model_name(self):
"""Test that model name check is case insensitive"""
import litellm
from litellm.utils import ProviderConfigManager
# Test with uppercase CLAUDE
config = ProviderConfigManager.get_provider_anthropic_messages_config(
model="CLAUDE-SONNET-4-5",
provider=litellm.LlmProviders.AZURE_AI,
)
assert config is not None
assert isinstance(config, AzureAnthropicMessagesConfig)
def test_get_provider_anthropic_messages_config_returns_none_for_non_claude_model(
self,
):
"""Test that ProviderConfigManager returns None for non-claude model on azure_ai"""
import litellm
from litellm.utils import ProviderConfigManager
config = ProviderConfigManager.get_provider_anthropic_messages_config(
model="gpt-4o",
provider=litellm.LlmProviders.AZURE_AI,
)
assert config is None

11
tests/test_litellm/llms/azure_ai/claude/test_azure_anthropic_transformation.py
View file

@ -103,8 +103,8 @@ class TestAzureAnthropicConfig:
call_args = mock_validate.call_args
assert call_args[1]["litellm_params"].api_key == "provided-api-key"
def test_validate_environment_converts_api_key_to_x_api_key(self):
"""Test that api-key header is converted to x-api-key (Azure Anthropic uses x-api-key)"""
def test_validate_environment_preserves_api_key_header(self):
"""Test that api-key header is preserved as-is (Azure handles the header internally)"""
config = AzureAnthropicConfig()
headers = {}
model = "claude-sonnet-4-5"
@ -127,10 +127,9 @@ class TestAzureAnthropicConfig:
litellm_params=litellm_params,
)
# Verify api-key was converted to x-api-key
assert "x-api-key" in result
assert result["x-api-key"] == "test-api-key"
assert "api-key" not in result
# Verify api-key header is preserved as-is
assert "api-key" in result
assert result["api-key"] == "test-api-key"
def test_validate_environment_sets_anthropic_version(self):
"""Test that anthropic-version header is set"""

69
tests/test_litellm/llms/bedrock/chat/test_writer_palmyra.py Normal file
View file

@ -0,0 +1,69 @@
"""
Tests for Writer Palmyra X5 and X4 models on Bedrock Converse.
"""
import os
import sys
import pytest
sys.path.insert(
0, os.path.abspath("../../../../..")
) # Adds the parent directory to the system path
from litellm.llms.bedrock.common_utils import BedrockModelInfo
def test_writer_palmyra_routes_to_converse():
"""
Test that Writer Palmyra models route to converse API.
"""
bedrock_model_info = BedrockModelInfo
# Test base model routes to converse
bedrock_route = bedrock_model_info.get_bedrock_route(
model="bedrock/writer.palmyra-x5-v1:0"
)
assert bedrock_route == "converse"
bedrock_route = bedrock_model_info.get_bedrock_route(
model="bedrock/writer.palmyra-x4-v1:0"
)
assert bedrock_route == "converse"
def test_writer_palmyra_cross_region_routes_to_converse():
"""
Test that Writer Palmyra models with cross-region inference prefix route to converse API.
"""
bedrock_model_info = BedrockModelInfo
# Test cross-region inference profile routes to converse
bedrock_route = bedrock_model_info.get_bedrock_route(
model="bedrock/us.writer.palmyra-x5-v1:0"
)
assert bedrock_route == "converse"
bedrock_route = bedrock_model_info.get_bedrock_route(
model="bedrock/us.writer.palmyra-x4-v1:0"
)
assert bedrock_route == "converse"
def test_writer_palmyra_base_model_extraction():
"""
Test that base model is correctly extracted from Writer Palmyra cross-region models.
"""
bedrock_model_info = BedrockModelInfo
# Test us. prefix is stripped correctly
base_model = bedrock_model_info.get_base_model(
model="bedrock/us.writer.palmyra-x5-v1:0"
)
assert base_model == "writer.palmyra-x5-v1:0"
base_model = bedrock_model_info.get_base_model(
model="bedrock/us.writer.palmyra-x4-v1:0"
)
assert base_model == "writer.palmyra-x4-v1:0"

125
tests/test_litellm/llms/openai/chat/test_openai_gpt_transformation.py Normal file
View file

@ -0,0 +1,125 @@
"""
Tests for OpenAI GPT transformation (litellm/llms/openai/chat/gpt_transformation.py)
"""
import pytest
import sys
import os
sys.path.insert(0, os.path.abspath("../../../../.."))
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
class TestOpenAIGPTConfig:
"""Tests for OpenAIGPTConfig class"""
def setup_method(self):
self.config = OpenAIGPTConfig()
def test_user_param_supported_for_regular_models(self):
"""Test that 'user' param is in supported params for regular OpenAI models."""
supported_params = self.config.get_supported_openai_params("gpt-4o")
assert "user" in supported_params
supported_params = self.config.get_supported_openai_params("gpt-4.1-mini")
assert "user" in supported_params
def test_user_param_supported_for_responses_api_models(self):
"""Test that 'user' param is in supported params for responses API models.
Regression test for: https://github.com/BerriAI/litellm/issues/17633
When using model="openai/responses/gpt-4.1", the 'user' parameter should
be included in supported params so it reaches OpenAI and SpendLogs.
"""
# responses/gpt-4.1-mini should support 'user' just like gpt-4.1-mini
supported_params = self.config.get_supported_openai_params("responses/gpt-4.1-mini")
assert "user" in supported_params
supported_params = self.config.get_supported_openai_params("responses/gpt-4o")
assert "user" in supported_params
supported_params = self.config.get_supported_openai_params("responses/gpt-4.1")
assert "user" in supported_params
def test_model_normalization_for_responses_prefix(self):
"""Test that models with 'responses/' prefix are normalized correctly.
The fix normalizes 'responses/gpt-4.1' to 'gpt-4.1' when checking
if the model is in the list of supported OpenAI models.
"""
# Both should have the same supported params
regular_params = self.config.get_supported_openai_params("gpt-4.1-mini")
responses_params = self.config.get_supported_openai_params("responses/gpt-4.1-mini")
# 'user' should be in both
assert "user" in regular_params
assert "user" in responses_params
def test_base_params_always_included(self):
"""Test that base params are always included regardless of model."""
base_expected_params = [
"frequency_penalty",
"max_tokens",
"temperature",
"top_p",
"stream",
"tools",
"tool_choice",
]
supported_params = self.config.get_supported_openai_params("responses/gpt-4.1-mini")
for param in base_expected_params:
assert param in supported_params, f"Expected '{param}' in supported params"
class TestGetOptionalParamsIntegration:
"""Integration tests using litellm.get_optional_params()"""
def test_user_in_optional_params_for_responses_model(self):
"""Test that 'user' ends up in optional_params when using responses API models.
Regression test for: https://github.com/BerriAI/litellm/issues/17633
This verifies the full flow through get_optional_params().
"""
from litellm.utils import get_optional_params
# Test with responses model
optional_params = get_optional_params(
model="responses/gpt-4.1-mini",
custom_llm_provider="openai",
user="test-user-123",
)
assert optional_params.get("user") == "test-user-123"
def test_user_in_optional_params_for_regular_model(self):
"""Test that 'user' ends up in optional_params for regular OpenAI models."""
from litellm.utils import get_optional_params
optional_params = get_optional_params(
model="gpt-4o",
custom_llm_provider="openai",
user="test-user-456",
)
assert optional_params.get("user") == "test-user-456"
def test_user_param_consistency_between_regular_and_responses(self):
"""Test that 'user' param behavior is consistent between regular and responses models."""
from litellm.utils import get_optional_params
regular_params = get_optional_params(
model="gpt-4.1-mini",
custom_llm_provider="openai",
user="my-end-user",
)
responses_params = get_optional_params(
model="responses/gpt-4.1-mini",
custom_llm_provider="openai",
user="my-end-user",
)
# Both should include user
assert regular_params.get("user") == "my-end-user"
assert responses_params.get("user") == "my-end-user"

142
tests/test_litellm/llms/sap/chat/test_sap_chat_calls.py Normal file
View file

@ -0,0 +1,142 @@
import httpx
from unittest.mock import patch, PropertyMock
import pytest
mock_response = {
"request_id": "e86a0b4e-53e3-97dc-a5f7-82e451376b23",
"intermediate_results": {
"templating": [{"content": "Say hello", "role": "user"}],
"llm": {
"id": "chatcmpl-CUB63bLTYnfO2CQR0r0rArkrbe8CH",
"object": "chat.completion",
"created": 1761308531,
"model": "gpt-4o-2024-08-06",
"system_fingerprint": "fp_4a331a0222",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "Hello from SAP!"},
"finish_reason": "stop",
}
],
"usage": {"completion_tokens": 7, "prompt_tokens": 3, "total_tokens": 10},
},
},
"final_result": {
"id": "chatcmpl-CUB63bLTYnfO2CQR0r0rArkrbe8CH",
"object": "chat.completion",
"created": 1761308531,
"model": "gpt-4o-2024-08-06",
"system_fingerprint": "fp_4a331a0222",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "Hello from SAP!"},
"finish_reason": "stop",
}
],
"usage": {"completion_tokens": 7, "prompt_tokens": 3, "total_tokens": 10},
},
}
mock_stream_response = [
b'data: {"request_id": "a07127d3-cb74-9427-a4dc-ef9bf424fb43", "intermediate_results": {"templating": [{"content": "Hi", "role": "user"}]}, "final_result": {"id": \'\', "object": \'\', "created": 0, "model": \'\', "system_fingerprint": null, "choices": [{"index": 0, "delta": {"content": ""}}]}}\n\n',
b'data: {"request_id": "a07127d3-cb74-9427-a4dc-ef9bf424fb43", "intermediate_results": {"llm": {"id": "chatcmpl-HelloMsg", "object": "chat.completion.chunk", "created": 1761319270, "model": "gpt-4o-2024-08-06", "system_fingerprint": "fp_HelloMsg", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "Hello "}}]}}, "final_result": {"id": "chatcmpl-HelloMsg", "object": "chat.completion.chunk", "created": 1761319270, "model": "gpt-4o-2024-08-06", "system_fingerprint": "fp_HelloMsg", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "Hello "}}]}}\n\n',
b'data: {"request_id": "a07127d3-cb74-9427-a4dc-ef9bf424fb43", "intermediate_results": {"llm": {"id": "chatcmpl-CUDtFmLex96SxakzBIzhLq2h8Axmk", "object": "chat.completion.chunk", "created": 1761319269, "model": "gpt-4o-2024-08-06", "system_fingerprint": "fp_4a331a0222", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "from SAP!"}, "finish_reason": "stop"}]}}, "final_result": {"id": "chatcmpl-CUDtFmLex96SxakzBIzhLq2h8Axmk", "object": "chat.completion.chunk", "created": 1761319269, "model": "gpt-4o-2024-08-06", "system_fingerprint": "fp_4a331a0222", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "from SAP!"}, "finish_reason": "stop"}]}}\n\n',
b"data: [DONE]\n\n",
]
@pytest.fixture
def sap_api_response():
return mock_response
@pytest.fixture
def sap_api_stream_response():
return mock_response
@pytest.fixture
def fake_token_creator():
return lambda: "Bearer FAKE_TOKEN", "https://api.ai.mock-sap.com", "fake-group"
@pytest.fixture
def fake_deployment_url():
return "https://api.ai.mock-sap.com/v2/inference/deployments/mockid"
@pytest.mark.parametrize("sync_mode", [True, False])
@pytest.mark.asyncio
async def test_sap_chat(
respx_mock,
sap_api_response,
fake_token_creator,
fake_deployment_url,
sync_mode,
):
import litellm
litellm.disable_aiohttp_transport = True
with patch(
"litellm.llms.sap.chat.transformation.GenAIHubOrchestrationConfig.deployment_url",
new_callable=PropertyMock,
return_value=fake_deployment_url,
), patch(
"litellm.llms.sap.chat.transformation.get_token_creator",
return_value=fake_token_creator,
):
model = "sap/gpt-4o"
messages = [{"role": "user", "content": "Hello"}]
respx_mock.post(f"{fake_deployment_url}/v2/completion").respond(
json=sap_api_response
)
if sync_mode:
response = litellm.completion(model=model, messages=messages)
else:
response = await litellm.acompletion(model=model, messages=messages)
assert response.choices[0].message.content == "Hello from SAP!"
assert response.model.startswith("gpt-4o")
assert response.usage.total_tokens == 10
@pytest.mark.asyncio
async def test_sap_streaming(
respx_mock,
sap_api_stream_response,
fake_token_creator,
fake_deployment_url,
):
import litellm
litellm.disable_aiohttp_transport = True
with patch(
"litellm.llms.sap.chat.transformation.GenAIHubOrchestrationConfig.deployment_url",
new_callable=PropertyMock,
return_value=fake_deployment_url,
), patch(
"litellm.llms.sap.chat.transformation.get_token_creator",
return_value=fake_token_creator,
):
model = "sap/gpt-4o"
messages = [{"role": "user", "content": "Hello"}]
respx_mock.post(f"{fake_deployment_url}/v2/completion").mock(
return_value=httpx.Response(
200,
content=mock_stream_response,
headers={"Content-Type": "text/event-stream"},
)
)
stream = litellm.completion(model=model, messages=messages, stream=True)
full = ""
for chunk in stream:
delta = getattr(chunk.choices[0].delta, "content", None) or ""
full += delta
assert full == "Hello from SAP!"

1607
tests/test_litellm/llms/sap/embed/test_sap_embedding.py Normal file
View file

File diff suppressed because it is too large Load diff

92
tests/test_litellm/proxy/_experimental/mcp_server/test_ui_session_utils.py Normal file
View file

@ -0,0 +1,92 @@
import pytest
from types import SimpleNamespace
from unittest.mock import AsyncMock
from litellm.constants import UI_SESSION_TOKEN_TEAM_ID
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy._experimental.mcp_server.ui_session_utils import (
build_effective_auth_contexts,
clone_user_api_key_auth_with_team,
resolve_ui_session_team_ids,
)
def test_clone_user_api_key_auth_with_team_creates_independent_copy():
original = UserAPIKeyAuth(team_id="team-original", user_id="user-123")
cloned = clone_user_api_key_auth_with_team(original, "team-override")
assert cloned is not original
assert cloned.team_id == "team-override"
assert original.team_id == "team-original"
@pytest.mark.asyncio
async def test_resolve_ui_session_team_ids_returns_unique_ids(monkeypatch):
user_auth = UserAPIKeyAuth(
team_id=UI_SESSION_TOKEN_TEAM_ID,
user_id="user-1",
)
fake_user = SimpleNamespace(
teams=["team-a", "team-b", "team-a", "", None, "team-c"]
)
monkeypatch.setattr(
"litellm.proxy.auth.auth_checks.get_user_object",
AsyncMock(return_value=fake_user),
)
import litellm.proxy.proxy_server as proxy_server
monkeypatch.setattr(proxy_server, "prisma_client", object())
monkeypatch.setattr(proxy_server, "proxy_logging_obj", None)
monkeypatch.setattr(proxy_server, "user_api_key_cache", None)
team_ids = await resolve_ui_session_team_ids(user_auth)
assert team_ids == ["team-a", "team-b", "team-c"]
@pytest.mark.asyncio
async def test_resolve_ui_session_team_ids_short_circuits_when_not_ui_session():
normal_user = UserAPIKeyAuth(team_id="regular-team", user_id="user-1")
result = await resolve_ui_session_team_ids(normal_user)
assert result == []
@pytest.mark.asyncio
async def test_build_effective_auth_contexts_returns_cloned_contexts(monkeypatch):
user_auth = UserAPIKeyAuth(team_id=UI_SESSION_TOKEN_TEAM_ID, user_id="user-42")
mock_resolve = AsyncMock(return_value=["team-one", "team-two"])
monkeypatch.setattr(
"litellm.proxy._experimental.mcp_server.ui_session_utils.resolve_ui_session_team_ids",
mock_resolve,
)
contexts = await build_effective_auth_contexts(user_auth)
assert [ctx.team_id for ctx in contexts] == ["team-one", "team-two"]
assert all(ctx is not user_auth for ctx in contexts)
mock_resolve.assert_awaited_once_with(user_auth)
@pytest.mark.asyncio
async def test_build_effective_auth_contexts_returns_original_when_no_resolution(monkeypatch):
user_auth = UserAPIKeyAuth(team_id="existing-team", user_id="user-7")
mock_resolve = AsyncMock(return_value=[])
monkeypatch.setattr(
"litellm.proxy._experimental.mcp_server.ui_session_utils.resolve_ui_session_team_ids",
mock_resolve,
)
contexts = await build_effective_auth_contexts(user_auth)
assert contexts == [user_auth]
mock_resolve.assert_awaited_once_with(user_auth)

30
tests/test_litellm/proxy/auth/test_route_checks.py
View file

@ -804,4 +804,34 @@ def test_proxy_admin_viewer_can_access_global_spend_tags():
pytest.fail(
f"proxy_admin_viewer should be able to access /global/spend/tags route. Got error: {str(e)}"
)
def test_route_in_additional_public_routes_wildcard_match():
"""
Test that route_in_additonal_public_routes supports wildcard patterns.
"""
from litellm.proxy.auth.auth_utils import route_in_additonal_public_routes
with patch("litellm.proxy.proxy_server.general_settings", {"public_routes": ["/api/*"]}), \
patch("litellm.proxy.proxy_server.premium_user", True):
# Wildcard should match subpaths
assert route_in_additonal_public_routes("/api/users") is True
assert route_in_additonal_public_routes("/api/users/123") is True
# Should not match different prefix
assert route_in_additonal_public_routes("/other/path") is False
def test_route_in_additional_public_routes_exact_match():
"""
Test that route_in_additonal_public_routes supports exact matches.
"""
from litellm.proxy.auth.auth_utils import route_in_additonal_public_routes
with patch("litellm.proxy.proxy_server.general_settings", {"public_routes": ["/health", "/status"]}), \
patch("litellm.proxy.proxy_server.premium_user", True):
# Exact matches should work
assert route_in_additonal_public_routes("/health") is True
assert route_in_additonal_public_routes("/status") is True
# Non-matching routes should fail
assert route_in_additonal_public_routes("/other") is False

94
tests/test_litellm/proxy/hooks/test_dynamic_rate_limiter_v3.py
View file

@ -1418,6 +1418,100 @@ async def test_async_log_success_event_increments_by_actual_tokens():
assert any("priority_model" in k and "dev" in k for k in keys), "Should increment priority_model with 'dev' priority"
@pytest.mark.asyncio
async def test_saturation_check_cache_ttl_configuration():
"""
Test that saturation_check_cache_ttl controls how long saturation values are cached locally.
This validates the configurable TTL for multi-node consistency:
- When saturation_check_cache_ttl is set, local cache should expire after that duration
- After expiration, fresh values should be fetched from Redis
- This prevents nodes from having stale saturation data in multi-node deployments
"""
os.environ["LITELLM_LICENSE"] = "test-license-key"
# Set a short TTL for testing (5 seconds)
original_ttl = litellm.priority_reservation_settings.saturation_check_cache_ttl
litellm.priority_reservation_settings.saturation_check_cache_ttl = 5
try:
dual_cache = DualCache()
handler = DynamicRateLimitHandler(internal_usage_cache=dual_cache)
model = "test-saturation-ttl"
llm_router = Router(
model_list=[
{
"model_name": model,
"litellm_params": {
"model": "gpt-3.5-turbo",
"api_key": "test-key",
"api_base": "test-base",
"rpm": 100,
"tpm": 1000,
},
}
]
)
handler.update_variables(llm_router=llm_router)
# Verify the TTL getter returns configured value
assert handler._get_saturation_check_cache_ttl() == 5, (
"TTL should be configurable via priority_reservation_settings"
)
# Track async_get_cache calls to verify TTL is passed
get_cache_calls = []
original_get_cache = handler.internal_usage_cache.async_get_cache
async def mock_get_cache(key, litellm_parent_otel_span=None, local_only=False, **kwargs):
get_cache_calls.append({
"key": key,
"ttl": kwargs.get("ttl"),
"local_only": local_only,
})
return None # Simulate cache miss
handler.internal_usage_cache.async_get_cache = mock_get_cache
# Call _get_saturation_value_from_cache
counter_key = handler.v3_limiter.create_rate_limit_keys(
key="model_saturation_check",
value=model,
rate_limit_type="requests",
)
await handler._get_saturation_value_from_cache(counter_key=counter_key)
# Verify async_get_cache was called with the configured TTL
assert len(get_cache_calls) == 1, "Expected 1 cache call"
assert get_cache_calls[0]["ttl"] == 5, (
f"Expected TTL of 5 seconds, got {get_cache_calls[0]['ttl']}"
)
assert get_cache_calls[0]["local_only"] is False, (
"Should check Redis (local_only=False) for multi-node consistency"
)
# Test with different TTL value
get_cache_calls.clear()
litellm.priority_reservation_settings.saturation_check_cache_ttl = 30
await handler._get_saturation_value_from_cache(counter_key=counter_key)
assert get_cache_calls[0]["ttl"] == 30, (
f"TTL should update to 30 seconds, got {get_cache_calls[0]['ttl']}"
)
print("Saturation check cache TTL test passed:")
print(" - TTL is configurable via priority_reservation_settings.saturation_check_cache_ttl")
print(" - TTL is passed to async_get_cache for local cache expiration control")
print(" - local_only=False ensures Redis is checked for multi-node consistency")
finally:
# Restore original TTL
litellm.priority_reservation_settings.saturation_check_cache_ttl = original_ttl
@pytest.mark.asyncio
async def test_async_log_success_event_uses_team_priority_from_auth_metadata():
"""

130
tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py
View file

@ -151,4 +151,132 @@ class TestAnthropicLoggingHandlerModelFallback:
if not model and hasattr(logging_obj, 'model_call_details') and logging_obj.model_call_details.get('model'):
model = logging_obj.model_call_details.get('model')
assert model == "" # Should remain empty
assert model == "" # Should remain empty
class TestAzureAnthropicCostCalculation:
"""Test the custom_llm_provider cost calculation logic for Azure AI Anthropic."""
def _create_mock_logging_obj(
self, model: str = None, custom_llm_provider: str = None
) -> LiteLLMLoggingObj:
"""Create a mock logging object with optional model and custom_llm_provider"""
mock_logging_obj = MagicMock()
mock_model_call_details = {}
if model:
mock_model_call_details["model"] = model
if custom_llm_provider:
mock_model_call_details["custom_llm_provider"] = custom_llm_provider
mock_logging_obj.model_call_details = mock_model_call_details
mock_logging_obj.litellm_call_id = "test-call-id"
return mock_logging_obj
@patch("litellm.completion_cost")
def test_cost_calculation_with_azure_ai_custom_llm_provider(
self, mock_completion_cost
):
"""Test that custom_llm_provider is passed to completion_cost for Azure AI Anthropic"""
from litellm.types.utils import ModelResponse
from datetime import datetime
mock_completion_cost.return_value = 0.001
logging_obj = self._create_mock_logging_obj(
model="claude-sonnet-4-5_gb_20250929", custom_llm_provider="azure_ai"
)
mock_response = MagicMock(spec=ModelResponse)
mock_response.id = "test-id"
mock_response.model = "claude-sonnet-4-5_gb_20250929"
kwargs = {}
start_time = datetime.now()
end_time = datetime.now()
AnthropicPassthroughLoggingHandler._create_anthropic_response_logging_payload(
litellm_model_response=mock_response,
model="claude-sonnet-4-5_gb_20250929",
kwargs=kwargs,
start_time=start_time,
end_time=end_time,
logging_obj=logging_obj,
)
# Verify completion_cost was called with the correct parameters
mock_completion_cost.assert_called_once()
call_kwargs = mock_completion_cost.call_args[1]
assert call_kwargs["model"] == "azure_ai/claude-sonnet-4-5_gb_20250929"
assert call_kwargs["custom_llm_provider"] == "azure_ai"
@patch("litellm.completion_cost")
def test_cost_calculation_without_custom_llm_provider(self, mock_completion_cost):
"""Test that cost calculation works without custom_llm_provider (standard Anthropic)"""
from litellm.types.utils import ModelResponse
from datetime import datetime
mock_completion_cost.return_value = 0.001
# No custom_llm_provider in model_call_details
logging_obj = self._create_mock_logging_obj(model="claude-3-sonnet-20240229")
mock_response = MagicMock(spec=ModelResponse)
mock_response.id = "test-id"
mock_response.model = "claude-3-sonnet-20240229"
kwargs = {}
start_time = datetime.now()
end_time = datetime.now()
AnthropicPassthroughLoggingHandler._create_anthropic_response_logging_payload(
litellm_model_response=mock_response,
model="claude-3-sonnet-20240229",
kwargs=kwargs,
start_time=start_time,
end_time=end_time,
logging_obj=logging_obj,
)
# Verify completion_cost was called without provider prefix
mock_completion_cost.assert_called_once()
call_kwargs = mock_completion_cost.call_args[1]
assert call_kwargs["model"] == "claude-3-sonnet-20240229"
assert call_kwargs["custom_llm_provider"] is None
@patch("litellm.completion_cost")
def test_cost_calculation_does_not_duplicate_provider_prefix(
self, mock_completion_cost
):
"""Test that provider prefix is not duplicated if already present in model name"""
from litellm.types.utils import ModelResponse
from datetime import datetime
mock_completion_cost.return_value = 0.001
logging_obj = self._create_mock_logging_obj(
model="azure_ai/claude-sonnet-4-5_gb_20250929",
custom_llm_provider="azure_ai",
)
mock_response = MagicMock(spec=ModelResponse)
mock_response.id = "test-id"
mock_response.model = "azure_ai/claude-sonnet-4-5_gb_20250929"
kwargs = {}
start_time = datetime.now()
end_time = datetime.now()
# Model already has the provider prefix
AnthropicPassthroughLoggingHandler._create_anthropic_response_logging_payload(
litellm_model_response=mock_response,
model="azure_ai/claude-sonnet-4-5_gb_20250929",
kwargs=kwargs,
start_time=start_time,
end_time=end_time,
logging_obj=logging_obj,
)
# Verify provider prefix was not duplicated
mock_completion_cost.assert_called_once()
call_kwargs = mock_completion_cost.call_args[1]
assert call_kwargs["model"] == "azure_ai/claude-sonnet-4-5_gb_20250929"
assert call_kwargs["custom_llm_provider"] == "azure_ai"

0
test_litellm/proxy/pass_through_endpoints/test_passthrough_guardrails_field_targeting.py → tests/test_litellm/proxy/pass_through_endpoints/test_passthrough_guardrails_field_targeting.py
View file

38
tests/test_litellm/proxy/test_proxy_server.py
View file

@ -124,6 +124,44 @@ def test_login_v2_returns_redirect_url_and_sets_cookie(monkeypatch):
)
def test_fallback_login_has_no_deprecation_banner(client_no_auth):
response = client_no_auth.get("/fallback/login")
assert response.status_code == 200
html = response.text
assert '<div class="deprecation-banner">' not in html
assert "Deprecated:" not in html
assert "<form" in html
def test_sso_key_generate_shows_deprecation_banner(client_no_auth, monkeypatch):
# Ensure the route returns the HTML form instead of redirecting
monkeypatch.setattr(
"litellm.proxy.management_endpoints.ui_sso.show_missing_vars_in_env",
lambda: None,
)
monkeypatch.setattr(
"litellm.proxy.management_endpoints.ui_sso.SSOAuthenticationHandler.get_redirect_url_for_sso",
lambda *args, **kwargs: "http://test/redirect",
)
monkeypatch.setattr(
"litellm.proxy.management_endpoints.ui_sso.SSOAuthenticationHandler._get_cli_state",
lambda *args, **kwargs: None,
)
monkeypatch.setattr(
"litellm.proxy.management_endpoints.ui_sso.SSOAuthenticationHandler.should_use_sso_handler",
lambda *args, **kwargs: False,
)
monkeypatch.setenv("UI_USERNAME", "admin")
response = client_no_auth.get("/sso/key/generate")
assert response.status_code == 200
html = response.text
assert '<div class="deprecation-banner">' in html
assert "Deprecated:" in html
@pytest.mark.asyncio
async def test_initialize_scheduled_jobs_credentials(monkeypatch):
"""

5
tests/test_litellm/router_utils/test_router_utils_common_utils.py
View file

@ -248,6 +248,11 @@ class TestFilterWebSearchDeployments:
result = filter_web_search_deployments(sample_deployments, {"tools": []})
assert result == sample_deployments
def test_none_tools_returns_all(self, sample_deployments):
"""When tools is explicitly None, return all deployments (regression test for #17672)"""
result = filter_web_search_deployments(sample_deployments, {"tools": None})
assert result == sample_deployments
def test_non_web_search_tools_returns_all(self, sample_deployments):
"""When tools don't include web_search, return all deployments"""
request_kwargs = {"tools": [{"type": "function", "function": {}}]}

33
ui/litellm-dashboard/src/components/add_model/add_model_tab.test.tsx
View file

@ -1,5 +1,5 @@
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { render, renderHook, screen } from "@testing-library/react";
import { render, renderHook, screen, waitFor } from "@testing-library/react";
import { Form } from "antd";
import type { UploadProps } from "antd/es/upload";
import { describe, expect, it, vi } from "vitest";
@ -37,6 +37,22 @@ vi.mock("../networking", async () => {
};
});
vi.mock("@/app/(dashboard)/hooks/providers/useProviderFields", () => ({
useProviderFields: vi.fn().mockReturnValue({
data: [
{
provider: "OpenAI",
provider_display_name: "OpenAI",
litellm_provider: "openai",
default_model_placeholder: "gpt-3.5-turbo",
credential_fields: [],
},
],
isLoading: false,
error: null,
}),
}));
const createQueryClient = () =>
new QueryClient({
defaultOptions: {
@ -231,8 +247,15 @@ describe("Add Model Tab", () => {
</QueryClientProvider>,
);
const testConnectButtons = await screen.findAllByRole("button", { name: "Test Connect" });
expect(testConnectButtons.length).toBeGreaterThan(0);
expect(await screen.findByRole("button", { name: "Add Model" })).toBeInTheDocument();
}, 10000); // 10 seconds timeout for complex logic
// Wait for async operations to complete and buttons to appear
await waitFor(
async () => {
const testConnectButtons = await screen.findAllByRole("button", { name: "Test Connect" });
expect(testConnectButtons.length).toBeGreaterThan(0);
const addModelButton = await screen.findByRole("button", { name: "Add Model" });
expect(addModelButton).toBeInTheDocument();
},
{ timeout: 10000 },
);
}, 15000); // 15 second timeout to allow waitFor to complete
});

74
ui/litellm-dashboard/src/components/useful_links_management.test.tsx Normal file
View file

@ -0,0 +1,74 @@
import { render, screen, waitFor } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { Modal } from "antd";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import UsefulLinksManagement from "./useful_links_management";
import NotificationsManager from "./molecules/notifications_manager";
import { getPublicModelHubInfo, updateUsefulLinksCall, getProxyBaseUrl } from "./networking";
vi.mock("./networking", () => ({
getPublicModelHubInfo: vi.fn(),
updateUsefulLinksCall: vi.fn(),
getProxyBaseUrl: vi.fn(),
}));
vi.mock("./molecules/notifications_manager", () => ({
__esModule: true,
default: {
success: vi.fn(),
fromBackend: vi.fn(),
},
}));
const mockedGetPublicModelHubInfo = vi.mocked(getPublicModelHubInfo);
const mockedUpdateUsefulLinksCall = vi.mocked(updateUsefulLinksCall);
const mockedGetProxyBaseUrl = vi.mocked(getProxyBaseUrl);
const mockedNotifications = vi.mocked(NotificationsManager);
let modalSuccessSpy: any;
describe("UsefulLinksManagement", () => {
beforeEach(() => {
mockedGetPublicModelHubInfo.mockResolvedValue({
docs_title: "Docs",
custom_docs_description: null,
litellm_version: "1.0.0",
useful_links: {},
});
mockedUpdateUsefulLinksCall.mockResolvedValue({});
mockedGetProxyBaseUrl.mockReturnValue("https://proxy.example.com");
modalSuccessSpy = vi.spyOn(Modal, "success").mockImplementation(() => ({ destroy: vi.fn() }) as any);
});
afterEach(() => {
modalSuccessSpy.mockRestore();
vi.clearAllMocks();
});
it("should render link management for admin users", async () => {
render(<UsefulLinksManagement accessToken="token" userRole="Admin" />);
expect(await screen.findByText("Link Management")).toBeInTheDocument();
await waitFor(() => expect(mockedGetPublicModelHubInfo).toHaveBeenCalled());
});
it("should add a new link when fields are valid", async () => {
const user = userEvent.setup();
render(<UsefulLinksManagement accessToken="token" userRole="Admin" />);
const displayNameInput = await screen.findByPlaceholderText("Friendly name");
const urlInput = screen.getByPlaceholderText("https://example.com");
await user.type(displayNameInput, "Docs");
await user.type(urlInput, "https://docs.example.com");
await user.click(screen.getByRole("button", { name: /add link/i }));
await waitFor(() =>
expect(mockedUpdateUsefulLinksCall).toHaveBeenCalledWith("token", { Docs: "https://docs.example.com" }),
);
expect(await screen.findByText("Docs")).toBeInTheDocument();
expect(screen.getByText("https://docs.example.com")).toBeInTheDocument();
expect(mockedNotifications.success).toHaveBeenCalledWith("Link added successfully");
});
});

30
ui/litellm-dashboard/src/components/useful_links_management.tsx
View file

@ -210,21 +210,6 @@ const UsefulLinksManagement: React.FC<UsefulLinksManagementProps> = ({ accessTok
<div className="mb-6">
<Text className="text-sm font-medium text-gray-700 mb-2">Add New Link</Text>
<div className="grid grid-cols-3 gap-4">
<div>
<label className="block text-xs text-gray-500 mb-1">URL</label>
<input
type="text"
value={newLink.url}
onChange={(e) =>
setNewLink({
...newLink,
url: e.target.value,
})
}
placeholder="https://example.com"
className="w-full px-3 py-2 border border-gray-300 rounded-md text-sm"
/>
</div>
<div>
<label className="block text-xs text-gray-500 mb-1">Display Name</label>
<input
@ -240,6 +225,21 @@ const UsefulLinksManagement: React.FC<UsefulLinksManagementProps> = ({ accessTok
className="w-full px-3 py-2 border border-gray-300 rounded-md text-sm"
/>
</div>
<div>
<label className="block text-xs text-gray-500 mb-1">URL</label>
<input
type="text"
value={newLink.url}
onChange={(e) =>
setNewLink({
...newLink,
url: e.target.value,
})
}
placeholder="https://example.com"
className="w-full px-3 py-2 border border-gray-300 rounded-md text-sm"
/>
</div>
<div className="flex items-end">
<button
onClick={handleAddLink}