scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-07 08:26:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 36

Merge branch 'main' into litellm_dev_03_02_2026_p1

Browse source
This commit is contained in:
Krish Dholakia 2026-03-03 18:18:41 -08:00 • committed by GitHub
commit c4ed73552f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
112 changed files with 9560 additions and 1051 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

175
docs/my-website/blog/gemini_3_1_flash_lite/index.md Normal file
View file

@ -0,0 +1,175 @@
---
slug: gemini_3_1_flash_lite_preview
title: "DAY 0 Support: Gemini 3.1 Flash Lite Preview on LiteLLM"
date: 2026-03-03T08:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
description: "Guide to using Gemini 3.1 Flash Lite Preview on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms, supernova]
hide_table_of_contents: false
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Gemini 3.1 Flash Lite Preview Day 0 Support
LiteLLM now supports `gemini-3.1-flash-lite-preview` with full day 0 support!
:::note
If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
:::
## Deploy this version
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.80.8-stable.1
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==v1.80.8-stable.1
```
</TabItem>
</Tabs>
## What's New
Supports all four thinking levels:
- **MINIMAL**: Ultra-fast responses with minimal reasoning
- **LOW**: Simple instruction following
- **MEDIUM**: Balanced reasoning for complex tasks
- **HIGH**: Maximum reasoning depth (dynamic)
---
## Quick Start
<Tabs>
<TabItem value="sdk" label="SDK">
**Basic Usage**
```python
from litellm import completion
response = completion(
model="gemini/gemini-3.1-flash-lite-preview",
messages=[{"role": "user", "content": "Extract key entities from this text: ..."}],
)
print(response.choices[0].message.content)
```
**With Thinking Levels**
```python
from litellm import completion
# Use MEDIUM thinking for complex reasoning tasks
response = completion(
model="gemini/gemini-3.1-flash-lite-preview",
messages=[{"role": "user", "content": "Analyze this dataset and identify patterns"}],
reasoning_effort="medium", # low, medium , high
)
print(response.choices[0].message.content)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
**1. Setup config.yaml**
```yaml
model_list:
- model_name: gemini-3.1-flash-lite
litellm_params:
model: gemini/gemini-3.1-flash-lite-preview
api_key: os.environ/GEMINI_API_KEY
# Or use Vertex AI
- model_name: vertex-gemini-3.1-flash-lite
litellm_params:
model: vertex_ai/gemini-3.1-flash-lite-preview
vertex_project: your-project-id
vertex_location: us-central1
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
```
**3. Make requests**
```bash
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-d '{
"model": "gemini-3.1-flash-lite",
"messages": [{"role": "user", "content": "Extract structured data from this text"}],
"reasoning_effort": "low"
}'
```
</TabItem>
</Tabs>
---
## Supported Endpoints
LiteLLM provides **full end-to-end support** for Gemini 3.1 Flash Lite Preview on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
- Function calling with thought signatures
- Multi-turn conversations
- All Gemini 3-specific features (thinking levels, thought signatures)
- Full multimodal support (text, image, audio, video)
---
## `reasoning_effort` Mapping for Gemini 3.1
LiteLLM automatically maps OpenAI's `reasoning_effort` parameter to Gemini's `thinkingLevel`:
| reasoning_effort | thinking_level | Use Case |
|------------------|----------------|----------|
| `minimal` | `minimal` | Ultra-fast responses, simple queries |
| `low` | `low` | Basic instruction following |
| `medium` | `medium` | Balanced reasoning for moderate complexity |
| `high` | `high` | Maximum reasoning depth, complex problems |
| `disable` | `minimal` | Disable extended reasoning |
| `none` | `minimal` | No extended reasoning |

132
docs/my-website/blog/httpx_cache_eviction_incident/index.md Normal file
View file

@ -0,0 +1,132 @@
---
slug: httpx-cache-eviction-incident
title: "Incident Report: Cache Eviction Closes In-Use httpx Clients"
date: 2026-02-27T10:00:00
authors:
- name: Ryan Crabbe
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
tags: [incident-report, caching, stability]
hide_table_of_contents: false
---
**Date:** February 27, 2026
**Duration:** ~6 days (Feb 21 merge -> Feb 27 fix)
**Severity:** High
**Status:** Resolved
> **Note:** This fix is available starting from LiteLLM `v1.81.14.rc.2` or higher.
## Summary
A change to improve Redis connection pool cleanup introduced a regression that closed **httpx clients** that were still actively being used by the proxy. The `LLMClientCache` (an in-memory TTL cache) stores both Redis clients *and* httpx clients under the same eviction policy. When a cache entry expired or was evicted, the new cleanup code called `aclose()`/`close()` on the evicted value which worked correctly for Redis clients, but destroyed httpx clients that other parts of the system still held references to and were actively using for LLM API calls.
**Impact:** Any proxy instance that hit the cache TTL (default 10 minutes) or capacity limit (200 entries) would have its httpx clients closed out from under it, causing requests to LLM providers to fail with connection errors.
---
## Background
`LLMClientCache` extends `InMemoryCache` and is used to cache SDK clients (OpenAI, Anthropic, etc.) to avoid re-creating them on every request. These clients are keyed by configuration + event loop ID. The cache has:
- **Max size:** 200 entries
- **Default TTL:** 10 minutes
When the cache is full or entries expire, `InMemoryCache.evict_cache()` calls `_remove_key()` to drop entries.
The cached values are a mix of:
- **Redis/async Redis clients** — owned exclusively by the cache, safe to close on eviction
- **httpx-backed SDK clients** (OpenAI, Anthropic, etc.) — shared references, still in use by router/model instances
---
## Root Cause
[PR #21717](https://github.com/BerriAI/litellm/pull/21717) overrode `_remove_key()` in `LLMClientCache` to close async clients on eviction:
<details>
<summary>Problematic code added in PR #21717</summary>
```python
class LLMClientCache(InMemoryCache):
def _remove_key(self, key: str) -> None:
value = self.cache_dict.get(key)
super()._remove_key(key)
if value is not None:
close_fn = getattr(value, "aclose", None) or getattr(value, "close", None)
if close_fn and asyncio.iscoroutinefunction(close_fn):
try:
asyncio.get_running_loop().create_task(close_fn())
except RuntimeError:
pass
elif close_fn and callable(close_fn):
try:
close_fn()
except Exception:
pass
```
</details>
The intent was correct for Redis clients — prevent connection pool leaks when cached Redis clients expire. But `LLMClientCache` also stores httpx-backed SDK clients (e.g., `AsyncOpenAI`, `AsyncAnthropic`). These clients:
1. Have an `aclose()` method (inherited from httpx)
2. Are still held by references elsewhere in the codebase (router, model instances)
3. Were being closed without any check on whether they were still in use
So when the cache evicted an entry, it would call `aclose()` on an httpx client that was still being used for active LLM requests → closed transport → connection errors.
---
## The Fix
[PR #22247](https://github.com/BerriAI/litellm/pull/22247) removed the `_remove_key` override entirely:
<details>
<summary>The fix (PR #22247)</summary>
```diff
class LLMClientCache(InMemoryCache):
- def _remove_key(self, key: str) -> None:
- """Close async clients before evicting them to prevent connection pool leaks."""
- value = self.cache_dict.get(key)
- super()._remove_key(key)
- if value is not None:
- close_fn = getattr(value, "aclose", None) or getattr(
- value, "close", None
- )
- ...
-
def update_cache_key_with_event_loop(self, key):
```
</details>
The eviction now simply drops the reference and lets Python's GC handle cleanup, which is safe because:
- httpx clients that are still referenced elsewhere stay alive
- Unreferenced clients get cleaned up by GC naturally
The other improvements from PR #21717 were kept:
- **`max_connections` respected for URL-based Redis configs**, previously silently dropped
- **`disconnect()` now closes both sync and async Redis clients**, sync client was previously leaked
- **Connection pool passthrough**, when a pool is provided with a URL config, it's used directly instead of creating a duplicate
---
## Remediation
| Action | Status | Code |
|--------|--------|------|
| Remove `_remove_key` override that closes shared clients on eviction | ✅ Done | [PR #22247](https://github.com/BerriAI/litellm/pull/22247) |
| Add e2e test: evicted client still usable (capacity) | ✅ Done | [PR #22313](https://github.com/BerriAI/litellm/pull/22313) |
| Add e2e test: expired client still usable (TTL) | ✅ Done | [PR #22313](https://github.com/BerriAI/litellm/pull/22313) |
The e2e tests go through `get_async_httpx_client()` the same code path the proxy uses in production and assert the client is still functional after eviction. These run in CI on every PR against `main`. If anyone modifies `LLMClientCache` eviction behavior, overrides `_remove_key`, or adds any form of client cleanup on eviction, these tests will fail regardless of the implementation approach.

321
docs/my-website/blog/responses_api_encrypted_content_incident/index.md Normal file
View file

@ -0,0 +1,321 @@
---
slug: responses-api-encrypted-content-incident
title: "Incident Report: Encrypted Content Failures in Multi-Region Responses API Load Balancing"
date: 2026-02-24T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
tags: [incident-report, proxy, responses-api, load-balancing]
hide_table_of_contents: false
---
**Date:** Feb 24, 2026
**Duration:** Ongoing (until fix deployed)
**Severity:** High (for users load balancing Responses API across different API keys)
**Status:** Resolved
## Summary
When load balancing OpenAI's Responses API across deployments with **different API keys** (e.g., different Azure regions or OpenAI organizations), follow-up requests containing encrypted content items (like `rs_...` reasoning items) would fail with:
```json
{
"error": {
"message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.",
"type": "invalid_request_error",
"code": "invalid_encrypted_content"
}
}
```
Encrypted content items are cryptographically tied to the API key's organization that created them. When the router load balanced a follow-up request to a deployment with a different API key, decryption failed.
- **Responses API calls with encrypted content:** Complete failure when routed to wrong deployment
- **Initial requests:** Unaffected — only follow-up requests containing encrypted items failed
- **Other API endpoints:** No impact — chat completions, embeddings, etc. functioned normally
{/* truncate */}
---
## Background
OpenAI's Responses API can return encrypted "reasoning items" (with IDs like `rs_...`) that contain intermediate reasoning steps. These items are encrypted with the organization's key and can only be decrypted by the same organization's API key.
When load balancing across deployments with different API keys, the existing affinity mechanisms were insufficient:
- **`responses_api_deployment_check`**: Requires `previous_response_id` which some clients (like Codex) don't provide
- **`deployment_affinity`**: Too broad — pins *all* requests from a user to one deployment, reducing effective quota by the number of users
- **`session_affinity`**: Requires explicit session IDs and still reduces quota
```mermaid
flowchart TD
A["1. Initial request to Responses API
router.aresponses()"] --> B["2. Router load balances to Deployment A
(API Key 1, Azure East US)"]
B --> C["3. Response contains encrypted item
rs_abc123 (encrypted with Org 1 key)"]
C --> D["4. Follow-up request includes rs_abc123 in input"]
D --> E["5. Router load balances to Deployment B
(API Key 2, Azure West Europe)"]
E -->|"Different API key"| F["6. ❌ Deployment B cannot decrypt rs_abc123
Error: invalid_encrypted_content"]
D -.->|"With encrypted_content_affinity"| G["5b. Router detects rs_abc123 was created by Deployment A"]
G --> H["6b. ✅ Routes to Deployment A (bypasses rate limits)
Request succeeds"]
style F fill:#f8d7da,stroke:#dc3545
style H fill:#d4edda,stroke:#28a745
style E fill:#fff3cd,stroke:#ffc107
style G fill:#d4edda,stroke:#28a745
```
---
## Root Cause
LiteLLM's router had no mechanism to track which deployment created specific encrypted content items and route follow-up requests accordingly. The router treated all deployments as interchangeable, leading to decryption failures when encrypted content crossed organizational boundaries.
**The Problem Flow:**
1. User calls `router.aresponses()` with model `gpt-5.1-codex`
2. Router load balances to Deployment A (Azure East US, API Key 1)
3. Response contains encrypted reasoning item `rs_abc123` (encrypted with Org 1's key)
4. User makes follow-up request with `rs_abc123` in the input
5. Router load balances to Deployment B (Azure West Europe, API Key 2)
6. Deployment B tries to decrypt `rs_abc123` with Org 2's key → **fails**
**Why Existing Solutions Didn't Work:**
- **`previous_response_id`**: Not provided by all clients (e.g., Codex)
- **`deployment_affinity`**: Pins *all* user requests to one deployment → reduces quota to 1/N where N = number of deployments
- **`session_affinity`**: Requires explicit session management and still reduces quota
**Timeline:**
1. Users configured multi-region Responses API load balancing with different API keys
2. Initial requests succeeded, but follow-up requests with encrypted content failed intermittently
3. Error rate correlated with number of deployments (more deployments = higher chance of routing to wrong one)
4. Investigation revealed encrypted content was organization-bound
5. Existing affinity mechanisms deemed unsuitable (quota reduction, missing `previous_response_id`)
6. New solution designed and implemented: `encrypted_content_affinity`
---
## The Fix
Implemented a new `encrypted_content_affinity` pre-call check that intelligently tracks encrypted content and routes follow-up requests **only when necessary**.
### Implementation
**1. Encoding `model_id` into output items** ([`responses/utils.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/responses/utils.py))
The same approach used for `previous_response_id` affinity — no cache needed. When a response contains output items with `encrypted_content`, LiteLLM encodes the originating deployment's `model_id` in **two places** for redundancy:
1. **Into the item ID** (if present): `rs_abc123` → `encitem_{base64("litellm:model_id:{model_id};item_id:rs_abc123")}`
2. **Into the encrypted_content itself**: Wraps the content with `litellm_enc:{base64("model_id:{model_id}")};{original_encrypted_content}`
```python
# Encoding item IDs (when present)
def _build_encrypted_item_id(model_id: str, item_id: str) -> str:
assembled = f"litellm:model_id:{model_id};item_id:{item_id}"
encoded = base64.b64encode(assembled.encode("utf-8")).decode("utf-8")
return f"encitem_{encoded}"
# Wrapping encrypted_content (always, for redundancy)
def _wrap_encrypted_content_with_model_id(encrypted_content: str, model_id: str) -> str:
metadata = f"model_id:{model_id}"
encoded_metadata = base64.b64encode(metadata.encode("utf-8")).decode("utf-8")
return f"litellm_enc:{encoded_metadata};{encrypted_content}"
```
**Why wrap encrypted_content directly?** Some clients (like Codex) don't consistently send item IDs in follow-up requests, but they always send the `encrypted_content` itself. By embedding `model_id` into the content, affinity works even when IDs are missing.
**Streaming responses:** The wrapping logic is applied to both:
- Final response objects (non-streaming)
- Individual streaming events (`response.output_item.added`, `response.output_item.done`)
This ensures clients receiving streaming responses get wrapped content they can send back.
Before forwarding to the upstream provider, LiteLLM restores the original item IDs and unwraps encrypted_content so the provider never sees the encoded form:
```python
# In responses/main.py — before calling the handler
input = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(input)
```
**2. `EncryptedContentAffinityCheck` — routing only** ([`encrypted_content_affinity_check.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py))
No `async_log_success_event` or cache lookups — the `model_id` is decoded directly from the item ID or encrypted_content:
```python
class EncryptedContentAffinityCheck(CustomLogger):
async def async_filter_deployments(self, model, healthy_deployments, ...):
"""Extract model_id from input items (ID or encrypted_content) and pin to that deployment."""
for item in request_kwargs.get("input", []):
# Try to extract model_id from two sources:
model_id = self._extract_model_id_from_input(item)
if model_id:
deployment = self._find_deployment_by_model_id(
healthy_deployments, model_id
)
if deployment:
request_kwargs["_encrypted_content_affinity_pinned"] = True
return [deployment]
return healthy_deployments
def _extract_model_id_from_input(self, item: dict) -> Optional[str]:
"""Extract model_id from either encoded ID or wrapped encrypted_content."""
# 1. Try decoding from item ID (if present)
item_id = item.get("id", "")
if item_id:
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(item_id)
if decoded:
return decoded["model_id"]
# 2. Try unwrapping from encrypted_content (fallback for clients that omit IDs)
encrypted_content = item.get("encrypted_content", "")
if encrypted_content and encrypted_content.startswith("litellm_enc:"):
model_id, _ = ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(
encrypted_content
)
return model_id
return None
```
**3. Rate Limit Bypass** ([`router.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/router.py))
When encrypted content requires a specific deployment, RPM/TPM limits are bypassed (the request would fail on any other deployment anyway):
```python
# In async_get_available_deployment, after filtering healthy deployments:
if (
request_kwargs.get("_encrypted_content_affinity_pinned")
and len(healthy_deployments) == 1
):
return healthy_deployments[0] # Bypass routing strategy (RPM/TPM checks)
```
**3. Configuration**
```yaml
router_settings:
routing_strategy: usage-based-routing-v2
enable_pre_call_checks: true
optional_pre_call_checks:
- encrypted_content_affinity
deployment_affinity_ttl_seconds: 86400 # 24 hours
```
### Key Benefits
✅ **No quota reduction**: Only pins requests containing encrypted items
✅ **Bypasses rate limits**: When encrypted content requires a specific deployment, RPM/TPM limits don't block it
✅ **No `previous_response_id` required**: Works by encoding `model_id` directly into the item ID
✅ **No cache required**: `model_id` is decoded on-the-fly from the item ID — no Redis, no TTL
✅ **Globally safe**: Can be enabled for all models; non-Responses-API calls are unaffected
✅ **Surgical precision**: Normal requests continue to load balance freely
---
## Remediation
| # | Action | Status | Code |
|---|---|---|---|
| 1 | Encode `model_id` into encrypted-content item IDs on response | ✅ Done | [`responses/utils.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/responses/utils.py) |
| 2 | Restore original item IDs before forwarding to upstream provider | ✅ Done | [`responses/main.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/responses/main.py) |
| 3 | `EncryptedContentAffinityCheck`: decode item IDs to route (no cache) | ✅ Done | [`encrypted_content_affinity_check.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py) |
| 4 | Add `encrypted_content_affinity` to `OptionalPreCallChecks` type | ✅ Done | [`types/router.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/types/router.py) |
| 5 | Implement rate limit bypass for affinity-pinned requests | ✅ Done | [`router.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/router.py) |
| 6 | Unit tests: encoding/decoding utilities, routing, RPM bypass | ✅ Done | [`test_encrypted_content_affinity_check.py`](https://github.com/BerriAI/litellm/blob/main/litellm/tests/test_litellm/router_utils/pre_call_checks/test_encrypted_content_affinity_check.py) |
| 7 | Documentation: Responses API guide, load balancing guide, config reference | ✅ Done | [Docs](https://docs.litellm.ai/docs/response_api#encrypted-content-affinity-multi-region-load-balancing) |
| 8 | **[Mar 3]** Fix streaming events to wrap encrypted_content | ✅ Done | [`responses/streaming_iterator.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm/responses/streaming_iterator.py) |
---
## Follow-up Fix: Streaming Responses (Mar 3, 2026)
### The Issue
After the initial fix was deployed, users reported that the `invalid_encrypted_content` error **still occurred** when using streaming responses with clients like Codex. Investigation revealed:
- ✅ Non-streaming responses: `encrypted_content` was correctly wrapped with `litellm_enc:` prefix
- ❌ Streaming responses: Individual `response.output_item.added` and `response.output_item.done` events contained **raw, unwrapped** `encrypted_content`
Since Codex and other clients consume responses as streams, they received unwrapped content in these events and sent it back in follow-up requests, causing the affinity check to fail.
### The Root Cause
The `_update_encrypted_content_item_ids_in_response` function only modified the **final** response object, which is used for non-streaming responses. For streaming responses, individual chunks are processed by `ResponsesAPIStreamingIterator._process_chunk`, which was **not** applying the wrapping logic to streaming events.
### The Fix
Modified `litellm/litellm/responses/streaming_iterator.py` to wrap `encrypted_content` in streaming events:
```python
# In ResponsesAPIStreamingIterator._process_chunk
if (
self.litellm_metadata
and self.litellm_metadata.get("encrypted_content_affinity_enabled")
):
event_type = getattr(openai_responses_api_chunk, "type", None)
if event_type in (
ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED,
ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
):
item = getattr(openai_responses_api_chunk, "item", None)
if item:
encrypted_content = getattr(item, "encrypted_content", None)
if encrypted_content and isinstance(encrypted_content, str):
model_id = (
self.litellm_metadata.get("model_info", {}).get("id")
if self.litellm_metadata
else None
)
if model_id:
wrapped_content = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
encrypted_content, model_id
)
setattr(item, "encrypted_content", wrapped_content)
```
This ensures that **all** `encrypted_content` sent to clients (streaming or non-streaming) is wrapped with `model_id` metadata, enabling consistent affinity routing.
---
## Migration Guide
### Before (Using `deployment_affinity`)
```yaml
router_settings:
optional_pre_call_checks:
- deployment_affinity # ❌ Reduces quota by number of users
```
**Problem:** All requests from a user pin to one deployment, reducing effective quota to 1/N.
### After (Using `encrypted_content_affinity`)
```yaml
router_settings:
optional_pre_call_checks:
- encrypted_content_affinity # ✅ Only pins requests with encrypted content
```
**Benefit:** Normal requests load balance freely, only encrypted content requests pin when necessary.
---

29
docs/my-website/docs/adding_provider/generic_guardrail_api.md
View file

@ -244,6 +244,35 @@ litellm_settings:
language: "en"
```
### Static and dynamic headers
You can send two kinds of headers to your guardrail endpoint:
- **Static headers** (`headers`): A key/value map sent with **every** request to your guardrail. Use this for fixed values (e.g. API keys, `X-Service-Name`). Configure in `litellm_params`:
```yaml
litellm_params:
guardrail: generic_guardrail_api
api_base: https://your-guardrail-api.com
headers:
X-Service-Name: "my-app"
X-API-Key: "secret"
```
- **Dynamic headers** (`extra_headers`): A list of **header names** that are forwarded from the **client request** to your guardrail. Only headers in this list (plus a small default allowlist such as `x-litellm-*`) have their values sent; others are sent as `[present]`. Use this to pass through client-provided headers (e.g. `x-request-id`, `x-correlation-id`). Configure in `litellm_params`:
```yaml
litellm_params:
guardrail: generic_guardrail_api
api_base: https://your-guardrail-api.com
extra_headers:
- x-request-id
- x-correlation-id
- x-custom-auth
```
This mirrors the [MCP static and extra headers](/docs/mcp#forwarding-custom-headers-to-mcp-servers) behavior.
### Example: Pillar Security
[Pillar Security](https://pillar.security) uses the Generic Guardrail API to provide comprehensive AI security scanning including prompt injection protection, PII/PCI detection, secret detection, and content moderation.

1
docs/my-website/docs/providers/gemini.md
View file

@ -2041,6 +2041,7 @@ response = litellm.completion(
| gemini-2.0-flash-lite-preview-02-05 | `completion(model='gemini/gemini-2.0-flash-lite-preview-02-05', messages)` | `os.environ['GEMINI_API_KEY']` |
| gemini-2.5-flash-preview-09-2025 | `completion(model='gemini/gemini-2.5-flash-preview-09-2025', messages)` | `os.environ['GEMINI_API_KEY']` |
| gemini-2.5-flash-lite-preview-09-2025 | `completion(model='gemini/gemini-2.5-flash-lite-preview-09-2025', messages)` | `os.environ['GEMINI_API_KEY']` |
| gemini-3.1-flash-lite-preview | `completion(model='gemini/gemini-3.1-flash-lite-preview', messages)` | `os.environ['GEMINI_API_KEY']` |
| gemini-flash-latest | `completion(model='gemini/gemini-flash-latest', messages)` | `os.environ['GEMINI_API_KEY']` |
| gemini-flash-lite-latest | `completion(model='gemini/gemini-flash-lite-latest', messages)` | `os.environ['GEMINI_API_KEY']` |

1
docs/my-website/docs/providers/vertex.md
View file

@ -1685,6 +1685,7 @@ litellm.vertex_location = "us-central1 # Your Location
| gemini-2.5-pro | `completion('gemini-2.5-pro', messages)`, `completion('vertex_ai/gemini-2.5-pro', messages)` |
| gemini-2.5-flash-preview-09-2025 | `completion('gemini-2.5-flash-preview-09-2025', messages)`, `completion('vertex_ai/gemini-2.5-flash-preview-09-2025', messages)` |
| gemini-2.5-flash-lite-preview-09-2025 | `completion('gemini-2.5-flash-lite-preview-09-2025', messages)`, `completion('vertex_ai/gemini-2.5-flash-lite-preview-09-2025', messages)` |
| gemini-3.1-flash-lite-preview | `completion('gemini-3.1-flash-lite-preview', messages)`, `completion('vertex_ai/gemini-3.1-flash-lite-preview', messages)` |
## Private Service Connect (PSC) Endpoints

2
docs/my-website/docs/proxy/config_settings.md
View file

@ -360,7 +360,7 @@ router_settings:
| redis_url | str | URL for Redis server. **Known performance issue with Redis URL.** |
| cache_responses | boolean | Flag to enable caching LLM Responses, if cache set under `router_settings`. If true, caches responses. Defaults to False. |
| router_general_settings | RouterGeneralSettings | [SDK-Only] Router general settings - contains optimizations like 'async_only_mode'. [Docs](../routing.md#router-general-settings) |
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `deployment_affinity`, `forward_client_headers_by_model_group` |
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `encrypted_content_affinity`, `deployment_affinity`, `session_affinity`, `forward_client_headers_by_model_group` |
| deployment_affinity_ttl_seconds | int | TTL (seconds) for user-key → deployment affinity mapping when `deployment_affinity` is enabled (configured at Router init / proxy startup). Defaults to `3600` (1 hour). |
| ignore_invalid_deployments | boolean | If true, ignores invalid deployments. Default for proxy is True - to prevent invalid models from blocking other models from being loaded. |
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search.md) |

41
docs/my-website/docs/proxy/guardrails/quick_start.md
View file

@ -73,6 +73,7 @@ guardrails:
plr_scanners: true
```
For generic guardrail APIs you can also set **static headers** (`headers`: key/value sent on every request) and **dynamic headers** (`extra_headers`: list of client header names to forward). See [Generic Guardrail API - Static and dynamic headers](/docs/adding_provider/generic_guardrail_api#static-and-dynamic-headers).
### Supported values for `mode` (Event Hooks)
@ -498,6 +499,11 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
Run guardrails based on the user-agent header. This is useful for running pre-call checks on OpenWebUI but only masking in logs for Claude CLI.
`default` can be a single mode string or a list of modes.
<Tabs>
<TabItem value="single" label="Single Default Mode">
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -518,6 +524,32 @@ guardrails:
default_on: true # run on every request
```
</TabItem>
<TabItem value="multi" label="Multiple Default Modes">
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "guardrails_ai-guard"
litellm_params:
guardrail: guardrails_ai
guard_name: "pii_detect"
mode:
tags:
"User-Agent: claude-cli": "logging_only"
default: ["pre_call", "post_call"] # Run on both pre and post call when no tags match
api_base: os.environ/GUARDRAILS_AI_API_BASE
default_on: true
```
</TabItem>
</Tabs>
### ✨ Model-level Guardrails
@ -639,13 +671,22 @@ guardrails:
Mode Specification
`default` accepts either a single string or a list of strings.
```python
from litellm.types.guardrails import Mode
# Single default mode
mode = Mode(
tags={"User-Agent: claude-cli": "logging_only"},
default="logging_only"
)
# Multiple default modes
mode = Mode(
tags={"User-Agent: claude-cli": "logging_only"},
default=["pre_call", "post_call"]
)
```
### `guardrails` Request Parameter

137
docs/my-website/docs/proxy/guardrails/team_based_guardrails.md Normal file
View file

@ -0,0 +1,137 @@
import Image from '@theme/IdealImage';
# Team-Based Guardrails
Team-based guardrails let **developers** register a guardrail for their team via the API; an **admin** then reviews and approves or rejects it in the LiteLLM UI. Only [Generic Guardrail API](/docs/adding_provider/generic_guardrail_api) guardrails can be registered this way.
## Overview
- **Developer flow:** Use a **team-scoped API key** to `POST /guardrails/register` with your guardrail config. The submission is stored with status `pending_review`.
- **Admin flow:** In the proxy UI, open **Guardrails → Team Guardrails**, review pending submissions, and **Approve** or **Reject**. Approved guardrails become active and are initialized in memory.
---
## Developer flow: Register a guardrail
### Prerequisites
- A **team-scoped** API key (the key must be associated with a team). Keys without a team cannot register guardrails.
- Your guardrail must follow the [Generic Guardrail API](/docs/adding_provider/generic_guardrail_api) contract and config.
### Request
**Endpoint:** `POST /guardrails/register`
**Headers:** `Authorization: Bearer <team_scoped_api_key>`
**Body:** JSON matching the Generic Guardrail API config.
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `guardrail_name` | string | Yes | Unique name for the guardrail. |
| `litellm_params` | object | Yes | Must include `guardrail: "generic_guardrail_api"`, `mode` (e.g. `pre_call`, `post_call`), and `api_base`. See [Generic Guardrail API](/docs/adding_provider/generic_guardrail_api#litellm-configuration). |
| `guardrail_info` | object | No | Optional metadata (e.g. `description`). |
### Requirements for `litellm_params`
- `guardrail` must be exactly `"generic_guardrail_api"`.
- `api_base` is required (your guardrail API base URL).
- `mode` is required (e.g. `pre_call`, `post_call`, `during_call`).
### Example
```bash
curl -X POST "http://localhost:4000/guardrails/register" \
-H "Authorization: Bearer <your_team_scoped_api_key>" \
-H "Content-Type: application/json" \
-d '{
"guardrail_name": "my-team-guard",
"litellm_params": {
"guardrail": "generic_guardrail_api",
"mode": "pre_call",
"api_base": "https://your-guardrail-api.com",
"api_key": "optional-api-key",
"unreachable_fallback": "fail_closed",
"forward_api_key": true
},
"guardrail_info": {
"description": "Team content moderation guardrail"
}
}'
```
### Example response
```json
{
"guardrail_id": "123e4567-e89b-12d3-a456-426614174000",
"guardrail_name": "my-team-guard",
"status": "pending_review",
"submitted_at": "2025-02-28T12:00:00.000Z"
}
```
### Errors
- **400** – Missing or invalid body (e.g. `guardrail` not `generic_guardrail_api`, missing `api_base` or `mode`), or a guardrail with the same `guardrail_name` already exists.
- **400** – "Registration requires an API key associated with a team. Use a team-scoped key." → Use an API key that has a team.
- **500** – Server/database error.
After a successful register, the guardrail stays in `pending_review` until an admin approves or rejects it.
---
## Admin flow: Approve or reject in the UI
Admins review and approve or reject team guardrail submissions in the LiteLLM proxy UI.
### 1. Open the Guardrails page
In the proxy dashboard, go to **Guardrails** (sidebar or navigation).
### 2. Open the Team Guardrails tab
Switch to the **Team Guardrails** tab. This tab lists all team-submitted guardrails and their status.
<Image img={require('../../../img/admin_team_guardrails.png')} alt="Team Guardrails admin view: status summary (Total, Pending Review, Active, Rejected), guardrail list with Pending Review tag, and detail panel with Approve/Reject buttons and configuration options." style={{ width: '100%', maxWidth: '900px', height: 'auto' }} />
### 3. Review submissions
The table shows:
- **Name**, **Team**, **Endpoint** (api_base), **Status** (Pending Review / Active / Rejected), **Submitted** date, **Submitted by** (user/email), and other config details.
Summary cards show counts for **Total**, **Pending Review**, **Active**, and **Rejected**.
<!-- Optional: screenshot of the Team Guardrails table and summary -->
### 4. Approve or reject
- **Pending Review:** Use **Approve** to activate the guardrail. The proxy sets its status to `active` and initializes it in memory so it can be used on requests.
- Use **Reject** to decline the submission (status becomes `rejected`).
Approval triggers the same initialization as adding a guardrail via config or the admin guardrail API; rejection only updates the status and does not load the guardrail.
<!-- Optional: screenshot of Approve/Reject actions or confirmation dialog -->
### API equivalent (admin only)
Admins can also use the REST API:
- **List submissions:** `GET /guardrails/submissions` (optional query: `status`, `team_id`, `search`)
- **Get one:** `GET /guardrails/submissions/{guardrail_id}`
- **Approve:** `POST /guardrails/submissions/{guardrail_id}/approve`
- **Reject:** `POST /guardrails/submissions/{guardrail_id}/reject`
These endpoints require **admin** (e.g. `PROXY_ADMIN`) authentication.
---
## Summary
| Role | Action |
|------|--------|
| **Developer** | Call `POST /guardrails/register` with a team-scoped key and a `generic_guardrail_api` config. Submission enters `pending_review`. |
| **Admin** | Open **Guardrails → Team Guardrails** in the UI (or use the submissions API), then **Approve** or **Reject** each submission. Approved guardrails become active. |
Only guardrails with `litellm_params.guardrail: "generic_guardrail_api"` are accepted for registration. For the full contract and config options, see [Generic Guardrail API](/docs/adding_provider/generic_guardrail_api).

33
docs/my-website/docs/proxy/load_balancing.md
View file

@ -347,3 +347,36 @@ If `order=1` deployment is unavailable (e.g., rate-limited), the router falls ba
- **Higher throughput**: More requests handled simultaneously across deployments
- **Improved reliability**: If one deployment fails, traffic automatically routes to healthy ones
- **Better resource utilization**: Load spread evenly across all available deployments
## Special Considerations for Responses API
When load balancing OpenAI's Responses API across deployments with **different API keys** (e.g., different Azure regions or organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the originating API key.
**Solution:** Use the `encrypted_content_affinity` pre-call check to automatically route follow-up requests containing encrypted items to the correct deployment:
```yaml
model_list:
- model_name: gpt-5.1-codex
litellm_params:
model: azure/gpt-5.1-codex
api_base: https://eastus.openai.azure.com/
api_key: os.environ/AZURE_API_KEY_EASTUS
model_info:
id: "deployment-eastus"
- model_name: gpt-5.1-codex
litellm_params:
model: azure/gpt-5.1-codex
api_base: https://westeurope.openai.azure.com/
api_key: os.environ/AZURE_API_KEY_WESTEUROPE
model_info:
id: "deployment-westeurope"
router_settings:
optional_pre_call_checks:
- encrypted_content_affinity # 👈 Prevents invalid_encrypted_content errors
```
This ensures requests containing encrypted content are routed to the deployment that created them, while other requests continue to load balance normally.
**[Learn more about Encrypted Content Affinity →](../response_api.md#encrypted-content-affinity-multi-region-load-balancing)**

141
docs/my-website/docs/response_api.md
View file

@ -920,9 +920,14 @@ follow_up = await router.aresponses(
To enable session continuity for Responses API in your LiteLLM proxy, set `optional_pre_call_checks` in your proxy config.yaml.
- `responses_api_deployment_check`: high priority routing when `previous_response_id` is provided
- `encrypted_content_affinity`: **[Recommended]** content-aware routing for encrypted items (e.g., `rs_...` reasoning items)
- `session_affinity`: sticky sessions based on session id (takes priority over `deployment_affinity`)
- `deployment_affinity`: sticky sessions based on user key (applies even without `previous_response_id`)
:::tip Recommended: Use `encrypted_content_affinity`
For Responses API with load balancing across deployments with **different API keys**, use `encrypted_content_affinity` instead of `deployment_affinity`. It only pins requests that contain encrypted content, avoiding quota reduction while preventing `invalid_encrypted_content` errors.
:::
Notes:
- User-key affinity is keyed on `metadata.user_api_key_hash` (the API key hash). The OpenAI `user` request parameter is an end-user identifier and is intentionally not used for deployment affinity.
- Session-ID affinity is keyed on `metadata.session_id`. For proxy requests, this can be passed via the `x-litellm-session-id` or `x-litellm-trace-id` HTTP header (they are interchangeable for call chaining). For Python SDK requests, you can pass it via `litellm_metadata={"session_id": "value"}` in request args.
@ -983,6 +988,142 @@ follow_up = client.responses.create(
</TabItem>
</Tabs>
## Encrypted Content Affinity (Multi-Region Load Balancing)
When load balancing Responses API across deployments with **different API keys** (e.g., different Azure regions or OpenAI organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the API key that created them.
### The Problem
```json
{
"error": {
"message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.",
"type": "invalid_request_error",
"code": "invalid_encrypted_content"
}
}
```
This error occurs when:
1. Initial request goes to Deployment A (API Key 1) → produces encrypted item `rs_xyz`
2. Follow-up request with `rs_xyz` in input gets load balanced to Deployment B (API Key 2)
3. Deployment B cannot decrypt content created by Deployment A → **request fails**
### The Solution: `encrypted_content_affinity`
The `encrypted_content_affinity` pre-call check routes follow-up requests containing encrypted items to the originating deployment **only when necessary**
**Key Benefits:**
- ✅ **No quota reduction**: Unlike `deployment_affinity`, only pins requests that contain encrypted items
- ✅ **Bypasses rate limits**: When encrypted content requires a specific deployment, RPM/TPM limits are bypassed (the request would fail on any other deployment anyway)
- ✅ **No `previous_response_id` required**: Works by encoding `model_id` directly into item IDs
- ✅ **No cache required**: `model_id` is decoded on-the-fly — no Redis dependency, no TTL to manage
- ✅ **Globally safe**: Can be enabled for all models; non-Responses-API calls (chat, embeddings) are unaffected
### How It Works
1. **Encoding Phase** (on response):
- For each output item that contains `encrypted_content`, LiteLLM rewrites the item ID to embed the originating `model_id`: `rs_xyz` → `encitem_{base64("litellm:model_id:{model_id};item_id:rs_xyz")}`
- The original item ID is restored before forwarding the request to the upstream provider
2. **Routing Phase** (before request):
- Scans request `input` for `encitem_` prefixed IDs
- If found → decodes `model_id`, pins to originating deployment, bypasses rate limits
- If no encoded items → normal load balancing
### Configuration
<Tabs>
<TabItem value="sdk" label="Python SDK">
```python
from litellm import Router
router = Router(
model_list=[
{
"model_name": "gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "org-1-api-key", # Different API key
},
"model_info": {"id": "deployment-us-east"},
},
{
"model_name": "gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "org-2-api-key", # Different API key
},
"model_info": {"id": "deployment-eu-west"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
)
# Initial request - routes to any deployment
response1 = await router.aresponses(
model="gpt-5.1-codex",
input="Explain quantum computing",
)
# Follow-up with encrypted items - automatically routes to same deployment
response2 = await router.aresponses(
model="gpt-5.1-codex",
input=response1.output, # Contains encrypted items from response1
)
```
</TabItem>
<TabItem value="proxy" label="Proxy Server">
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gpt-5.1-codex
litellm_params:
model: azure/gpt-5.1-codex
api_base: https://eastus.openai.azure.com/
api_key: os.environ/AZURE_API_KEY_EASTUS
rpm: 600
tpm: 100000
model_info:
id: "gpt-5.1-codex-eastus"
- model_name: gpt-5.1-codex
litellm_params:
model: azure/gpt-5.1-codex
api_base: https://westeurope.openai.azure.com/
api_key: os.environ/AZURE_API_KEY_WESTEUROPE
rpm: 600
tpm: 100000
model_info:
id: "gpt-5.1-codex-westeurope"
router_settings:
routing_strategy: usage-based-routing-v2
enable_pre_call_checks: true
optional_pre_call_checks:
- encrypted_content_affinity
```
**Start proxy:**
```bash
litellm --config config.yaml
```
</TabItem>
</Tabs>
### When to Use Each Affinity Type
| Affinity Type | Use Case | Scope | Quota Impact |
|---------------|----------|-------|--------------|
| **`encrypted_content_affinity`** | **[Recommended]** Multi-region Responses API with different API keys | Only requests with tracked encrypted items | ✅ None (surgical pinning) |
| `responses_api_deployment_check` | When `previous_response_id` is available | Requests with `previous_response_id` | ✅ None |
| `session_affinity` | Session-based applications | All requests with same `session_id` | ⚠️ Reduces quota by # of sessions |
| `deployment_affinity` | Simple sticky sessions | All requests from same API key | ❌ Reduces quota by # of users |
## Calling non-Responses API endpoints (`/responses` to `/chat/completions` Bridge)
LiteLLM allows you to call non-Responses API models via a bridge to LiteLLM's `/chat/completions` endpoint. This is useful for calling Anthropic, Gemini and even non-Responses API OpenAI models.

BIN
docs/my-website/img/admin_team_guardrails.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 523 KiB

1
docs/my-website/sidebars.js
View file

@ -42,6 +42,7 @@ const sidebars = {
label: "Guardrails",
items: [
"proxy/guardrails/quick_start",
"proxy/guardrails/team_based_guardrails",
"proxy/guardrails/guardrail_load_balancing",
"proxy/guardrails/test_playground",
"proxy/guardrails/litellm_content_filter",

37
enterprise/litellm_enterprise/integrations/custom_guardrail.py
View file

@ -10,10 +10,15 @@ class EnterpriseCustomGuardrailHelper:
event_hook: Optional[
Union[GuardrailEventHooks, List[GuardrailEventHooks], Mode]
],
event_type: Optional[GuardrailEventHooks] = None,
) -> Optional[bool]:
"""
Assumes check for event match is done in `should_run_guardrail`
Returns True if the guardrail should be run by tag
Returns True if the guardrail should be run for this request and event_type.
Logic:
- If a request tag matches a Mode tag key, only run if event_type matches
the tag's value (the mode for that tag).
- If no request tag matches, fall back to default mode(s).
"""
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
@ -36,11 +41,29 @@ class EnterpriseCustomGuardrailHelper:
proxy_server_request=proxy_server_request,
)
if request_tags and any(tag in event_hook.tags for tag in request_tags):
return True
elif event_hook.default and any(
tag in event_hook.default for tag in request_tags
):
# Check if any request tag matches a Mode tag key
matched_mode = None
if request_tags:
for tag in request_tags:
if tag in event_hook.tags:
matched_mode = event_hook.tags[tag]
break
if matched_mode is not None:
# Tag matched: only run if event_type matches the tag's mode value
if event_type is not None:
return event_type.value == matched_mode
return True
# No tag matched: fall back to default mode(s)
if event_hook.default is not None:
if event_type is not None:
default_list = (
event_hook.default
if isinstance(event_hook.default, list)
else [event_hook.default]
)
return event_type.value in default_list
return False
return False

53
enterprise/litellm_enterprise/proxy/audit_logging_endpoints.py
View file

@ -1,13 +1,13 @@
"""
AUDIT LOGGING
All /audit logging endpoints. Attempting to write these as CRUD endpoints.
All /audit logging endpoints. Attempting to write these as CRUD endpoints.
GET - /audit/{id} - Get audit log by id
GET - /audit - Get all audit logs
"""
from typing import Any, Dict, Optional
from typing import Any, Dict, List, Optional
#### AUDIT LOGGING ####
from fastapi import APIRouter, Depends, HTTPException, Query
@ -22,6 +22,27 @@ from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
router = APIRouter()
def _build_json_field_or_condition(json_key: str, value: str) -> Dict[str, Any]:
"""
Build an OR condition that matches a value inside a JSON column at the
given key, checking both before_value and updated_values.
Uses Prisma's JSON path filtering (PostgreSQL only).
Example result (team_id="t1"):
{"OR": [
{"before_value": {"path": ["team_id"], "string_contains": "t1"}},
{"updated_values": {"path": ["team_id"], "string_contains": "t1"}},
]}
"""
return {
"OR": [
{"before_value": {"path": [json_key], "string_contains": value}},
{"updated_values": {"path": [json_key], "string_contains": value}},
]
}
@router.get(
"/audit",
tags=["Audit Logging"],
@ -49,6 +70,14 @@ async def get_audit_logs(
),
start_date: Optional[str] = Query(None, description="Filter logs after this date"),
end_date: Optional[str] = Query(None, description="Filter logs before this date"),
object_team_id: Optional[str] = Query(
None,
description="Filter by team_id present in before_value or updated_values JSON (PostgreSQL only)",
),
object_key_hash: Optional[str] = Query(
None,
description="Filter by token (key hash) present in before_value or updated_values JSON (PostgreSQL only)",
),
# Sorting parameters
sort_by: Optional[str] = Query(
None,
@ -60,6 +89,9 @@ async def get_audit_logs(
Get all audit logs with filtering and pagination.
Returns a paginated response of audit logs matching the specified filters.
Note: object_team_id and object_key_hash use Prisma JSON path filtering,
which requires PostgreSQL.
"""
from litellm.proxy.proxy_server import prisma_client
@ -82,18 +114,29 @@ async def get_audit_logs(
if object_id:
where_conditions["object_id"] = object_id
if start_date or end_date:
date_filter = {}
date_filter: Dict[str, Any] = {}
if start_date:
date_filter["gte"] = start_date
if end_date:
date_filter["lte"] = end_date
where_conditions["updated_at"] = date_filter
# JSON field filters (PostgreSQL only) — each filter is AND'd with the
# others, but checks both before_value and updated_values internally (OR).
if object_team_id:
where_conditions["AND"] = where_conditions.get("AND", []) + [
_build_json_field_or_condition("team_id", object_team_id)
]
if object_key_hash:
where_conditions["AND"] = where_conditions.get("AND", []) + [
_build_json_field_or_condition("token", object_key_hash)
]
# Build sort conditions
order_by = {}
order_by: Dict[str, Any] = {}
if sort_by and isinstance(sort_by, str):
order_by[sort_by] = sort_order
elif sort_order and isinstance(sort_order, str):
else:
order_by["updated_at"] = sort_order # Default sort by updated_at
# Get paginated results

7
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -589,7 +589,14 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
model_file_id_mapping = cast(
Optional[Dict[str, Dict[str, str]]], kwargs.get("model_file_id_mapping")
)
# model_info may be at top-level or nested under litellm_metadata
# (batch/file operations use litellm_metadata)
model_id = cast(Optional[str], kwargs.get("model_info", {}).get("id", None))
if model_id is None:
model_id = cast(
Optional[str],
kwargs.get("litellm_metadata", {}).get("model_info", {}).get("id", None),
)
mapped_file_id: Optional[str] = None
if input_file_id and model_file_id_mapping and model_id:
mapped_file_id = model_file_id_mapping.get(input_file_id, {}).get(

8
litellm-proxy-extras/litellm_proxy_extras/migrations/20260228170127_support_team_based_guardrails/migration.sql Normal file
View file

@ -0,0 +1,8 @@
-- AlterTable
ALTER TABLE "LiteLLM_GuardrailsTable" ADD COLUMN "reviewed_at" TIMESTAMP(3),
ADD COLUMN "status" TEXT NOT NULL DEFAULT 'active',
ADD COLUMN "submitted_at" TIMESTAMP(3);
-- CreateIndex
CREATE INDEX "LiteLLM_GuardrailsTable_status_idx" ON "LiteLLM_GuardrailsTable"("status");

7
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -871,6 +871,13 @@ model LiteLLM_GuardrailsTable {
team_id String?
created_at DateTime @default(now())
updated_at DateTime @updatedAt
// Submission lifecycle. Possible values: pending_review (team-registered, awaiting approval), active (approved), rejected
status String @default("active")
submitted_at DateTime?
reviewed_at DateTime?
// submitted_by_user_id and submitted_by_email live in guardrail_info JSON
@@index([status])
}
// Daily guardrail metrics for usage dashboard (one row per guardrail per day)

1
litellm/__init__.py
View file

@ -1441,6 +1441,7 @@ if TYPE_CHECKING:
from .llms.manus.responses.transformation import ManusResponsesAPIConfig as ManusResponsesAPIConfig
from .llms.perplexity.responses.transformation import PerplexityResponsesConfig as PerplexityResponsesConfig
from .llms.databricks.responses.transformation import DatabricksResponsesAPIConfig as DatabricksResponsesAPIConfig
from .llms.openrouter.responses.transformation import OpenRouterResponsesAPIConfig as OpenRouterResponsesAPIConfig
from .llms.gemini.interactions.transformation import GoogleAIStudioInteractionsConfig as GoogleAIStudioInteractionsConfig
from .llms.openai.chat.o_series_transformation import OpenAIOSeriesConfig as OpenAIOSeriesConfig, OpenAIOSeriesConfig as OpenAIO1Config
from .llms.anthropic.skills.transformation import AnthropicSkillsConfig as AnthropicSkillsConfig

5
litellm/_lazy_imports_registry.py
View file

@ -231,6 +231,7 @@ LLM_CONFIG_NAMES = (
"VolcEngineResponsesAPIConfig",
"PerplexityResponsesConfig",
"DatabricksResponsesAPIConfig",
"OpenRouterResponsesAPIConfig",
"GoogleAIStudioInteractionsConfig",
"OpenAIOSeriesConfig",
"AnthropicSkillsConfig",
@ -923,6 +924,10 @@ _LLM_CONFIGS_IMPORT_MAP = {
".llms.databricks.responses.transformation",
"DatabricksResponsesAPIConfig",
),
"OpenRouterResponsesAPIConfig": (
".llms.openrouter.responses.transformation",
"OpenRouterResponsesAPIConfig",
),
"GoogleAIStudioInteractionsConfig": (
".llms.gemini.interactions.transformation",
"GoogleAIStudioInteractionsConfig",

101
litellm/batches/batch_utils.py
View file

@ -128,73 +128,58 @@ def calculate_vertex_ai_batch_cost_and_usage(
model_name: Optional[str] = None,
) -> Tuple[float, Usage]:
"""
Calculate both cost and usage from Vertex AI batch responses
Calculate both cost and usage from Vertex AI batch responses.
Vertex AI batch output lines have format:
{"request": ..., "status": "", "response": {"candidates": [...], "usageMetadata": {...}}}
usageMetadata contains promptTokenCount, candidatesTokenCount, totalTokenCount.
"""
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
from litellm.cost_calculator import batch_cost_calculator
total_cost = 0.0
total_tokens = 0
prompt_tokens = 0
completion_tokens = 0
for response in vertex_ai_batch_responses:
if response.get("status") == "JOB_STATE_SUCCEEDED": # Check if response was successful
# Transform Vertex AI response to OpenAI format if needed
actual_model_name = model_name or "gemini-2.0-flash-001"
# Create required arguments for the transformation method
model_response = ModelResponse()
# Ensure model_name is not None
actual_model_name = model_name or "gemini-2.5-flash"
# Create a real LiteLLM logging object
logging_obj = Logging(
for response in vertex_ai_batch_responses:
response_body = response.get("response")
if response_body is None:
continue
usage_metadata = response_body.get("usageMetadata", {})
_prompt = usage_metadata.get("promptTokenCount", 0) or 0
_completion = usage_metadata.get("candidatesTokenCount", 0) or 0
_total = usage_metadata.get("totalTokenCount", 0) or (_prompt + _completion)
line_usage = Usage(
prompt_tokens=_prompt,
completion_tokens=_completion,
total_tokens=_total,
)
try:
p_cost, c_cost = batch_cost_calculator(
usage=line_usage,
model=actual_model_name,
messages=[{"role": "user", "content": "batch_request"}],
stream=False,
call_type=CallTypes.aretrieve_batch,
start_time=time.time(),
litellm_call_id="batch_" + str(uuid.uuid4()),
function_id="batch_processing",
litellm_trace_id=str(uuid.uuid4()),
kwargs={"optional_params": {}}
)
# Add the optional_params attribute that the Vertex AI transformation expects
logging_obj.optional_params = {}
raw_response = httpx.Response(200) # Mock response object
openai_format_response = VertexGeminiConfig()._transform_google_generate_content_to_openai_model_response(
completion_response=response["response"],
model_response=model_response,
model=actual_model_name,
logging_obj=logging_obj,
raw_response=raw_response,
)
# Calculate cost using existing function
cost = litellm.completion_cost(
completion_response=openai_format_response,
custom_llm_provider="vertex_ai",
call_type=CallTypes.aretrieve_batch.value,
)
total_cost += cost
# Extract usage from the transformed response
usage_obj = getattr(openai_format_response, 'usage', None)
if usage_obj:
usage = usage_obj
else:
# Fallback: create usage from response dict
response_dict = openai_format_response.dict() if hasattr(openai_format_response, 'dict') else {}
usage = _get_batch_job_usage_from_response_body(response_dict)
total_tokens += usage.total_tokens
prompt_tokens += usage.prompt_tokens
completion_tokens += usage.completion_tokens
total_cost += p_cost + c_cost
except Exception as e:
verbose_logger.debug(
"vertex_ai batch cost calculation error for line: %s", str(e)
)
prompt_tokens += _prompt
completion_tokens += _completion
total_tokens += _total
verbose_logger.info(
"vertex_ai batch cost: cost=%s, prompt=%d, completion=%d, total=%d",
total_cost, prompt_tokens, completion_tokens, total_tokens,
)
return total_cost, Usage(
total_tokens=total_tokens,
prompt_tokens=prompt_tokens,

7
litellm/batches/main.py
View file

@ -112,6 +112,7 @@ async def acreate_batch(
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
output_expires_after: Optional[Dict[str, Any]] = None,
**kwargs,
) -> LiteLLMBatch:
"""
@ -133,6 +134,7 @@ async def acreate_batch(
metadata,
extra_headers,
extra_body,
output_expires_after,
**kwargs,
)
@ -152,7 +154,7 @@ async def acreate_batch(
@client
def create_batch(
def create_batch( # noqa: PLR0915
completion_window: Literal["24h"],
endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"],
input_file_id: str,
@ -160,6 +162,7 @@ def create_batch(
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
output_expires_after: Optional[Dict[str, Any]] = None,
**kwargs,
) -> Union[LiteLLMBatch, Coroutine[Any, Any, LiteLLMBatch]]:
"""
@ -215,6 +218,8 @@ def create_batch(
extra_headers=extra_headers,
extra_body=extra_body,
)
if output_expires_after is not None:
_create_batch_request["output_expires_after"] = output_expires_after
if model is not None:
provider_config = ProviderConfigManager.get_provider_batches_config(
model=model,

2
litellm/files/main.py
View file

@ -295,7 +295,7 @@ def create_file(
@client
async def afile_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure", "gemini", "hosted_vllm", "manus"] = "openai",
custom_llm_provider: Literal["openai", "azure", "gemini", "vertex_ai", "hosted_vllm", "manus"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,

55
litellm/fine_tuning/main.py
View file

@ -34,6 +34,44 @@ vertex_fine_tuning_apis_instance = VertexFineTuningAPI()
#################################################
def _prepare_azure_extra_body(
extra_body: Optional[Dict[str, Any]],
kwargs: Dict[str, Any],
azure_specific_hyperparams: Dict[str, Any],
) -> Dict[str, Any]:
"""
Prepare extra_body for Azure fine-tuning API by combining Azure-specific parameters.
Azure fine-tuning API accepts additional parameters beyond the standard OpenAI spec:
- trainingType: Type of training (e.g., 1 for supervised fine-tuning)
- prompt_loss_weight: Weight for prompt loss in training
These parameters must be passed in the extra_body field when calling the Azure OpenAI SDK.
Args:
extra_body: Optional existing extra_body dict
kwargs: Request kwargs that may contain Azure-specific parameters
azure_specific_hyperparams: Dict of Azure-specific hyperparameters already extracted
Returns:
Dict containing all Azure-specific parameters to be passed in extra_body
"""
if extra_body is None:
extra_body = {}
# Azure-specific root-level parameters
azure_specific_params = ["trainingType"]
for param in azure_specific_params:
if param in kwargs:
extra_body[param] = kwargs[param]
# Add Azure-specific hyperparameters
if azure_specific_hyperparams:
extra_body.update(azure_specific_hyperparams)
return extra_body
@client
async def acreate_fine_tuning_job(
model: str,
@ -114,6 +152,15 @@ def create_fine_tuning_job(
# handle hyperparameters
hyperparameters = hyperparameters or {} # original hyperparameters
# For Azure, extract Azure-specific hyperparameters before creating OpenAI-spec hyperparameters
azure_specific_hyperparams = {}
if custom_llm_provider == "azure":
azure_hyperparameter_keys = ["prompt_loss_weight"]
for key in azure_hyperparameter_keys:
if key in hyperparameters:
azure_specific_hyperparams[key] = hyperparameters.pop(key)
_oai_hyperparameters: Hyperparameters = Hyperparameters(
**hyperparameters
) # Typed Hyperparameters for OpenAI Spec
@ -207,6 +254,10 @@ def create_fine_tuning_job(
extra_body.pop("azure_ad_token", None)
else:
get_secret_str("AZURE_AD_TOKEN") # type: ignore
# Prepare Azure-specific parameters for extra_body
extra_body = _prepare_azure_extra_body(extra_body, kwargs, azure_specific_hyperparams)
create_fine_tuning_job_data = FineTuningJobCreate(
model=model,
training_file=training_file,
@ -220,6 +271,10 @@ def create_fine_tuning_job(
create_fine_tuning_job_data_dict = create_fine_tuning_job_data.model_dump(
exclude_none=True
)
# Add extra_body if it has Azure-specific parameters
if extra_body:
create_fine_tuning_job_data_dict["extra_body"] = extra_body
response = azure_fine_tuning_apis_instance.create_fine_tuning_job(
api_base=api_base,

22
litellm/integrations/custom_guardrail.py
View file

@ -235,8 +235,13 @@ class CustomGuardrail(CustomLogger):
list(event_hook.tags.values()), supported_event_hooks
)
if event_hook.default:
default_list = (
event_hook.default
if isinstance(event_hook.default, list)
else [event_hook.default]
)
_validate_event_hook_list_is_in_supported_event_hooks(
[event_hook.default], supported_event_hooks
default_list, supported_event_hooks
)
elif isinstance(event_hook, GuardrailEventHooks):
if event_hook not in supported_event_hooks:
@ -415,7 +420,7 @@ class CustomGuardrail(CustomLogger):
"Setting tag-based guardrails is only available in litellm-enterprise. You must be a premium user to use this feature."
)
result = EnterpriseCustomGuardrailHelper._should_run_if_mode_by_tag(
data, self.event_hook
data, self.event_hook, event_type
)
if result is not None:
return result
@ -442,7 +447,7 @@ class CustomGuardrail(CustomLogger):
"Setting tag-based guardrails is only available in litellm-enterprise. You must be a premium user to use this feature."
)
result = EnterpriseCustomGuardrailHelper._should_run_if_mode_by_tag(
data, self.event_hook
data, self.event_hook, event_type
)
if result is not None:
return result
@ -461,7 +466,16 @@ class CustomGuardrail(CustomLogger):
if isinstance(self.event_hook, list):
return event_type.value in self.event_hook
if isinstance(self.event_hook, Mode):
return event_type.value in self.event_hook.tags.values()
if event_type.value in self.event_hook.tags.values():
return True
if self.event_hook.default:
default_list = (
self.event_hook.default
if isinstance(self.event_hook.default, list)
else [self.event_hook.default]
)
return event_type.value in default_list
return False
return self.event_hook == event_type.value
def get_guardrail_dynamic_request_body_params(self, request_data: dict) -> dict:

45
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -1,9 +1,9 @@
import json
import re
import traceback
from typing import Any, Optional
import httpx
import re
import litellm
from litellm._logging import verbose_logger
@ -443,6 +443,27 @@ def exception_type( # type: ignore # noqa: PLR0915
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
)
elif "invalid_encrypted_content" in error_str or "could not be verified" in error_str:
exception_mapping_worked = True
helpful_message = (
f"{exception_provider} - {message}\n\n"
" This error occurs when load balancing Responses API across deployments with different API keys.\n"
" Encrypted content is tied to the organization that created it and cannot be decrypted by other organizations.\n\n"
" Solution: Enable 'encrypted_content_affinity' to route follow-up requests to the correct deployment:\n\n"
" router_settings:\n"
" enable_pre_call_checks: true\n"
" optional_pre_call_checks:\n"
" - encrypted_content_affinity\n\n"
" Learn more: https://docs.litellm.ai/docs/response_api#encrypted-content-affinity-multi-region-load-balancing"
)
raise BadRequestError(
message=helpful_message,
llm_provider=custom_llm_provider,
model=model,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
body=getattr(original_exception, "body", None),
)
elif (
"invalid_request_error" in error_str
and "Incorrect API key provided" not in error_str
@ -2126,7 +2147,27 @@ def exception_type( # type: ignore # noqa: PLR0915
extra_information=extra_information,
original_exception=original_exception,
)
elif azure_error_code == "invalid_encrypted_content" or "could not be verified" in error_str:
exception_mapping_worked = True
helpful_message = (
f"AzureException - {message}\n\n"
"This error occurs when load balancing Responses API across deployments with different API keys.\n"
" Encrypted content is tied to the organization that created it and cannot be decrypted by other organizations.\n\n"
" Solution: Enable 'encrypted_content_affinity' to route follow-up requests to the correct deployment:\n\n"
" router_settings:\n"
" enable_pre_call_checks: true\n"
" optional_pre_call_checks:\n"
" - encrypted_content_affinity\n\n"
" Learn more: https://docs.litellm.ai/docs/response_api#encrypted-content-affinity-multi-region-load-balancing"
)
raise BadRequestError(
message=helpful_message,
llm_provider="azure",
model=model,
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
body=getattr(original_exception, "body", None),
)
elif "invalid_request_error" in error_str:
exception_mapping_worked = True
raise BadRequestError(

136
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
View file

@ -41,7 +41,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
type="text",
text="",
)
pending_new_content_block: bool = False
chunk_queue: deque = deque() # Queue for buffering multiple chunks
def __init__(
@ -80,38 +79,40 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
from .transformation import LiteLLMAnthropicMessagesAdapter
try:
# Always return queued chunks first
if self.chunk_queue:
return self.chunk_queue.popleft()
# Queue initial chunks if not sent yet
if self.sent_first_chunk is False:
self.sent_first_chunk = True
return {
"type": "message_start",
"message": {
"id": "msg_{}".format(uuid.uuid4()),
"type": "message",
"role": "assistant",
"content": [],
"model": self.model,
"stop_reason": None,
"stop_sequence": None,
"usage": self._create_initial_usage_delta(),
},
}
self.chunk_queue.append(
{
"type": "message_start",
"message": {
"id": "msg_{}".format(uuid.uuid4()),
"type": "message",
"role": "assistant",
"content": [],
"model": self.model,
"stop_reason": None,
"stop_sequence": None,
"usage": self._create_initial_usage_delta(),
},
}
)
return self.chunk_queue.popleft()
if self.sent_content_block_start is False:
self.sent_content_block_start = True
return {
"type": "content_block_start",
"index": self.current_content_block_index,
"content_block": {"type": "text", "text": ""},
}
# Handle pending new content block start
if self.pending_new_content_block:
self.pending_new_content_block = False
self.sent_content_block_finish = False # Reset for new block
return {
"type": "content_block_start",
"index": self.current_content_block_index,
"content_block": self.current_content_block_start,
}
self.chunk_queue.append(
{
"type": "content_block_start",
"index": self.current_content_block_index,
"content_block": {"type": "text", "text": ""},
}
)
return self.chunk_queue.popleft()
for chunk in self.completion_stream:
if chunk == "None" or chunk is None:
@ -126,45 +127,65 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
current_content_block_index=self.current_content_block_index,
)
# Check if we need to start a new content block
# This is where you'd add your logic to detect when a new content block should start
# For example, if the chunk indicates a tool call or different content type
if should_start_new_block and not self.sent_content_block_finish:
# End current content block and prepare for new one
self.holding_chunk = processed_chunk
self.sent_content_block_finish = True
self.pending_new_content_block = True
return {
"type": "content_block_stop",
"index": max(self.current_content_block_index - 1, 0),
}
# Queue the sequence: content_block_stop -> content_block_start
# The trigger chunk itself is not emitted as a delta since the
# content_block_start already carries the relevant information.
self.chunk_queue.append(
{
"type": "content_block_stop",
"index": max(self.current_content_block_index - 1, 0),
}
)
self.chunk_queue.append(
{
"type": "content_block_start",
"index": self.current_content_block_index,
"content_block": self.current_content_block_start,
}
)
self.sent_content_block_finish = False
return self.chunk_queue.popleft()
if (
processed_chunk["type"] == "message_delta"
and self.sent_content_block_finish is False
):
self.holding_chunk = processed_chunk
# Queue both the content_block_stop and the message_delta
self.chunk_queue.append(
{
"type": "content_block_stop",
"index": self.current_content_block_index,
}
)
self.sent_content_block_finish = True
return {
"type": "content_block_stop",
"index": self.current_content_block_index,
}
self.chunk_queue.append(processed_chunk)
return self.chunk_queue.popleft()
elif self.holding_chunk is not None:
return_chunk = self.holding_chunk
self.holding_chunk = processed_chunk
return return_chunk
self.chunk_queue.append(self.holding_chunk)
self.chunk_queue.append(processed_chunk)
self.holding_chunk = None
return self.chunk_queue.popleft()
else:
return processed_chunk
self.chunk_queue.append(processed_chunk)
return self.chunk_queue.popleft()
# Handle any remaining held chunks after stream ends
if self.holding_chunk is not None:
return_chunk = self.holding_chunk
self.chunk_queue.append(self.holding_chunk)
self.holding_chunk = None
return return_chunk
if self.sent_last_message is False:
if not self.sent_last_message:
self.sent_last_message = True
return {"type": "message_stop"}
self.chunk_queue.append({"type": "message_stop"})
if self.chunk_queue:
return self.chunk_queue.popleft()
raise StopIteration
except StopIteration:
if self.chunk_queue:
return self.chunk_queue.popleft()
if self.sent_last_message is False:
self.sent_last_message = True
return {"type": "message_stop"}
@ -265,7 +286,9 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
if not self.queued_usage_chunk:
if should_start_new_block and not self.sent_content_block_finish:
# Queue the sequence: content_block_stop -> content_block_start -> current_chunk
# Queue the sequence: content_block_stop -> content_block_start
# The trigger chunk itself is not emitted as a delta since the
# content_block_start already carries the relevant information.
# 1. Stop current content block
self.chunk_queue.append(
@ -284,9 +307,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
}
)
# 3. Queue the current chunk (don't lose it!)
self.chunk_queue.append(processed_chunk)
# Reset state for new block
self.sent_content_block_finish = False

8
litellm/llms/azure/chat/gpt_5_transformation.py
View file

@ -43,8 +43,12 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
if "tool_choice" not in params:
params.append("tool_choice")
# Only gpt-5.2 has been verified to support logprobs on Azure
if self.is_model_gpt_5_2_model(model):
# Only gpt-5.2 has been verified to support logprobs on Azure.
# The base OpenAI class includes logprobs for gpt-5.1+, but Azure
# hasn't verified support for gpt-5.1, so remove them unless gpt-5.2.
if self.is_model_gpt_5_1_model(model) and not self.is_model_gpt_5_2_model(model):
params = [p for p in params if p not in ["logprobs", "top_logprobs"]]
elif self.is_model_gpt_5_2_model(model):
azure_supported_params = ["logprobs", "top_logprobs"]
params.extend(azure_supported_params)

3
litellm/llms/gemini/common_utils.py
View file

@ -166,7 +166,8 @@ class GoogleAIStudioTokenCounter(BaseTokenCounter):
contents: Optional[List[Dict[str, Any]]],
deployment: Optional[Dict[str, Any]] = None,
request_model: str = "",
**kwargs,
tools: Optional[List[Dict[str, Any]]] = None,
system: Optional[Any] = None,
) -> Optional[TokenCountResponse]:
import copy

77
litellm/llms/openrouter/responses/transformation.py Normal file
View file

@ -0,0 +1,77 @@
"""
OpenRouter Responses API Configuration.
OpenRouter supports the Responses API at https://openrouter.ai/api/v1/responses
with OpenAI-compatible request/response format, including reasoning with
encrypted_content for multi-turn stateless workflows.
Docs: https://openrouter.ai/docs/api/reference/responses/overview
"""
from typing import Optional
import litellm
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
class OpenRouterResponsesAPIConfig(OpenAIResponsesAPIConfig):
"""
Configuration for OpenRouter's Responses API.
Inherits from OpenAIResponsesAPIConfig since OpenRouter's Responses API
is compatible with OpenAI's Responses API specification.
Key difference from direct OpenAI:
- Uses https://openrouter.ai/api/v1 as the API base
- Uses OPENROUTER_API_KEY for authentication
"""
@property
def custom_llm_provider(self) -> LlmProviders:
return LlmProviders.OPENROUTER
def validate_environment(
self,
headers: dict,
model: str,
litellm_params: Optional[GenericLiteLLMParams],
) -> dict:
litellm_params = litellm_params or GenericLiteLLMParams()
api_key = (
litellm_params.api_key
or litellm.api_key
or get_secret_str("OPENROUTER_API_KEY")
or get_secret_str("OR_API_KEY")
)
if not api_key:
raise ValueError(
"OpenRouter API key is required. Set OPENROUTER_API_KEY "
"environment variable or pass api_key parameter."
)
headers.update(
{
"Authorization": f"Bearer {api_key}",
}
)
return headers
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
api_base = (
api_base
or litellm.api_base
or get_secret_str("OPENROUTER_API_BASE")
or "https://openrouter.ai/api/v1"
)
api_base = api_base.rstrip("/")
return f"{api_base}/responses"

17
litellm/llms/vertex_ai/batches/handler.py
View file

@ -108,11 +108,18 @@ class VertexAIBatchPrediction(VertexLLM):
client = get_async_httpx_client(
llm_provider=litellm.LlmProviders.VERTEX_AI,
)
response = await client.post(
url=api_base,
headers=headers,
data=json.dumps(vertex_batch_request),
)
try:
response = await client.post(
url=api_base,
headers=headers,
data=json.dumps(vertex_batch_request),
)
except httpx.HTTPStatusError as e:
error_body = e.response.text if hasattr(e, 'response') else "N/A"
litellm.verbose_logger.error(
f"Vertex AI batch create failed: status={e.response.status_code}, body={error_body[:1000]}"
)
raise
if response.status_code != 200:
raise Exception(f"Error: {response.status_code} {response.text}")

2
litellm/llms/vertex_ai/batches/transformation.py
View file

@ -29,7 +29,7 @@ class VertexAIBatchTransformation:
if input_file_id is None:
raise ValueError("input_file_id is required, but not provided")
input_config: InputConfig = InputConfig(
gcsSource=GcsSource(uris=input_file_id), instancesFormat="jsonl"
gcsSource=GcsSource(uris=[input_file_id]), instancesFormat="jsonl"
)
model: str = cls._get_model_from_gcs_file(input_file_id)
output_config: OutputConfig = OutputConfig(

3
litellm/llms/vertex_ai/common_utils.py
View file

@ -1030,7 +1030,8 @@ class VertexAITokenCounter(BaseTokenCounter):
contents: Optional[List[Dict[str, Any]]],
deployment: Optional[Dict[str, Any]] = None,
request_model: str = "",
**kwargs,
tools: Optional[List[Dict[str, Any]]] = None,
system: Optional[Any] = None,
) -> Optional[TokenCountResponse]:
import copy

62
litellm/llms/vertex_ai/files/transformation.py
View file

@ -335,13 +335,37 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
status_code=status_code, message=error_message, headers=headers
)
def _parse_gcs_uri(self, file_id: str) -> Tuple[str, str]:
"""
Parse a GCS URI (gs://bucket/path/to/object) into (bucket, url-encoded-object-path).
Handles both raw and URL-encoded input.
"""
import urllib.parse
decoded = urllib.parse.unquote(file_id)
if decoded.startswith("gs://"):
full_path = decoded[5:]
else:
full_path = decoded
if "/" in full_path:
bucket_name, object_path = full_path.split("/", 1)
else:
bucket_name = full_path
object_path = ""
encoded_object = urllib.parse.quote(object_path, safe="")
return bucket_name, encoded_object
def transform_retrieve_file_request(
self,
file_id: str,
optional_params: dict,
litellm_params: dict,
) -> tuple[str, dict]:
raise NotImplementedError("VertexAIFilesConfig does not support file retrieval")
bucket, encoded_object = self._parse_gcs_uri(file_id)
url = f"https://storage.googleapis.com/storage/v1/b/{bucket}/o/{encoded_object}"
return url, {}
def transform_retrieve_file_response(
self,
@ -349,7 +373,21 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
logging_obj: LiteLLMLoggingObj,
litellm_params: dict,
) -> OpenAIFileObject:
raise NotImplementedError("VertexAIFilesConfig does not support file retrieval")
response_json = raw_response.json()
gcs_id = response_json.get("id", "")
gcs_id = "/".join(gcs_id.split("/")[:-1]) if gcs_id else ""
return OpenAIFileObject(
id=f"gs://{gcs_id}",
bytes=int(response_json.get("size", 0)),
created_at=_convert_vertex_datetime_to_openai_datetime(
vertex_datetime=response_json.get("timeCreated", "")
),
filename=response_json.get("name", ""),
object="file",
purpose=response_json.get("metadata", {}).get("purpose", "batch"),
status="processed",
status_details=None,
)
def transform_delete_file_request(
self,
@ -357,7 +395,9 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
optional_params: dict,
litellm_params: dict,
) -> tuple[str, dict]:
raise NotImplementedError("VertexAIFilesConfig does not support file deletion")
bucket, encoded_object = self._parse_gcs_uri(file_id)
url = f"https://storage.googleapis.com/storage/v1/b/{bucket}/o/{encoded_object}"
return url, {}
def transform_delete_file_response(
self,
@ -365,7 +405,14 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
logging_obj: LiteLLMLoggingObj,
litellm_params: dict,
) -> FileDeleted:
raise NotImplementedError("VertexAIFilesConfig does not support file deletion")
file_id = "deleted"
if hasattr(raw_response, "request") and raw_response.request:
url = str(raw_response.request.url)
if "/o/" in url:
import urllib.parse
encoded_name = url.split("/o/")[-1].split("?")[0]
file_id = f"gs://{urllib.parse.unquote(encoded_name)}"
return FileDeleted(id=file_id, deleted=True, object="file")
def transform_list_files_request(
self,
@ -389,7 +436,10 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
optional_params: dict,
litellm_params: dict,
) -> tuple[str, dict]:
raise NotImplementedError("VertexAIFilesConfig does not support file content retrieval")
file_id = file_content_request.get("file_id", "")
bucket, encoded_object = self._parse_gcs_uri(file_id)
url = f"https://storage.googleapis.com/storage/v1/b/{bucket}/o/{encoded_object}?alt=media"
return url, {}
def transform_file_content_response(
self,
@ -397,7 +447,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
logging_obj: LiteLLMLoggingObj,
litellm_params: dict,
) -> HttpxBinaryResponseContent:
raise NotImplementedError("VertexAIFilesConfig does not support file content retrieval")
return HttpxBinaryResponseContent(response=raw_response)
class VertexAIJsonlFilesTransformation(VertexGeminiConfig):

17
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -1136,23 +1136,6 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
if VertexGeminiConfig._is_gemini_3_or_newer(model):
if "temperature" not in optional_params:
optional_params["temperature"] = 1.0
# Only add thinkingLevel if model supports it (exclude image models)
if "image" not in model.lower():
thinking_config = optional_params.get("thinkingConfig", {})
if (
"thinkingLevel" not in thinking_config
and "thinkingBudget" not in thinking_config
):
# For gemini-3-flash-preview, default to "minimal" to match Gemini 2.5 Flash behavior
# For other Gemini 3 models, default to "low"
is_gemini3flash = (
"gemini-3-flash-preview" in model.lower()
or "gemini-3-flash" in model.lower()
)
thinking_config["thinkingLevel"] = (
"minimal" if is_gemini3flash else "low"
)
optional_params["thinkingConfig"] = thinking_config
return optional_params

488
litellm/model_prices_and_context_window_backup.json
View file

@ -14334,6 +14334,57 @@
"supports_vision": true,
"supports_web_search": true
},
"gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_per_audio_token": 5e-08,
"input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
"max_images_per_prompt": 3000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_pdf_size_mb": 30,
"max_tokens": 65536,
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
"output_cost_per_reasoning_token": 1.5e-06,
"output_cost_per_token": 1.5e-06,
"source": "https://ai.google.dev/gemini-api/docs/models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_code_execution": true,
"supports_file_search": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_url_context": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true
},
"deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -17090,6 +17141,59 @@
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
"supports_service_tier": true
},
"gemini/gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_per_audio_token": 5e-08,
"input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
"max_images_per_prompt": 3000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_pdf_size_mb": 30,
"max_tokens": 65536,
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
"output_cost_per_reasoning_token": 1.5e-06,
"output_cost_per_token": 1.5e-06,
"rpm": 15,
"source": "https://ai.google.dev/gemini-api/docs/models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_code_execution": true,
"supports_file_search": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_url_context": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true,
"tpm": 250000
},
"gemini/gemini-3-flash-preview": {
"cache_read_input_token_cost": 5e-08,
"input_cost_per_audio_token": 1e-06,
@ -24226,6 +24330,335 @@
"/v1/images/generations"
]
},
"nebius/deepseek-ai/DeepSeek-R1": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 8e-07,
"output_cost_per_token": 2.4e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_reasoning": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-R1-0528": {
"max_tokens": 164000,
"max_input_tokens": 164000,
"max_output_tokens": 164000,
"input_cost_per_token": 8e-07,
"output_cost_per_token": 2.4e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_reasoning": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-R1-Distill-Llama-70B": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 2.5e-07,
"output_cost_per_token": 7.5e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-V3": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 5e-07,
"output_cost_per_token": 1.5e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-V3-0324": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 5e-07,
"output_cost_per_token": 1.5e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/google/gemma-3-27b-it": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 6e-08,
"output_cost_per_token": 2e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Llama-3.3-70B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Llama-Guard-3-8B": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 2e-08,
"output_cost_per_token": 6e-08,
"litellm_provider": "nebius",
"mode": "chat",
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Meta-Llama-3.1-8B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 2e-08,
"output_cost_per_token": 6e-08,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Meta-Llama-3.1-70B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Meta-Llama-3.1-405B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1e-06,
"output_cost_per_token": 3e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/mistralai/Mistral-Nemo-Instruct-2407": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 4e-08,
"output_cost_per_token": 1.2e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/NousResearch/Hermes-3-Llama-3.1-405B": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1e-06,
"output_cost_per_token": 3e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/nvidia/Llama-3.1-Nemotron-Ultra-253B-v1": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 6e-07,
"output_cost_per_token": 1.8e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/nvidia/Llama-3.3-Nemotron-Super-49B-v1": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-235B-A22B": {
"max_tokens": 262144,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"input_cost_per_token": 2e-07,
"output_cost_per_token": 6e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-32B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 3e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-30B-A3B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 3e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-14B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 8e-08,
"output_cost_per_token": 2.4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-4B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 8e-08,
"output_cost_per_token": 2.4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/QwQ-32B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1.5e-07,
"output_cost_per_token": 4.5e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_reasoning": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-72B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-32B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 6e-08,
"output_cost_per_token": 2e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-Coder-7B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 3e-08,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-VL-72B-Instruct": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2-VL-72B-Instruct": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2-VL-7B-Instruct": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 2e-08,
"output_cost_per_token": 6e-08,
"litellm_provider": "nebius",
"mode": "chat",
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/BAAI/bge-en-icl": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "nebius",
"mode": "embedding",
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/BAAI/bge-multilingual-gemma2": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "nebius",
"mode": "embedding",
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/intfloat/e5-mistral-7b-instruct": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "nebius",
"mode": "embedding",
"source": "https://nebius.com/prices-ai-studio"
},
"nvidia.nemotron-nano-12b-v2": {
"input_cost_per_token": 2e-07,
"litellm_provider": "bedrock_converse",
@ -27188,7 +27621,7 @@
"supports_function_calling": true
},
"perplexity/pplx-embed-v1-0.6b": {
"input_cost_per_token": 0.000000004,
"input_cost_per_token": 4e-09,
"litellm_provider": "perplexity",
"max_input_tokens": 32768,
"max_tokens": 32768,
@ -27198,7 +27631,7 @@
"source": "https://docs.perplexity.ai/docs/embeddings/quickstart"
},
"perplexity/pplx-embed-v1-4b": {
"input_cost_per_token": 0.00000003,
"input_cost_per_token": 3e-08,
"litellm_provider": "perplexity",
"max_input_tokens": 32768,
"max_tokens": 32768,
@ -31973,6 +32406,57 @@
"output_cost_per_token": 3e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models"
},
"vertex_ai/gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_per_audio_token": 5e-08,
"input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
"max_images_per_prompt": 3000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_pdf_size_mb": 30,
"max_tokens": 65536,
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
"output_cost_per_reasoning_token": 1.5e-06,
"output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_code_execution": true,
"supports_file_search": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_url_context": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true
},
"vertex_ai/deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,

11
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -642,6 +642,7 @@ class MCPServerManager:
available_on_public_internet=bool(
getattr(mcp_server, "available_on_public_internet", True)
),
created_at=getattr(mcp_server, "created_at", None),
updated_at=getattr(mcp_server, "updated_at", None),
)
return new_server
@ -2540,8 +2541,8 @@ class MCPServerManager:
url=server.url,
transport=server.transport,
auth_type=server.auth_type,
created_at=datetime.now(),
updated_at=datetime.now(),
created_at=server.created_at,
updated_at=server.updated_at,
teams=[],
mcp_access_groups=server.access_groups or [],
allowed_tools=server.allowed_tools or [],
@ -2620,8 +2621,6 @@ class MCPServerManager:
return list_mcp_servers
def _build_mcp_server_table(self, server: MCPServer) -> LiteLLM_MCPServerTable:
from datetime import datetime
return LiteLLM_MCPServerTable(
server_id=server.server_id,
server_name=server.server_name,
@ -2633,8 +2632,8 @@ class MCPServerManager:
spec_path=server.spec_path,
transport=server.transport,
auth_type=server.auth_type,
created_at=datetime.now(),
updated_at=datetime.now(),
created_at=server.created_at,
updated_at=server.updated_at,
teams=[],
mcp_access_groups=server.access_groups or [],
allowed_tools=server.allowed_tools or [],

9
litellm/proxy/_types.py
View file

@ -512,6 +512,7 @@ class LiteLLMRoutes(enum.Enum):
KeyManagementRoutes.KEY_UNBLOCK.value,
KeyManagementRoutes.KEY_BULK_UPDATE.value,
KeyManagementRoutes.TEAM_DAILY_ACTIVITY.value,
KeyManagementRoutes.KEY_RESET_SPEND.value,
]
management_routes = [
@ -646,6 +647,8 @@ class LiteLLMRoutes(enum.Enum):
# Invitation routes - org/team admins checked in endpoint via _user_has_admin_privileges
"/invitation/new",
"/invitation/delete",
# Team guardrail submission - requires team-scoped key; endpoint enforces team_id
"/guardrails/register",
] # routes that manage their own allowed/disallowed logic
## Org Admin Routes ##
@ -1549,6 +1552,8 @@ class NewTeamRequest(TeamBase):
] = None # allow user to set TPM limit for all team members
team_member_key_duration: Optional[str] = None # e.g. "1d", "1w", "1m"
allowed_vector_store_indexes: Optional[List[AllowedVectorStoreIndexItem]] = None
enforced_batch_output_expires_after: Optional[dict] = None
enforced_file_expires_after: Optional[dict] = None
model_config = ConfigDict(protected_namespaces=())
@ -1604,6 +1609,8 @@ class UpdateTeamRequest(LiteLLMPydanticObjectBase):
model_rpm_limit: Optional[Dict[str, int]] = None
model_tpm_limit: Optional[Dict[str, int]] = None
allowed_vector_store_indexes: Optional[List[AllowedVectorStoreIndexItem]] = None
enforced_batch_output_expires_after: Optional[dict] = None
enforced_file_expires_after: Optional[dict] = None
router_settings: Optional[dict] = None
access_group_ids: Optional[List[str]] = None
@ -3781,6 +3788,8 @@ LiteLLM_ManagementEndpoint_MetadataFields = [
"temp_budget_increase",
"temp_budget_expiry",
"allowed_vector_store_indexes",
"enforced_batch_output_expires_after",
"enforced_file_expires_after",
]
LiteLLM_ManagementEndpoint_MetadataFields_Premium = [

28
litellm/proxy/auth/auth_checks.py
View file

@ -234,6 +234,7 @@ async def common_checks(
request: Request,
skip_budget_checks: bool = False,
project_object: Optional[LiteLLM_ProjectTableCachedObj] = None,
skip_route_check: bool = False,
) -> bool:
"""
Common checks across jwt + key-based auth.
@ -453,18 +454,21 @@ async def common_checks(
user_object=user_object, route=route, request_body=request_body
)
token_team = getattr(valid_token, "team_id", None)
token_type: Literal["ui", "api"] = (
"ui" if token_team is not None and token_team == "litellm-dashboard" else "api"
)
_is_route_allowed = _is_allowed_route(
route=route,
token_type=token_type,
user_obj=user_object,
request=request,
request_data=request_body,
valid_token=valid_token,
)
if not skip_route_check:
token_team = getattr(valid_token, "team_id", None)
token_type: Literal["ui", "api"] = (
"ui"
if token_team is not None and token_team == "litellm-dashboard"
else "api"
)
_is_route_allowed = _is_allowed_route(
route=route,
token_type=token_type,
user_obj=user_object,
request=request,
request_data=request_body,
valid_token=valid_token,
)
# 11. [OPTIONAL] Vector store checks - is the object allowed to access the vector store
await vector_store_access_check(

1
litellm/proxy/auth/user_api_key_auth.py
View file

@ -1766,6 +1766,7 @@ async def _run_post_custom_auth_checks(
valid_token=valid_token,
skip_budget_checks=False,
project_object=_project_obj,
skip_route_check=True,
)
return valid_token

16
litellm/proxy/batches_endpoints/endpoints.py
View file

@ -118,6 +118,22 @@ async def create_batch( # noqa: PLR0915
or "openai"
)
_create_batch_data = LiteLLMBatchCreateRequest(**data)
# Apply team-level batch output expiry enforcement
team_metadata = user_api_key_dict.team_metadata or {}
enforced_batch_expiry = team_metadata.get(
"enforced_batch_output_expires_after"
)
if enforced_batch_expiry is not None:
if "anchor" not in enforced_batch_expiry or "seconds" not in enforced_batch_expiry:
raise HTTPException(
status_code=400,
detail={
"error": "enforced_batch_output_expires_after must contain 'anchor' and 'seconds' keys",
},
)
_create_batch_data["output_expires_after"] = enforced_batch_expiry
input_file_id = _create_batch_data.get("input_file_id", None)
unified_file_id: Union[str, Literal[False]] = False

16
litellm/proxy/db/db_transaction_queue/spend_log_cleanup.py
View file

@ -49,14 +49,18 @@ class SpendLogCleanup:
try:
if isinstance(retention_setting, int):
retention_setting = str(retention_setting)
verbose_proxy_logger.warning(
f"maximum_spend_logs_retention_period is an integer ({retention_setting}); treating as days. "
"Use a string like '3d' to be explicit."
)
retention_setting = f"{retention_setting}d"
self.retention_seconds = duration_in_seconds(retention_setting)
verbose_proxy_logger.info(
f"Retention period set to {self.retention_seconds} seconds"
)
return True
except ValueError as e:
verbose_proxy_logger.error(
verbose_proxy_logger.warning(
f"Invalid maximum_spend_logs_retention_period value: {retention_setting}, error: {str(e)}"
)
return False
@ -112,13 +116,11 @@ class SpendLogCleanup:
If pod_lock_manager is available, ensures only one pod runs cleanup.
If no pod_lock_manager, runs cleanup without distributed locking.
"""
lock_acquired = False
try:
verbose_proxy_logger.info(f"Cleanup job triggered at {datetime.now()}")
if not self._should_delete_spend_logs():
verbose_proxy_logger.info(
"Skipping cleanup — invalid or missing retention setting."
)
return
if self.retention_seconds is None:
@ -155,8 +157,8 @@ class SpendLogCleanup:
verbose_proxy_logger.error(f"Error during cleanup: {str(e)}")
return # Return after error handling
finally:
# Always release the lock if we have a pod lock manager
if self.pod_lock_manager and self.pod_lock_manager.redis_cache:
# Only release the lock if it was actually acquired
if lock_acquired and self.pod_lock_manager and self.pod_lock_manager.redis_cache:
await self.pod_lock_manager.release_lock(
cronjob_id=SPEND_LOG_CLEANUP_JOB_NAME
)

454
litellm/proxy/guardrails/guardrail_endpoints.py
View file

@ -4,7 +4,10 @@ CRUD ENDPOINTS FOR GUARDRAILS
import concurrent.futures
import inspect
import json
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Type, TypeVar, Union, cast
from urllib.parse import urlparse
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
@ -12,6 +15,7 @@ from pydantic import BaseModel
from litellm._logging import verbose_proxy_logger
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.guardrails.guardrail_hooks.custom_code.code_validator import (
@ -525,6 +529,456 @@ async def delete_guardrail(
raise HTTPException(status_code=500, detail=str(e))
# --- Team guardrail registration (Generic Guardrail API spec) ---
GENERIC_GUARDRAIL_API = "generic_guardrail_api"
class RegisterGuardrailRequest(BaseModel):
"""Request body for POST /guardrails/register. Follows Generic Guardrail API config."""
guardrail_name: str
litellm_params: Dict[
str, Any
] # guardrail, mode, api_base required; api_key, headers, etc. optional
guardrail_info: Optional[Dict[str, Any]] = None
def get_litellm_params_dict(self) -> Dict[str, Any]:
return dict(self.litellm_params)
class RegisterGuardrailResponse(BaseModel):
guardrail_id: str
guardrail_name: str
status: str
submitted_at: Optional[datetime] = None
class GuardrailSubmissionSummary(BaseModel):
total: int
pending_review: int
active: int
rejected: int
class GuardrailSubmissionItem(BaseModel):
guardrail_id: str
guardrail_name: str
status: str # pending_review | active | rejected
team_id: Optional[str] = None
team_guardrail: bool = (
False # True when submitted via team (team_id set); use to distinguish team vs regular guardrails
)
litellm_params: Optional[Dict[str, Any]] = None
guardrail_info: Optional[Dict[str, Any]] = None
submitted_by_user_id: Optional[str] = None
submitted_by_email: Optional[str] = None
submitted_at: Optional[datetime] = None
reviewed_at: Optional[datetime] = None
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
class ListGuardrailSubmissionsResponse(BaseModel):
submissions: List[GuardrailSubmissionItem]
summary: GuardrailSubmissionSummary
@router.post(
"/guardrails/register",
tags=["Guardrails"],
response_model=RegisterGuardrailResponse,
)
async def register_guardrail(
request: RegisterGuardrailRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
Register a guardrail for onboarding (team submission).
Accepts a guardrail config in the
[Generic Guardrail API](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api) format.
The submission is stored with status `pending_review` until an admin approves it.
"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
raise HTTPException(status_code=500, detail="Prisma client not initialized")
if not user_api_key_dict.team_id:
raise HTTPException(
status_code=400,
detail="Registration requires an API key associated with a team. Use a team-scoped key.",
)
params = request.get_litellm_params_dict()
if params.get("guardrail") != GENERIC_GUARDRAIL_API:
raise HTTPException(
status_code=400,
detail=f"Only guardrails with litellm_params.guardrail={GENERIC_GUARDRAIL_API!r} are accepted for registration",
)
api_base = params.get("api_base")
if not api_base:
raise HTTPException(
status_code=400,
detail="litellm_params.api_base is required for generic_guardrail_api",
)
parsed = urlparse(api_base)
if parsed.scheme not in ("http", "https"):
raise HTTPException(
status_code=400,
detail="litellm_params.api_base must use http or https scheme",
)
if not parsed.hostname:
raise HTTPException(
status_code=400,
detail="litellm_params.api_base must contain a valid hostname",
)
mode = params.get("mode")
if mode is None:
raise HTTPException(
status_code=400,
detail="litellm_params.mode is required (e.g. pre_call, post_call)",
)
try:
existing = await prisma_client.db.litellm_guardrailstable.find_unique(
where={"guardrail_name": request.guardrail_name}
)
if existing is not None:
raise HTTPException(
status_code=400,
detail=f"Guardrail with name {request.guardrail_name!r} already exists",
)
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.exception(
"Error checking guardrail name uniqueness: %s", e
)
raise HTTPException(status_code=500, detail=str(e))
now = datetime.now(timezone.utc)
litellm_params_str = safe_dumps(params)
guardrail_info = dict(request.guardrail_info or {})
guardrail_info["submitted_by_user_id"] = user_api_key_dict.user_id
guardrail_info["submitted_by_email"] = user_api_key_dict.user_email
guardrail_info["team_guardrail"] = (
True # Mark as team submission for filtering/display
)
guardrail_info_str = safe_dumps(guardrail_info)
try:
created = await prisma_client.db.litellm_guardrailstable.create(
data={
"guardrail_name": request.guardrail_name,
"litellm_params": litellm_params_str,
"guardrail_info": guardrail_info_str,
"status": "pending_review",
"team_id": user_api_key_dict.team_id,
"submitted_at": now,
"created_at": now,
"updated_at": now,
}
)
return RegisterGuardrailResponse(
guardrail_id=created.guardrail_id,
guardrail_name=created.guardrail_name,
status=created.status,
submitted_at=created.submitted_at,
)
except Exception as e:
verbose_proxy_logger.exception("Error registering guardrail: %s", e)
raise HTTPException(status_code=500, detail=str(e))
def _parse_json_field(value: Any) -> Optional[Dict[str, Any]]:
if value is None:
return None
if isinstance(value, dict):
return value
if isinstance(value, str):
try:
return json.loads(value)
except Exception:
return None
return None
def _row_to_submission_item(row: Any) -> GuardrailSubmissionItem:
guardrail_info = _parse_json_field(row.guardrail_info) or {}
team_guardrail = row.team_id is not None
return GuardrailSubmissionItem(
guardrail_id=row.guardrail_id,
guardrail_name=row.guardrail_name,
status=row.status or "active",
team_id=row.team_id,
team_guardrail=team_guardrail,
litellm_params=_parse_json_field(row.litellm_params),
guardrail_info=guardrail_info,
submitted_by_user_id=guardrail_info.get("submitted_by_user_id"),
submitted_by_email=guardrail_info.get("submitted_by_email"),
submitted_at=getattr(row, "submitted_at", None),
reviewed_at=getattr(row, "reviewed_at", None),
created_at=row.created_at,
updated_at=row.updated_at,
)
@router.get(
"/guardrails/submissions",
tags=["Guardrails"],
response_model=ListGuardrailSubmissionsResponse,
)
async def list_guardrail_submissions(
status: Optional[str] = None,
team_id: Optional[str] = None,
search: Optional[str] = None,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
List team guardrail submissions (admin only). Returns only guardrails with a team_id.
Status values: pending_review (team-registered, awaiting approval), active (approved), rejected.
Optional filters:
- status: pending_review | active | rejected
- team_id: filter by specific team
- search: name/description
"""
from litellm.proxy.proxy_server import prisma_client
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(status_code=403, detail="Admin access required")
if prisma_client is None:
raise HTTPException(status_code=500, detail="Prisma client not initialized")
try:
# Single query: fetch all team guardrails (team_id is not null)
all_team_rows = await prisma_client.db.litellm_guardrailstable.find_many(
where={"team_id": {"not": None}},
order={"created_at": "desc"},
)
# Derive summary counts from the full result set
total = len(all_team_rows)
pending_review = sum(
1 for r in all_team_rows if (r.status or "active") == "pending_review"
)
active_count = sum(
1 for r in all_team_rows if (r.status or "active") == "active"
)
rejected = sum(
1 for r in all_team_rows if (r.status or "active") == "rejected"
)
# Apply filters to get the submissions list
rows = all_team_rows
if status:
rows = [r for r in rows if r.status == status]
if team_id:
rows = [r for r in rows if r.team_id == team_id]
if search:
search_lower = search.lower()
rows = [
r
for r in rows
if search_lower in (r.guardrail_name or "").lower()
or (
isinstance(r.guardrail_info, dict)
and search_lower
in str((r.guardrail_info or {}).get("description", "")).lower()
)
or (
isinstance(r.guardrail_info, str)
and search_lower in r.guardrail_info.lower()
)
]
items = [_row_to_submission_item(r) for r in rows]
return ListGuardrailSubmissionsResponse(
submissions=items,
summary=GuardrailSubmissionSummary(
total=total,
pending_review=pending_review,
active=active_count,
rejected=rejected,
),
)
except Exception as e:
verbose_proxy_logger.exception("Error listing guardrail submissions: %s", e)
raise HTTPException(status_code=500, detail=str(e))
@router.get(
"/guardrails/submissions/{guardrail_id}",
tags=["Guardrails"],
response_model=GuardrailSubmissionItem,
)
async def get_guardrail_submission(
guardrail_id: str,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""Get a single guardrail submission by id (admin only)."""
from litellm.proxy.proxy_server import prisma_client
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(status_code=403, detail="Admin access required")
if prisma_client is None:
raise HTTPException(status_code=500, detail="Prisma client not initialized")
try:
row = await prisma_client.db.litellm_guardrailstable.find_unique(
where={"guardrail_id": guardrail_id}
)
if row is None:
raise HTTPException(
status_code=404, detail="Guardrail submission not found"
)
return _row_to_submission_item(row)
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.exception("Error getting guardrail submission: %s", e)
raise HTTPException(status_code=500, detail=str(e))
@router.post(
"/guardrails/submissions/{guardrail_id}/approve",
tags=["Guardrails"],
)
async def approve_guardrail_submission(
guardrail_id: str,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""Approve a pending guardrail submission: set status to active and initialize in memory (admin only)."""
from litellm.proxy.guardrails.guardrail_registry import IN_MEMORY_GUARDRAIL_HANDLER
from litellm.proxy.proxy_server import prisma_client
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(status_code=403, detail="Admin access required")
if prisma_client is None:
raise HTTPException(status_code=500, detail="Prisma client not initialized")
try:
row = await prisma_client.db.litellm_guardrailstable.find_unique(
where={"guardrail_id": guardrail_id}
)
if row is None:
raise HTTPException(
status_code=404, detail="Guardrail submission not found"
)
if row.status != "pending_review":
raise HTTPException(
status_code=400,
detail=f"Guardrail is not pending review (status={row.status})",
)
now = datetime.now(timezone.utc)
await prisma_client.db.litellm_guardrailstable.update(
where={"guardrail_id": guardrail_id},
data={"status": "active", "reviewed_at": now, "updated_at": now},
)
litellm_params = _parse_json_field(row.litellm_params)
guardrail_info = _parse_json_field(row.guardrail_info)
if not litellm_params:
raise HTTPException(
status_code=500,
detail="Guardrail litellm_params is missing or invalid",
)
guardrail_dict = {
"guardrail_id": row.guardrail_id,
"guardrail_name": row.guardrail_name,
"litellm_params": litellm_params,
"guardrail_info": guardrail_info or {},
}
try:
IN_MEMORY_GUARDRAIL_HANDLER.initialize_guardrail(
guardrail=cast(Guardrail, guardrail_dict)
)
verbose_proxy_logger.info(
"Approved guardrail %s (ID: %s) and initialized in memory",
row.guardrail_name,
guardrail_id,
)
except Exception as init_err:
verbose_proxy_logger.warning(
"Failed to initialize approved guardrail %s in memory: %s",
guardrail_id,
init_err,
)
return {
"guardrail_id": guardrail_id,
"status": "active",
"message": "Guardrail approved",
"warning": f"Guardrail was marked active but failed to initialize in memory: {init_err}. "
"It will be picked up on the next sync cycle.",
}
return {
"guardrail_id": guardrail_id,
"status": "active",
"message": "Guardrail approved",
}
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.exception("Error approving guardrail submission: %s", e)
raise HTTPException(status_code=500, detail=str(e))
@router.post(
"/guardrails/submissions/{guardrail_id}/reject",
tags=["Guardrails"],
)
async def reject_guardrail_submission(
guardrail_id: str,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""Reject a guardrail submission (admin only)."""
from litellm.proxy.proxy_server import prisma_client
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(status_code=403, detail="Admin access required")
if prisma_client is None:
raise HTTPException(status_code=500, detail="Prisma client not initialized")
try:
row = await prisma_client.db.litellm_guardrailstable.find_unique(
where={"guardrail_id": guardrail_id}
)
if row is None:
raise HTTPException(
status_code=404, detail="Guardrail submission not found"
)
if row.status != "pending_review":
raise HTTPException(
status_code=400,
detail=f"Guardrail is not pending review (status={row.status})",
)
now = datetime.now(timezone.utc)
await prisma_client.db.litellm_guardrailstable.update(
where={"guardrail_id": guardrail_id},
data={"status": "rejected", "reviewed_at": now, "updated_at": now},
)
return {
"guardrail_id": guardrail_id,
"status": "rejected",
"message": "Guardrail rejected",
}
except HTTPException:
raise
except Exception as e:
verbose_proxy_logger.exception("Error rejecting guardrail submission: %s", e)
raise HTTPException(status_code=500, detail=str(e))
@router.patch(
"/guardrails/{guardrail_id}",
tags=["Guardrails"],

1
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/__init__.py
View file

@ -21,6 +21,7 @@ def initialize_guardrail(litellm_params: "LitellmParams", guardrail: "Guardrail"
unreachable_fallback=getattr(
litellm_params, "unreachable_fallback", "fail_closed"
),
extra_headers=getattr(litellm_params, "extra_headers", None),
guardrail_name=guardrail.get("guardrail_name", ""),
event_hook=litellm_params.mode,
default_on=litellm_params.default_on,

51
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/generic_guardrail_api.py
View file

@ -7,7 +7,7 @@
import fnmatch
import os
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Set
import httpx
@ -54,22 +54,30 @@ _HEADER_VALUE_ALLOWLIST = frozenset(
_HEADER_PRESENT_PLACEHOLDER = "[present]"
def _header_value_allowed(header_name: str) -> bool:
"""Return True if this header's value may be forwarded (allowlist, including globs)."""
def _header_value_allowed(
header_name: str,
extra_allowlist: Optional[Set[str]] = None,
) -> bool:
"""Return True if this header's value may be forwarded (allowlist, including globs and extra_headers)."""
lower = header_name.lower()
if lower in _HEADER_VALUE_ALLOWLIST:
return True
for pattern in _HEADER_VALUE_ALLOWLIST:
if "*" in pattern and fnmatch.fnmatch(lower, pattern):
return True
if extra_allowlist and lower in extra_allowlist:
return True
return False
def _sanitize_inbound_headers(headers: Any) -> Optional[Dict[str, str]]:
def _sanitize_inbound_headers(
headers: Any,
extra_allowlist: Optional[Set[str]] = None,
) -> Optional[Dict[str, str]]:
"""
Sanitize inbound headers before passing them to a 3rd party guardrail service.
- Allowlist: only headers in the allowlist have their values forwarded (exact + glob: x-stainless-*, x-litellm-*).
- Allowlist: default allowlist + extra_allowlist (from litellm_params.extra_headers); only these have values forwarded.
- All other headers are included with value "[present]" so the guardrail knows the header existed.
- Coerces values to str (for JSON serialization).
"""
@ -81,7 +89,7 @@ def _sanitize_inbound_headers(headers: Any) -> Optional[Dict[str, str]]:
if k is None:
continue
key = str(k)
if _header_value_allowed(key):
if _header_value_allowed(key, extra_allowlist=extra_allowlist):
try:
sanitized[key] = str(v)
except Exception:
@ -93,7 +101,9 @@ def _sanitize_inbound_headers(headers: Any) -> Optional[Dict[str, str]]:
def _extract_inbound_headers(
request_data: dict, logging_obj: Optional["LiteLLMLoggingObj"]
request_data: dict,
logging_obj: Optional["LiteLLMLoggingObj"],
extra_allowlist: Optional[Set[str]] = None,
) -> Optional[Dict[str, str]]:
"""
Extract inbound headers from available request context.
@ -107,23 +117,27 @@ def _extract_inbound_headers(
# 1) Most common path (proxy): full request context in proxy_server_request
headers = request_data.get("proxy_server_request", {}).get("headers")
if headers:
return _sanitize_inbound_headers(headers)
return _sanitize_inbound_headers(headers, extra_allowlist=extra_allowlist)
# 2) Some guardrails pass proxy_server_request as request_data itself
headers = request_data.get("headers")
if headers:
return _sanitize_inbound_headers(headers)
return _sanitize_inbound_headers(headers, extra_allowlist=extra_allowlist)
# 3) Pre-call: headers stored in request metadata
metadata_headers = (request_data.get("metadata") or {}).get("headers")
if metadata_headers:
return _sanitize_inbound_headers(metadata_headers)
return _sanitize_inbound_headers(
metadata_headers, extra_allowlist=extra_allowlist
)
litellm_metadata_headers = (request_data.get("litellm_metadata") or {}).get(
"headers"
)
if litellm_metadata_headers:
return _sanitize_inbound_headers(litellm_metadata_headers)
return _sanitize_inbound_headers(
litellm_metadata_headers, extra_allowlist=extra_allowlist
)
# 4) Post-call: headers not present on response; fallback to logging object
if logging_obj and getattr(logging_obj, "model_call_details", None):
@ -135,7 +149,9 @@ def _extract_inbound_headers(
.get("headers", None)
)
if headers:
return _sanitize_inbound_headers(headers)
return _sanitize_inbound_headers(
headers, extra_allowlist=extra_allowlist
)
except Exception:
pass
@ -171,12 +187,14 @@ class GenericGuardrailAPI(CustomGuardrail):
api_key: Optional[str] = None,
additional_provider_specific_params: Optional[Dict[str, Any]] = None,
unreachable_fallback: Literal["fail_closed", "fail_open"] = "fail_closed",
extra_headers: Optional[list] = None,
**kwargs,
):
self.async_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.GuardrailCallback
)
self.headers = headers or {}
self.extra_headers = extra_headers or []
# If api_key is provided, add it as x-api-key header
if api_key:
@ -370,8 +388,15 @@ class GenericGuardrailAPI(CustomGuardrail):
# Extract user API key metadata
user_metadata = self._extract_user_api_key_metadata(request_data)
extra_allowlist = (
{h.lower() for h in self.extra_headers if isinstance(h, str)}
if self.extra_headers
else None
)
inbound_headers = _extract_inbound_headers(
request_data=request_data, logging_obj=logging_obj
request_data=request_data,
logging_obj=logging_obj,
extra_allowlist=extra_allowlist,
)
# Create request payload

14
litellm/proxy/guardrails/guardrail_registry.py
View file

@ -11,8 +11,12 @@ from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.proxy.utils import PrismaClient
from litellm.proxy.guardrails.guardrail_hooks.grayswan import GraySwanGuardrail
from litellm.proxy.guardrails.guardrail_hooks.grayswan import (
initialize_guardrail as initialize_grayswan,
)
from litellm.proxy.types_utils.utils import get_instance_fn
from litellm.proxy.utils import PrismaClient
from litellm.secret_managers.main import get_secret
from litellm.types.guardrails import (
Guardrail,
@ -21,10 +25,6 @@ from litellm.types.guardrails import (
LitellmParams,
SupportedGuardrailIntegrations,
)
from litellm.proxy.guardrails.guardrail_hooks.grayswan import (
GraySwanGuardrail,
initialize_guardrail as initialize_grayswan,
)
from .guardrail_initializers import (
initialize_bedrock,
@ -327,11 +327,13 @@ class GuardrailRegistry:
prisma_client: PrismaClient,
) -> List[Guardrail]:
"""
Get all guardrails from the database
Get all active guardrails from the database.
Only rows with status == "active" are returned (pending_review and rejected are excluded).
"""
try:
guardrails_from_db = (
await prisma_client.db.litellm_guardrailstable.find_many(
where={"status": "active"},
order={"created_at": "desc"},
)
)

15
litellm/proxy/management_endpoints/cost_tracking_settings.py
View file

@ -54,16 +54,27 @@ def _resolve_model_for_cost_lookup(model: str) -> Tuple[str, Optional[str]]:
deployments = llm_router.get_model_list(model_name=model)
if deployments and len(deployments) > 0:
# Get the first deployment's litellm model
first_deployment = deployments[0]
litellm_params = first_deployment.get("litellm_params", {})
model_info = first_deployment.get("model_info", {})
# Check base_model first (needed for Azure custom deployment names)
base_model = model_info.get("base_model") or litellm_params.get(
"base_model"
)
if base_model:
verbose_proxy_logger.debug(
f"Resolved model '{model}' to base_model '{base_model}' from router"
)
custom_llm_provider = litellm_params.get("custom_llm_provider")
return base_model, custom_llm_provider
resolved_model = litellm_params.get("model")
if resolved_model:
verbose_proxy_logger.debug(
f"Resolved model '{model}' to '{resolved_model}' from router"
)
# Extract custom_llm_provider if present
custom_llm_provider = litellm_params.get("custom_llm_provider")
return resolved_model, custom_llm_provider
except Exception as e:

29
litellm/proxy/openai_files_endpoints/files_endpoints.py
View file

@ -454,8 +454,35 @@ async def create_file( # noqa: PLR0915
model=router_model, llm_router=llm_router
)
# Apply team-level file expiry enforcement
team_metadata = user_api_key_dict.team_metadata or {}
enforced_file_expiry = team_metadata.get("enforced_file_expires_after")
if enforced_file_expiry is not None:
if "anchor" not in enforced_file_expiry or "seconds" not in enforced_file_expiry:
raise HTTPException(
status_code=400,
detail={
"error": "enforced_file_expires_after must contain 'anchor' and 'seconds' keys",
},
)
if enforced_file_expiry["anchor"] != "created_at":
raise HTTPException(
status_code=400,
detail={
"error": f"enforced_file_expires_after anchor must be 'created_at', got '{enforced_file_expiry['anchor']}'",
},
)
expires_after = FileExpiresAfter(
anchor="created_at",
seconds=enforced_file_expiry["seconds"],
)
verbose_proxy_logger.debug(
"create_file expires_after: %s", expires_after
)
_create_file_request = CreateFileRequest(
file=file_data,
file=file_data,
purpose=cast(CREATE_FILE_REQUESTS_PURPOSE, purpose),
expires_after=expires_after,
**data

7
litellm/proxy/schema.prisma
View file

@ -871,6 +871,13 @@ model LiteLLM_GuardrailsTable {
team_id String?
created_at DateTime @default(now())
updated_at DateTime @updatedAt
// Submission lifecycle. Possible values: pending_review (team-registered, awaiting approval), active (approved), rejected
status String @default("active")
submitted_at DateTime?
reviewed_at DateTime?
// submitted_by_user_id and submitted_by_email live in guardrail_info JSON
@@index([status])
}
// Daily guardrail metrics for usage dashboard (one row per guardrail per day)

11
litellm/responses/main.py
View file

@ -745,6 +745,11 @@ def responses(
custom_llm_provider=custom_llm_provider,
)
# Decode any litellm-encoded encrypted-content item IDs back to their original IDs
input = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(
input
)
# Call the handler with _is_async flag instead of directly calling the async handler
response = base_llm_http_handler.response_api_handler(
model=model,
@ -1617,6 +1622,12 @@ def compact_responses(
custom_llm_provider=custom_llm_provider,
)
# Decode any litellm-encoded encrypted-content item IDs back to their original IDs
# before forwarding to the upstream provider.
input = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(
input
)
# Call the handler with _is_async flag instead of directly calling the async handler
response = base_llm_http_handler.compact_response_api_handler(
model=model,

30
litellm/responses/streaming_iterator.py
View file

@ -8,7 +8,10 @@ from typing import Any, Dict, Optional
import httpx
import litellm
from litellm.constants import LITELLM_MAX_STREAMING_DURATION_SECONDS, STREAM_SSE_DONE_STRING
from litellm.constants import (
LITELLM_MAX_STREAMING_DURATION_SECONDS,
STREAM_SSE_DONE_STRING,
)
from litellm.litellm_core_utils.asyncify import run_async_function
from litellm.litellm_core_utils.core_helpers import process_response_headers
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -137,6 +140,31 @@ class BaseResponsesAPIStreamingIterator:
)
setattr(openai_responses_api_chunk, "response", response)
# Wrap encrypted_content in streaming events (output_item.added, output_item.done)
if (
self.litellm_metadata
and self.litellm_metadata.get("encrypted_content_affinity_enabled")
):
event_type = getattr(openai_responses_api_chunk, "type", None)
if event_type in (
ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED,
ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
):
item = getattr(openai_responses_api_chunk, "item", None)
if item:
encrypted_content = getattr(item, "encrypted_content", None)
if encrypted_content and isinstance(encrypted_content, str):
model_id = (
self.litellm_metadata.get("model_info", {}).get("id")
if self.litellm_metadata
else None
)
if model_id:
wrapped_content = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
encrypted_content, model_id
)
setattr(item, "encrypted_content", wrapped_content)
# Store the completed response
if (
openai_responses_api_chunk

196
litellm/responses/utils.py
View file

@ -217,8 +217,204 @@ class ResponsesAPIRequestUtils:
responses_api_response["id"] = updated_id
else:
responses_api_response.id = updated_id
if litellm_metadata.get("encrypted_content_affinity_enabled"):
responses_api_response = (
ResponsesAPIRequestUtils._update_encrypted_content_item_ids_in_response(
response=responses_api_response,
model_id=model_id,
)
)
return responses_api_response
@staticmethod
def _build_encrypted_item_id(model_id: str, item_id: str) -> str:
"""Encode model_id into an output item ID for encrypted-content items.
Format: ``encitem_{base64("litellm:model_id:{model_id};item_id:{original_id}")}``
"""
assembled = f"litellm:model_id:{model_id};item_id:{item_id}"
encoded = base64.b64encode(assembled.encode("utf-8")).decode("utf-8")
return f"encitem_{encoded}"
@staticmethod
def _decode_encrypted_item_id(encoded_id: str) -> Optional[Dict[str, str]]:
"""Decode a litellm-encoded encrypted-content item ID.
Returns a dict with ``model_id`` and ``item_id`` keys, or ``None`` if
the string is not a litellm-encoded item ID.
"""
if not encoded_id.startswith("encitem_"):
return None
try:
cleaned = encoded_id[len("encitem_"):]
# Restore any padding that may have been stripped in transit
missing = len(cleaned) % 4
if missing:
cleaned += "=" * (4 - missing)
decoded = base64.b64decode(cleaned.encode("utf-8")).decode("utf-8")
# Split on first ";" only so that semicolons inside item_id are preserved
parts = decoded.split(";", 1)
if len(parts) < 2:
return None
model_id = parts[0].replace("litellm:model_id:", "")
item_id = parts[1].replace("item_id:", "")
return {"model_id": model_id, "item_id": item_id}
except Exception:
return None
@staticmethod
def _wrap_encrypted_content_with_model_id(
encrypted_content: str, model_id: str
) -> str:
"""Wrap encrypted_content with model_id metadata for affinity routing.
When Codex or other clients send items with encrypted_content but no ID,
we encode the model_id directly into the encrypted_content itself.
Format: ``litellm_enc:{base64("model_id:{model_id}")};{original_encrypted_content}``
"""
metadata = f"model_id:{model_id}"
encoded_metadata = base64.b64encode(metadata.encode("utf-8")).decode("utf-8")
return f"litellm_enc:{encoded_metadata};{encrypted_content}"
@staticmethod
def _unwrap_encrypted_content_with_model_id(
wrapped_content: str,
) -> tuple[Optional[str], str]:
"""Unwrap encrypted_content to extract model_id and original content.
Returns:
Tuple of (model_id, original_encrypted_content).
If not wrapped, returns (None, original_content).
"""
if not wrapped_content.startswith("litellm_enc:"):
return None, wrapped_content
try:
# Split on first ";" to separate metadata from content
parts = wrapped_content.split(";", 1)
if len(parts) < 2:
return None, wrapped_content
metadata_b64 = parts[0].replace("litellm_enc:", "")
original_content = parts[1]
# Restore padding if needed
missing = len(metadata_b64) % 4
if missing:
metadata_b64 += "=" * (4 - missing)
decoded_metadata = base64.b64decode(metadata_b64.encode("utf-8")).decode(
"utf-8"
)
model_id = decoded_metadata.replace("model_id:", "")
return model_id, original_content
except Exception:
return None, wrapped_content
@staticmethod
def _update_encrypted_content_item_ids_in_response(
response: Union["ResponsesAPIResponse", Dict[str, Any]],
model_id: Optional[str],
) -> Union["ResponsesAPIResponse", Dict[str, Any]]:
"""Rewrite item IDs for output items that contain ``encrypted_content``.
Encodes ``model_id`` into the item ID so that follow-up requests can be
routed back to the originating deployment without any cache lookup.
For items without an ID (e.g., from Codex), encodes model_id directly
into the encrypted_content itself.
"""
if not model_id:
return response
output: Optional[list] = None
if isinstance(response, dict):
output = response.get("output")
else:
output = getattr(response, "output", None)
if not isinstance(output, list):
return response
for item in output:
if isinstance(item, dict):
item_id = item.get("id")
encrypted_content = item.get("encrypted_content")
if encrypted_content and isinstance(encrypted_content, str):
# Always wrap encrypted_content with model_id for redundancy
item["encrypted_content"] = (
ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
encrypted_content, model_id
)
)
# Also encode the ID if present
if item_id and isinstance(item_id, str):
item["id"] = ResponsesAPIRequestUtils._build_encrypted_item_id(
model_id, item_id
)
else:
item_id = getattr(item, "id", None)
encrypted_content = getattr(item, "encrypted_content", None)
if encrypted_content and isinstance(encrypted_content, str):
# Always wrap encrypted_content with model_id for redundancy
try:
item.encrypted_content = (
ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
encrypted_content, model_id
)
)
except AttributeError:
pass
# Also encode the ID if present
if item_id and isinstance(item_id, str):
try:
item.id = ResponsesAPIRequestUtils._build_encrypted_item_id(
model_id, item_id
)
except AttributeError:
pass
return response
@staticmethod
def _restore_encrypted_content_item_ids_in_input(request_input: Any) -> Any:
"""Decode litellm-encoded item IDs in request input back to original IDs.
Called before forwarding the request to the upstream provider so the
provider receives the original item IDs and unwrapped encrypted_content.
Handles both:
1. Items with encoded IDs (encitem_...)
2. Items with wrapped encrypted_content (litellm_enc:...)
"""
if not isinstance(request_input, list):
return request_input
for item in request_input:
if isinstance(item, dict):
item_id = item.get("id")
if item_id and isinstance(item_id, str):
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(item_id)
if decoded:
item["id"] = decoded["item_id"]
encrypted_content = item.get("encrypted_content")
if encrypted_content and isinstance(encrypted_content, str):
_, unwrapped = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(
encrypted_content
)
)
if unwrapped != encrypted_content:
item["encrypted_content"] = unwrapped
return request_input
@staticmethod
def _build_responses_api_response_id(
custom_llm_provider: Optional[str],

31
litellm/router.py
View file

@ -115,6 +115,9 @@ from litellm.router_utils.handle_error import (
from litellm.router_utils.pre_call_checks.deployment_affinity_check import (
DeploymentAffinityCheck,
)
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
from litellm.router_utils.pre_call_checks.model_rate_limit_check import (
ModelRateLimitingCheck,
)
@ -1248,6 +1251,26 @@ class Router:
self.optional_callbacks.append(affinity_callback)
litellm.logging_callback_manager.add_litellm_callback(affinity_callback)
# ---------------------------------------------------------------------
# Encrypted content affinity
# ---------------------------------------------------------------------
if "encrypted_content_affinity" in optional_pre_call_checks:
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
if self.optional_callbacks is None:
self.optional_callbacks = []
already_registered = any(
isinstance(cb, EncryptedContentAffinityCheck)
for cb in self.optional_callbacks
)
if not already_registered:
ec_callback = EncryptedContentAffinityCheck()
self.optional_callbacks.append(ec_callback)
litellm.logging_callback_manager.add_litellm_callback(ec_callback)
# ---------------------------------------------------------------------
# Remaining optional pre-call checks
# ---------------------------------------------------------------------
@ -1257,6 +1280,7 @@ class Router:
"deployment_affinity",
"responses_api_deployment_check",
"session_affinity",
"encrypted_content_affinity",
):
continue
if pre_call_check == "prompt_caching":
@ -8808,6 +8832,13 @@ class Router:
if isinstance(healthy_deployments, dict):
return healthy_deployments
# When encrypted content affinity pins to a specific deployment,
if (
request_kwargs.get("_encrypted_content_affinity_pinned")
and len(healthy_deployments) == 1
):
return healthy_deployments[0]
start_time = time.time()
if (
self.routing_strategy == "usage-based-routing-v2"

172
litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py Normal file
View file

@ -0,0 +1,172 @@
"""
Encrypted-content-aware deployment affinity for the Router.
When Codex or other models use `store: false` with `include: ["reasoning.encrypted_content"]`,
the response output items contain encrypted reasoning tokens tied to the originating
organization's API key. If a follow-up request containing those items is routed to a
different deployment (different org), OpenAI rejects it with an `invalid_encrypted_content`
error because the organization_id doesn't match.
This callback solves the problem by encoding the originating deployment's ``model_id``
into the response output items that carry ``encrypted_content``. Two encoding strategies:
1. **Items with IDs**: Encode model_id into the item ID itself (e.g., ``encitem_...``)
2. **Items without IDs** (Codex): Wrap the encrypted_content with model_id metadata
(e.g., ``litellm_enc:{base64_metadata};{original_encrypted_content}``)
The encoded model_id is decoded on the next request so the router can pin to the correct
deployment without any cache lookup.
Response post-processing (encoding) is handled by
``ResponsesAPIRequestUtils._update_encrypted_content_item_ids_in_response`` which is
called inside ``_update_responses_api_response_id_with_model_id`` in ``responses/utils.py``.
Request pre-processing (ID/content restoration before forwarding to upstream) is handled by
``ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input`` which is called
in ``get_optional_params_responses_api``.
This pre-call check is responsible only for the routing decision: it reads the encoded
``model_id`` from either item IDs or wrapped encrypted_content and pins the request to
the matching deployment.
Safe to enable globally:
- Only activates when encoded markers appear in the request ``input``.
- No effect on embedding models, chat completions, or first-time requests.
- No quota reduction -- first requests are fully load balanced.
- No cache required.
"""
from typing import Any, List, Optional, cast
from litellm._logging import verbose_router_logger
from litellm.integrations.custom_logger import CustomLogger, Span
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.types.llms.openai import AllMessageValues
class EncryptedContentAffinityCheck(CustomLogger):
"""
Routes follow-up Responses API requests to the deployment that produced
the encrypted output items they reference.
The ``model_id`` is decoded directly from the litellm-encoded item IDs –
no caching or TTL management needed.
Wired via ``Router(optional_pre_call_checks=["encrypted_content_affinity"])``.
"""
def __init__(self) -> None:
super().__init__()
# ------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------
@staticmethod
def _extract_model_id_from_input(request_input: Any) -> Optional[str]:
"""
Scan ``input`` items for litellm-encoded encrypted-content markers and
return the ``model_id`` embedded in the first one found.
Checks both:
1. Encoded item IDs (encitem_...) - for clients that send IDs
2. Wrapped encrypted_content (litellm_enc:...) - for clients like Codex that don't send IDs
``input`` can be:
- a plain string -> no encoded markers
- a list of items -> check each item's ``id`` and ``encrypted_content`` fields
"""
if not isinstance(request_input, list):
return None
for item in request_input:
if not isinstance(item, dict):
continue
# First, try to decode from item ID (if present)
item_id = item.get("id")
if item_id and isinstance(item_id, str):
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(item_id)
if decoded:
return decoded.get("model_id")
# If no encoded ID, check if encrypted_content itself is wrapped
encrypted_content = item.get("encrypted_content")
if encrypted_content and isinstance(encrypted_content, str):
(
model_id,
_,
) = ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(
encrypted_content
)
if model_id:
return model_id
return None
@staticmethod
def _find_deployment_by_model_id(
healthy_deployments: List[dict], model_id: str
) -> Optional[dict]:
for deployment in healthy_deployments:
model_info = deployment.get("model_info")
if not isinstance(model_info, dict):
continue
deployment_model_id = model_info.get("id")
if deployment_model_id is not None and str(deployment_model_id) == str(
model_id
):
return deployment
return None
# ------------------------------------------------------------------
# Request routing (pre-call filter)
# ------------------------------------------------------------------
async def async_filter_deployments(
self,
model: str,
healthy_deployments: List,
messages: Optional[List[AllMessageValues]],
request_kwargs: Optional[dict] = None,
parent_otel_span: Optional[Span] = None,
) -> List[dict]:
"""
If the request ``input`` contains litellm-encoded item IDs, decode the
embedded ``model_id`` and pin the request to that deployment.
"""
request_kwargs = request_kwargs or {}
typed_healthy_deployments = cast(List[dict], healthy_deployments)
# Signal to the response post-processor that encrypted item IDs should be
# encoded in the output of this request.
litellm_metadata = request_kwargs.setdefault("litellm_metadata", {})
litellm_metadata["encrypted_content_affinity_enabled"] = True
request_input = request_kwargs.get("input")
model_id = self._extract_model_id_from_input(request_input)
if not model_id:
return typed_healthy_deployments
verbose_router_logger.debug(
"EncryptedContentAffinityCheck: decoded model_id=%s from input item IDs",
model_id,
)
deployment = self._find_deployment_by_model_id(
healthy_deployments=typed_healthy_deployments,
model_id=model_id,
)
if deployment is not None:
verbose_router_logger.debug(
"EncryptedContentAffinityCheck: pinning -> deployment=%s",
model_id,
)
request_kwargs["_encrypted_content_affinity_pinned"] = True
return [deployment]
verbose_router_logger.error(
"EncryptedContentAffinityCheck: decoded deployment=%s not found in healthy_deployments",
model_id,
)
return typed_healthy_deployments

11
litellm/types/guardrails.py
View file

@ -698,6 +698,15 @@ class BaseLitellmParams(
),
)
extra_headers: Optional[List[str]] = Field(
default=None,
description=(
"Header names to forward from the client request to the guardrail (e.g. x-request-id). "
"Only these headers' values are sent; others may be omitted or sent as [present]. "
"Used by generic_guardrail_api (similar to MCP extra_headers)."
),
)
# Custom code guardrail params
custom_code: Optional[str] = Field(
default=None,
@ -709,7 +718,7 @@ class BaseLitellmParams(
class Mode(BaseModel):
tags: Dict[str, str] = Field(description="Tags for the guardrail mode")
default: Optional[str] = Field(
default: Optional[Union[str, List[str]]] = Field(
default=None, description="Default mode when no tags match"
)

14
litellm/types/llms/openai.py
View file

@ -71,7 +71,14 @@ from openai.types.responses.response_create_params import (
ToolParam,
)
from openai.types.responses.response_function_tool_call import ResponseFunctionToolCall
from pydantic import BaseModel, ConfigDict, Discriminator, PrivateAttr, field_serializer, field_validator
from pydantic import (
BaseModel,
ConfigDict,
Discriminator,
PrivateAttr,
field_serializer,
field_validator,
)
from typing_extensions import Annotated, Dict, Required, TypedDict, override
from litellm.types.llms.base import BaseLiteLLMOpenAIResponseObject
@ -417,6 +424,7 @@ class CreateBatchRequest(TypedDict, total=False):
endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"]
input_file_id: str
metadata: Optional[Dict[str, str]]
output_expires_after: Optional[FileExpiresAfter]
extra_headers: Optional[Dict[str, str]]
extra_body: Optional[Dict[str, str]]
timeout: Optional[float]
@ -964,6 +972,10 @@ class Hyperparameters(BaseModel):
n_epochs: Optional[Union[str, int]] = (
None # "The number of epochs to train the model for"
)
model_config = {
"extra": "allow"
}
class FineTuningJobCreate(BaseModel):

2
litellm/types/llms/vertex_ai.py
View file

@ -560,7 +560,7 @@ class VertexAIBatchEmbeddingsResponseObject(TypedDict):
class GcsSource(TypedDict):
uris: str
uris: List[str]
class InputConfig(TypedDict):

1
litellm/types/mcp_server/mcp_server_manager.py
View file

@ -53,6 +53,7 @@ class MCPServer(BaseModel):
access_groups: Optional[List[str]] = None
allow_all_keys: bool = False
available_on_public_internet: bool = True
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
model_config = ConfigDict(arbitrary_types_allowed=True)

1
litellm/types/router.py
View file

@ -813,6 +813,7 @@ OptionalPreCallChecks = List[
"session_affinity",
"forward_client_headers_by_model_group",
"enforce_model_rate_limits",
"encrypted_content_affinity",
]
]

2
litellm/utils.py
View file

@ -8326,6 +8326,8 @@ class ProviderConfigManager:
if model and "gpt" in model.lower():
return litellm.DatabricksResponsesAPIConfig()
return None
elif litellm.LlmProviders.OPENROUTER == provider:
return litellm.OpenRouterResponsesAPIConfig()
elif litellm.LlmProviders.HOSTED_VLLM == provider:
return litellm.HostedVLLMResponsesAPIConfig()
return None

740
model_prices_and_context_window.json
View file

@ -9779,6 +9779,122 @@
}
]
},
"dashscope/qwen3-max-2026-01-23": {
"litellm_provider": "dashscope",
"max_input_tokens": 258048,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"source": "https://www.alibabacloud.com/help/en/model-studio/models",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"tiered_pricing": [
{
"input_cost_per_token": 1.2e-06,
"output_cost_per_token": 6e-06,
"range": [
0,
32000.0
]
},
{
"input_cost_per_token": 2.4e-06,
"output_cost_per_token": 1.2e-05,
"range": [
32000.0,
128000.0
]
},
{
"input_cost_per_token": 3e-06,
"output_cost_per_token": 1.5e-05,
"range": [
128000.0,
252000.0
]
}
]
},
"dashscope/qwen3-next-80b-a3b-instruct": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "dashscope",
"max_input_tokens": 262144,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"source": "https://www.alibabacloud.com/help/en/model-studio/model-pricing",
"supports_function_calling": true,
"supports_tool_choice": true
},
"dashscope/qwen3-next-80b-a3b-thinking": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "dashscope",
"max_input_tokens": 262144,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"source": "https://www.alibabacloud.com/help/en/model-studio/model-pricing",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"dashscope/qwen3-vl-235b-a22b-instruct": {
"input_cost_per_token": 4e-07,
"litellm_provider": "dashscope",
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 1.6e-06,
"source": "https://www.alibabacloud.com/help/en/model-studio/model-pricing",
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"dashscope/qwen3-vl-235b-a22b-thinking": {
"input_cost_per_token": 4e-07,
"litellm_provider": "dashscope",
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 4e-06,
"source": "https://www.alibabacloud.com/help/en/model-studio/model-pricing",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": true
},
"dashscope/qwen3-vl-32b-instruct": {
"input_cost_per_token": 1.6e-07,
"litellm_provider": "dashscope",
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 6.4e-07,
"source": "https://www.alibabacloud.com/help/en/model-studio/model-pricing",
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"dashscope/qwen3-vl-32b-thinking": {
"input_cost_per_token": 1.6e-07,
"litellm_provider": "dashscope",
"max_input_tokens": 131072,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 2.87e-06,
"source": "https://www.alibabacloud.com/help/en/model-studio/model-pricing",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": true
},
"dashscope/qwen3-vl-plus": {
"litellm_provider": "dashscope",
"max_input_tokens": 260096,
@ -14334,6 +14450,57 @@
"supports_vision": true,
"supports_web_search": true
},
"gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_per_audio_token": 5e-08,
"input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
"max_images_per_prompt": 3000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_pdf_size_mb": 30,
"max_tokens": 65536,
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
"output_cost_per_reasoning_token": 1.5e-06,
"output_cost_per_token": 1.5e-06,
"source": "https://ai.google.dev/gemini-api/docs/models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_code_execution": true,
"supports_file_search": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_url_context": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true
},
"deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -17090,6 +17257,59 @@
"cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07,
"supports_service_tier": true
},
"gemini/gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_per_audio_token": 5e-08,
"input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
"max_images_per_prompt": 3000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_pdf_size_mb": 30,
"max_tokens": 65536,
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
"output_cost_per_reasoning_token": 1.5e-06,
"output_cost_per_token": 1.5e-06,
"rpm": 15,
"source": "https://ai.google.dev/gemini-api/docs/models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_code_execution": true,
"supports_file_search": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_url_context": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true,
"tpm": 250000
},
"gemini/gemini-3-flash-preview": {
"cache_read_input_token_cost": 5e-08,
"input_cost_per_audio_token": 1e-06,
@ -24226,6 +24446,335 @@
"/v1/images/generations"
]
},
"nebius/deepseek-ai/DeepSeek-R1": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 8e-07,
"output_cost_per_token": 2.4e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_reasoning": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-R1-0528": {
"max_tokens": 164000,
"max_input_tokens": 164000,
"max_output_tokens": 164000,
"input_cost_per_token": 8e-07,
"output_cost_per_token": 2.4e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_reasoning": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-R1-Distill-Llama-70B": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 2.5e-07,
"output_cost_per_token": 7.5e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-V3": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 5e-07,
"output_cost_per_token": 1.5e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/deepseek-ai/DeepSeek-V3-0324": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 5e-07,
"output_cost_per_token": 1.5e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/google/gemma-3-27b-it": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 6e-08,
"output_cost_per_token": 2e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Llama-3.3-70B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Llama-Guard-3-8B": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 2e-08,
"output_cost_per_token": 6e-08,
"litellm_provider": "nebius",
"mode": "chat",
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Meta-Llama-3.1-8B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 2e-08,
"output_cost_per_token": 6e-08,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Meta-Llama-3.1-70B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/meta-llama/Meta-Llama-3.1-405B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1e-06,
"output_cost_per_token": 3e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/mistralai/Mistral-Nemo-Instruct-2407": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 4e-08,
"output_cost_per_token": 1.2e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/NousResearch/Hermes-3-Llama-3.1-405B": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1e-06,
"output_cost_per_token": 3e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/nvidia/Llama-3.1-Nemotron-Ultra-253B-v1": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 6e-07,
"output_cost_per_token": 1.8e-06,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/nvidia/Llama-3.3-Nemotron-Super-49B-v1": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-235B-A22B": {
"max_tokens": 262144,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"input_cost_per_token": 2e-07,
"output_cost_per_token": 6e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-32B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 3e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-30B-A3B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 3e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-14B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 8e-08,
"output_cost_per_token": 2.4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen3-4B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 8e-08,
"output_cost_per_token": 2.4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/QwQ-32B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1.5e-07,
"output_cost_per_token": 4.5e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_reasoning": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-72B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-32B-Instruct": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 6e-08,
"output_cost_per_token": 2e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-Coder-7B": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 3e-08,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2.5-VL-72B-Instruct": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2-VL-72B-Instruct": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 1.3e-07,
"output_cost_per_token": 4e-07,
"litellm_provider": "nebius",
"mode": "chat",
"supports_function_calling": true,
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/Qwen/Qwen2-VL-7B-Instruct": {
"max_tokens": 131072,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"input_cost_per_token": 2e-08,
"output_cost_per_token": 6e-08,
"litellm_provider": "nebius",
"mode": "chat",
"supports_vision": true,
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/BAAI/bge-en-icl": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "nebius",
"mode": "embedding",
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/BAAI/bge-multilingual-gemma2": {
"max_tokens": 8192,
"max_input_tokens": 8192,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "nebius",
"mode": "embedding",
"source": "https://nebius.com/prices-ai-studio"
},
"nebius/intfloat/e5-mistral-7b-instruct": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"input_cost_per_token": 1e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "nebius",
"mode": "embedding",
"source": "https://nebius.com/prices-ai-studio"
},
"nvidia.nemotron-nano-12b-v2": {
"input_cost_per_token": 2e-07,
"litellm_provider": "bedrock_converse",
@ -25373,6 +25922,30 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"openrouter/anthropic/claude-sonnet-4.6": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost": 3e-07,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"litellm_provider": "openrouter",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"source": "https://openrouter.ai/anthropic/claude-sonnet-4.6",
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"openrouter/anthropic/claude-opus-4.5": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_read_input_token_cost": 5e-07,
@ -25723,6 +26296,39 @@
"supports_web_search": true,
"tpm": 800000
},
"openrouter/google/gemini-3.1-pro-preview": {
"cache_read_input_token_cost": 2e-07,
"cache_read_input_token_cost_above_200k_tokens": 4e-07,
"cache_creation_input_token_cost_above_200k_tokens": 2.5e-07,
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_200k_tokens": 4e-06,
"litellm_provider": "openrouter",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_above_200k_tokens": 1.8e-05,
"source": "https://openrouter.ai/google/gemini-3.1-pro-preview",
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
},
"openrouter/gryphe/mythomax-l2-13b": {
"input_cost_per_token": 1.875e-06,
"litellm_provider": "openrouter",
@ -26100,6 +26706,29 @@
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/openai/gpt-5.1-codex-max": {
"cache_read_input_token_cost": 1.25e-07,
"input_cost_per_token": 1.25e-06,
"litellm_provider": "openrouter",
"max_input_tokens": 400000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"source": "https://openrouter.ai/openai/gpt-5.1-codex-max",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": true
},
"openrouter/openai/gpt-5.2": {
"input_cost_per_image": 0,
"cache_read_input_token_cost": 1.75e-07,
@ -26254,6 +26883,19 @@
"supports_tool_choice": true,
"supports_function_calling": true
},
"openrouter/qwen/qwen3-coder-plus": {
"input_cost_per_token": 1e-06,
"litellm_provider": "openrouter",
"max_input_tokens": 997952,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 5e-06,
"source": "https://openrouter.ai/qwen/qwen3-coder-plus",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/qwen/qwen3-235b-a22b-2507": {
"input_cost_per_token": 7.1e-08,
"litellm_provider": "openrouter",
@ -26389,6 +27031,19 @@
"supports_vision": true,
"supports_prompt_caching": false
},
"openrouter/z-ai/glm-5": {
"input_cost_per_token": 8e-07,
"litellm_provider": "openrouter",
"max_input_tokens": 202752,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 2.56e-06,
"source": "https://openrouter.ai/z-ai/glm-5",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/minimax/minimax-m2.1": {
"input_cost_per_token": 2.7e-07,
"output_cost_per_token": 1.2e-06,
@ -27188,7 +27843,7 @@
"supports_function_calling": true
},
"perplexity/pplx-embed-v1-0.6b": {
"input_cost_per_token": 0.000000004,
"input_cost_per_token": 4e-09,
"litellm_provider": "perplexity",
"max_input_tokens": 32768,
"max_tokens": 32768,
@ -27198,7 +27853,7 @@
"source": "https://docs.perplexity.ai/docs/embeddings/quickstart"
},
"perplexity/pplx-embed-v1-4b": {
"input_cost_per_token": 0.00000003,
"input_cost_per_token": 3e-08,
"litellm_provider": "perplexity",
"max_input_tokens": 32768,
"max_tokens": 32768,
@ -31973,6 +32628,57 @@
"output_cost_per_token": 3e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models"
},
"vertex_ai/gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"cache_read_input_token_cost_per_audio_token": 5e-08,
"input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
"max_images_per_prompt": 3000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_pdf_size_mb": 30,
"max_tokens": 65536,
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
"output_cost_per_reasoning_token": 1.5e-06,
"output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"audio",
"video"
],
"supported_output_modalities": [
"text"
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_code_execution": true,
"supports_file_search": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_url_context": true,
"supports_video_input": true,
"supports_vision": true,
"supports_web_search": true,
"supports_native_streaming": true
},
"vertex_ai/deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -33831,6 +34537,36 @@
"supports_tool_choice": true,
"source": "https://aws.amazon.com/bedrock/pricing/"
},
"zai/glm-5": {
"cache_creation_input_token_cost": 0,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 1e-06,
"output_cost_per_token": 3.2e-06,
"litellm_provider": "zai",
"max_input_tokens": 200000,
"max_output_tokens": 128000,
"mode": "chat",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"source": "https://docs.z.ai/guides/overview/pricing"
},
"zai/glm-5-code": {
"cache_creation_input_token_cost": 0,
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 1.2e-06,
"output_cost_per_token": 5e-06,
"litellm_provider": "zai",
"max_input_tokens": 200000,
"max_output_tokens": 128000,
"mode": "chat",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"source": "https://docs.z.ai/guides/overview/pricing"
},
"zai/glm-4.7": {
"cache_creation_input_token_cost": 0,
"cache_read_input_token_cost": 1.1e-07,

4
pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm"
version = "1.82.0"
version = "1.82.1"
description = "Library to easily interface with LLM API providers"
authors = ["BerriAI"]
license = "MIT"
@ -183,7 +183,7 @@ requires = ["poetry-core", "wheel"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "1.82.0"
version = "1.82.1"
version_files = [
"pyproject.toml:^version"
]

7
schema.prisma
View file

@ -871,6 +871,13 @@ model LiteLLM_GuardrailsTable {
team_id String?
created_at DateTime @default(now())
updated_at DateTime @updatedAt
// Submission lifecycle. Possible values: pending_review (team-registered, awaiting approval), active (approved), rejected
status String @default("active")
submitted_at DateTime?
reviewed_at DateTime?
// submitted_by_user_id and submitted_by_email live in guardrail_info JSON
@@index([status])
}
// Daily guardrail metrics for usage dashboard (one row per guardrail per day)

92
scripts/create_team_key_and_submit_guardrail.sh Executable file
View file

@ -0,0 +1,92 @@
#!/usr/bin/env bash
#
# Creates a team, generates a team key, and submits a test guardrail with it.
# Requires: curl, jq
#
# Usage:
# ADMIN_KEY=sk-your-admin-key ./scripts/create_team_key_and_submit_guardrail.sh
# BASE_URL=http://localhost:4000 ADMIN_KEY=sk-your-admin-key ./scripts/create_team_key_and_submit_guardrail.sh
set -e
BASE_URL="${BASE_URL:-http://localhost:4000}"
BASE_URL="${BASE_URL%/}"
if [ -z "${ADMIN_KEY}" ]; then
echo "Error: ADMIN_KEY is required (admin API key for the proxy)."
echo "Usage: ADMIN_KEY=sk-your-admin-key $0"
exit 1
fi
AUTH_HEADER="Authorization: Bearer ${ADMIN_KEY}"
echo "Using BASE_URL=${BASE_URL}"
echo "Creating team..."
TEAM_RESP=$(curl -s -X POST "${BASE_URL}/team/new" \
-H "${AUTH_HEADER}" \
-H "Content-Type: application/json" \
-d '{
"team_alias": "guardrail-test-team"
}')
if ! echo "$TEAM_RESP" | jq -e .team_id >/dev/null 2>&1; then
echo "Failed to create team. Response:"
echo "$TEAM_RESP" | jq . 2>/dev/null || echo "$TEAM_RESP"
exit 1
fi
TEAM_ID=$(echo "$TEAM_RESP" | jq -r .team_id)
echo "Created team_id: ${TEAM_ID}"
echo "Creating key for team..."
KEY_RESP=$(curl -s -X POST "${BASE_URL}/key/generate" \
-H "${AUTH_HEADER}" \
-H "Content-Type: application/json" \
-d "{
\"team_id\": \"${TEAM_ID}\"
}")
if ! echo "$KEY_RESP" | jq -e .key >/dev/null 2>&1; then
echo "Failed to create key. Response:"
echo "$KEY_RESP" | jq . 2>/dev/null || echo "$KEY_RESP"
exit 1
fi
TEAM_KEY=$(echo "$KEY_RESP" | jq -r .key)
echo "Created team key: ${TEAM_KEY}"
GUARDRAIL_NAME="test-guardrail-$(date +%s)"
echo "Submitting guardrail: ${GUARDRAIL_NAME}"
REGISTER_RESP=$(curl -s -X POST "${BASE_URL}/guardrails/register" \
-H "Authorization: Bearer ${TEAM_KEY}" \
-H "Content-Type: application/json" \
-d "{
\"guardrail_name\": \"${GUARDRAIL_NAME}\",
\"litellm_params\": {
\"guardrail\": \"generic_guardrail_api\",
\"mode\": \"pre_call\",
\"api_base\": \"https://example.com/guardrail\"
},
\"guardrail_info\": {
\"description\": \"Test guardrail submitted via team key\"
}
}")
if ! echo "$REGISTER_RESP" | jq -e .guardrail_id >/dev/null 2>&1; then
echo "Failed to register guardrail. Response:"
echo "$REGISTER_RESP" | jq . 2>/dev/null || echo "$REGISTER_RESP"
exit 1
fi
GUARDRAIL_ID=$(echo "$REGISTER_RESP" | jq -r .guardrail_id)
echo "Registered guardrail_id: ${GUARDRAIL_ID}"
echo ""
echo "Done."
echo " team_id: ${TEAM_ID}"
echo " team_key: ${TEAM_KEY}"
echo " guardrail_id: ${GUARDRAIL_ID}"
echo " guardrail_name: ${GUARDRAIL_NAME}"

126
scripts/test_guardrails_register_endpoints.sh Executable file
View file

@ -0,0 +1,126 @@
#!/usr/bin/env bash
#
# Test guardrail register and submissions endpoints.
# Requires: proxy running with DB (migrations applied), valid admin API key.
#
# Usage:
# export LITELLM_API_KEY="sk-..." # required, use an admin key
# ./scripts/test_guardrails_register_endpoints.sh
# BASE_URL=http://localhost:4000 LITELLM_API_KEY="sk-..." ./scripts/test_guardrails_register_endpoints.sh
#
set -euo pipefail
BASE_URL="${BASE_URL:-http://localhost:4000}"
API_KEY="${LITELLM_API_KEY:-}"
if ! command -v jq &>/dev/null; then
echo "Error: jq is required. Install with: brew install jq (macOS) or apt-get install jq (Linux)"
exit 1
fi
if [[ -z "$API_KEY" ]]; then
echo "Error: LITELLM_API_KEY is not set. Use an admin key to test list/approve/reject."
exit 1
fi
AUTH_HEADER="Authorization: Bearer $API_KEY"
TIMESTAMP=$(date +%s)
NAME_APPROVE="test-guardrail-approve-$TIMESTAMP"
NAME_REJECT="test-guardrail-reject-$TIMESTAMP"
echo "BASE_URL=$BASE_URL"
echo "Testing guardrail register and submissions endpoints..."
echo ""
# --- 1. Register a guardrail (will approve later) ---
echo "[1/6] POST /guardrails/register (guardrail: $NAME_APPROVE)"
REGISTER_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$BASE_URL/guardrails/register" \
-H "$AUTH_HEADER" \
-H "Content-Type: application/json" \
-d "{
\"guardrail_name\": \"$NAME_APPROVE\",
\"litellm_params\": {
\"guardrail\": \"generic_guardrail_api\",
\"mode\": \"pre_call\",
\"api_base\": \"https://guardrails.example.com/validate\"
},
\"guardrail_info\": { \"description\": \"Test guardrail for approve flow\" }
}")
REGISTER_HTTP=$(echo "$REGISTER_RESPONSE" | tail -n1)
REGISTER_BODY=$(echo "$REGISTER_RESPONSE" | sed '$d')
if [[ "$REGISTER_HTTP" -ne 200 ]]; then
echo " FAIL: expected 200, got $REGISTER_HTTP"
echo "$REGISTER_BODY" | jq . 2>/dev/null || echo "$REGISTER_BODY"
exit 1
fi
GUARDRAIL_ID_APPROVE=$(echo "$REGISTER_BODY" | jq -r '.guardrail_id')
echo " OK (201/200) guardrail_id=$GUARDRAIL_ID_APPROVE"
# --- 2. Register a second guardrail (will reject later) ---
echo "[2/6] POST /guardrails/register (guardrail: $NAME_REJECT)"
REJECT_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$BASE_URL/guardrails/register" \
-H "$AUTH_HEADER" \
-H "Content-Type: application/json" \
-d "{
\"guardrail_name\": \"$NAME_REJECT\",
\"litellm_params\": {
\"guardrail\": \"generic_guardrail_api\",
\"mode\": \"post_call\",
\"api_base\": \"https://guardrails.example.com/reject-test\"
},
\"guardrail_info\": { \"description\": \"Test guardrail for reject flow\" }
}")
REJECT_HTTP=$(echo "$REJECT_RESPONSE" | tail -n1)
if [[ "$REJECT_HTTP" -ne 200 ]]; then
echo " FAIL: expected 200, got $REJECT_HTTP"
echo "$REJECT_RESPONSE" | sed '$d' | jq . 2>/dev/null || echo "$REJECT_RESPONSE"
exit 1
fi
GUARDRAIL_ID_REJECT=$(echo "$REJECT_RESPONSE" | sed '$d' | jq -r '.guardrail_id')
echo " OK guardrail_id=$GUARDRAIL_ID_REJECT"
# --- 3. List submissions (admin) ---
echo "[3/6] GET /guardrails/submissions"
LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET "$BASE_URL/guardrails/submissions" -H "$AUTH_HEADER")
LIST_HTTP=$(echo "$LIST_RESPONSE" | tail -n1)
LIST_BODY=$(echo "$LIST_RESPONSE" | sed '$d')
if [[ "$LIST_HTTP" -ne 200 ]]; then
echo " FAIL: expected 200, got $LIST_HTTP"
echo "$LIST_BODY" | jq . 2>/dev/null || echo "$LIST_BODY"
exit 1
fi
echo " OK summary: $(echo "$LIST_BODY" | jq -c '.summary' 2>/dev/null || echo "N/A")"
# --- 4. Get one submission by id ---
echo "[4/6] GET /guardrails/submissions/$GUARDRAIL_ID_APPROVE"
GET_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET "$BASE_URL/guardrails/submissions/$GUARDRAIL_ID_APPROVE" -H "$AUTH_HEADER")
GET_HTTP=$(echo "$GET_RESPONSE" | tail -n1)
if [[ "$GET_HTTP" -ne 200 ]]; then
echo " FAIL: expected 200, got $GET_HTTP"
exit 1
fi
echo " OK status=$(echo "$GET_RESPONSE" | sed '$d' | jq -r '.status')"
# --- 5. Approve first submission ---
echo "[5/6] POST /guardrails/submissions/$GUARDRAIL_ID_APPROVE/approve"
APPROVE_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$BASE_URL/guardrails/submissions/$GUARDRAIL_ID_APPROVE/approve" -H "$AUTH_HEADER")
APPROVE_HTTP=$(echo "$APPROVE_RESPONSE" | tail -n1)
if [[ "$APPROVE_HTTP" -ne 200 ]]; then
echo " FAIL: expected 200, got $APPROVE_HTTP"
echo "$APPROVE_RESPONSE" | sed '$d' | jq . 2>/dev/null || echo "$APPROVE_RESPONSE"
exit 1
fi
echo " OK $(echo "$APPROVE_RESPONSE" | sed '$d' | jq -c '.' 2>/dev/null)"
# --- 6. Reject second submission ---
echo "[6/6] POST /guardrails/submissions/$GUARDRAIL_ID_REJECT/reject"
REJECT_POST_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$BASE_URL/guardrails/submissions/$GUARDRAIL_ID_REJECT/reject" -H "$AUTH_HEADER")
REJECT_POST_HTTP=$(echo "$REJECT_POST_RESPONSE" | tail -n1)
if [[ "$REJECT_POST_HTTP" -ne 200 ]]; then
echo " FAIL: expected 200, got $REJECT_POST_HTTP"
exit 1
fi
echo " OK $(echo "$REJECT_POST_RESPONSE" | sed '$d' | jq -c '.' 2>/dev/null)"
echo ""
echo "All 6 requests succeeded. Guardrail register and submissions endpoints are working."

58
tests/batches_tests/test_fine_tuning_api.py
View file

@ -596,3 +596,61 @@ async def test_mock_openai_retrieve_fine_tune_job():
# Verify the request
mock_retrieve.assert_called_once_with(fine_tuning_job_id="ft-123")
@pytest.mark.asyncio
async def test_mock_azure_create_fine_tune_job_with_azure_specific_params():
"""Test that Azure-specific parameters are passed through extra_body"""
from openai import AsyncAzureOpenAI
from openai.types.fine_tuning.fine_tuning_job import FineTuningJob
from openai.types.fine_tuning.fine_tuning_job import Hyperparameters as OAIHyperparameters
mock_response = FineTuningJob(
id="ft-azure-123",
model="gpt-4.1-mini-2025-04-14",
created_at=1677610602,
status="validating_files",
fine_tuned_model=None,
object="fine_tuning.job",
hyperparameters=OAIHyperparameters(n_epochs=3),
organization_id="org-123",
seed=42,
training_file="file-123",
result_files=[],
)
with patch("litellm.llms.azure.fine_tuning.handler.AzureOpenAIFineTuningAPI.create_fine_tuning_job") as mock_create:
mock_create.return_value = mock_response
response = await litellm.acreate_fine_tuning_job(
model="gpt-4.1-mini-2025-04-14",
training_file="file-123",
custom_llm_provider="azure",
api_base="https://test.openai.azure.com",
api_key="test-key",
api_version="2025-04-01-preview",
trainingType=1,
hyperparameters={
"n_epochs": 3,
"prompt_loss_weight": 0.1
},
)
# Verify the request
mock_create.assert_called_once()
request_params = mock_create.call_args.kwargs
# Check that create_fine_tuning_job_data contains the correct structure
create_data = request_params["create_fine_tuning_job_data"]
assert create_data["model"] == "gpt-4.1-mini-2025-04-14"
assert create_data["training_file"] == "file-123"
assert create_data["hyperparameters"] == {"n_epochs": 3}
# Azure-specific parameters should be in extra_body
assert "extra_body" in create_data
assert create_data["extra_body"]["trainingType"] == 1
assert create_data["extra_body"]["prompt_loss_weight"] == 0.1
# Verify the response
assert response.id == "ft-azure-123"
assert response.model == "gpt-4.1-mini-2025-04-14"

43
tests/batches_tests/test_openai_batches_and_files.py
View file

@ -29,6 +29,7 @@ verbose_logger.setLevel(logging.DEBUG)
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.utils import StandardLoggingPayload
import random
import httpx
from unittest.mock import patch, MagicMock
@ -579,6 +580,48 @@ async def test_vertex_list_batches(monkeypatch):
assert list_response["data"][1].id == "test-batch-id-789"
@pytest.mark.asyncio
async def test_vertex_async_create_batch_logs_error_body_on_http_error():
"""
When Vertex AI returns an HTTP error (e.g. 400), _async_create_batch should
re-raise httpx.HTTPStatusError (not swallow it) and log the response body.
Before the fix the error body was lost because AsyncHTTPHandler.post()
calls raise_for_status() internally, raising before the handler's own
status-code check could log the body.
"""
from litellm.llms.vertex_ai.batches.handler import VertexAIBatchPrediction
handler = VertexAIBatchPrediction(gcs_bucket_name="test-bucket")
error_body = '{"error": {"code": 400, "message": "Do not support publisher model gemini-2.0-flash"}}'
mock_response = MagicMock(spec=httpx.Response)
mock_response.status_code = 400
mock_response.text = error_body
mock_response.headers = {}
http_error = httpx.HTTPStatusError(
message="Bad Request",
request=httpx.Request("POST", "https://fake-vertex-url"),
response=mock_response,
)
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
side_effect=http_error,
):
with pytest.raises(httpx.HTTPStatusError) as exc_info:
await handler._async_create_batch(
vertex_batch_request={},
api_base="https://us-central1-aiplatform.googleapis.com/v1/projects/test/locations/us-central1/batchPredictionJobs",
headers={"Authorization": "Bearer fake-token"},
)
assert exc_info.value.response.status_code == 400
assert "gemini-2.0-flash" in exc_info.value.response.text
@pytest.mark.asyncio
async def test_delete_batch_output_file():
"""

130
tests/enterprise/litellm_enterprise/integrations/test_custom_guardrail.py
View file

@ -1,9 +1,5 @@
import datetime
import json
import os
import sys
import unittest
from unittest.mock import ANY, MagicMock, patch
sys.path.insert(
0, os.path.abspath("../..")
@ -12,6 +8,132 @@ from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.types.guardrails import GuardrailEventHooks, Mode
def test_custom_guardrail_with_mode_default_list(monkeypatch):
"""Test Mode with default as a list of modes (e.g. default: ["pre_call", "post_call"])"""
monkeypatch.setattr("litellm.proxy.proxy_server.premium_user", True)
cg = CustomGuardrail(
guardrail_name="test_guardrail",
supported_event_hooks=[
GuardrailEventHooks.pre_call,
GuardrailEventHooks.post_call,
GuardrailEventHooks.logging_only,
],
event_hook=Mode(
tags={"test_tag": "logging_only"},
default=["pre_call", "post_call"],
),
default_on=True,
)
# No tag match → default fires for pre_call
assert (
cg.should_run_guardrail(
data={"messages": [{"role": "user", "content": "test"}]},
event_type=GuardrailEventHooks.pre_call,
)
is True
)
# No tag match → default fires for post_call
assert (
cg.should_run_guardrail(
data={"messages": [{"role": "user", "content": "test"}]},
event_type=GuardrailEventHooks.post_call,
)
is True
)
# No tag match → logging_only NOT in default list, should not fire
assert (
cg.should_run_guardrail(
data={"messages": [{"role": "user", "content": "test"}]},
event_type=GuardrailEventHooks.logging_only,
)
is False
)
# Tag matches → only logging_only should fire
assert (
cg.should_run_guardrail(
data={
"messages": [{"role": "user", "content": "test"}],
"litellm_metadata": {"tags": ["test_tag"]},
},
event_type=GuardrailEventHooks.logging_only,
)
is True
)
# Tag matches → pre_call should NOT fire (tag says logging_only)
assert (
cg.should_run_guardrail(
data={
"messages": [{"role": "user", "content": "test"}],
"litellm_metadata": {"tags": ["test_tag"]},
},
event_type=GuardrailEventHooks.pre_call,
)
is False
)
# Tag matches → post_call should NOT fire (tag says logging_only)
assert (
cg.should_run_guardrail(
data={
"messages": [{"role": "user", "content": "test"}],
"litellm_metadata": {"tags": ["test_tag"]},
},
event_type=GuardrailEventHooks.post_call,
)
is False
)
def test_custom_guardrail_with_mode_no_default(monkeypatch):
"""Test Mode with no default — guardrail only fires when tag matches"""
monkeypatch.setattr("litellm.proxy.proxy_server.premium_user", True)
cg = CustomGuardrail(
guardrail_name="test_guardrail",
supported_event_hooks=[
GuardrailEventHooks.pre_call,
GuardrailEventHooks.logging_only,
],
event_hook=Mode(
tags={"test_tag": "logging_only"},
),
default_on=True,
)
# No tag, no default → nothing fires
assert (
cg.should_run_guardrail(
data={"messages": [{"role": "user", "content": "test"}]},
event_type=GuardrailEventHooks.pre_call,
)
is False
)
assert (
cg.should_run_guardrail(
data={"messages": [{"role": "user", "content": "test"}]},
event_type=GuardrailEventHooks.logging_only,
)
is False
)
# Tag matches → only logging_only fires
assert (
cg.should_run_guardrail(
data={
"messages": [{"role": "user", "content": "test"}],
"litellm_metadata": {"tags": ["test_tag"]},
},
event_type=GuardrailEventHooks.logging_only,
)
is True
)
def test_custom_guardrail_with_mode(monkeypatch):
monkeypatch.setattr(
"litellm.proxy.proxy_server.premium_user", True

104
tests/enterprise/litellm_enterprise/proxy/hooks/test_managed_files.py
View file

@ -6,6 +6,7 @@ from fastapi import HTTPException
from litellm_enterprise.proxy.hooks.managed_files import _PROXY_LiteLLMManagedFiles
from litellm.caching import DualCache
from litellm.proxy._types import CallTypes
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
)
@ -61,6 +62,109 @@ async def test_async_pre_call_hook_batch_retrieve():
assert response["model"] == "my-general-azure-deployment"
@pytest.mark.asyncio
async def test_async_pre_call_deployment_hook_resolves_model_id_from_litellm_metadata():
"""
For batch operations the router stores model_info under
kwargs["litellm_metadata"]["model_info"] (not top-level kwargs["model_info"]).
async_pre_call_deployment_hook must check both locations so the managed
file ID is resolved to the provider-specific file ID.
"""
proxy_managed_files = _PROXY_LiteLLMManagedFiles(
DualCache(), prisma_client=MagicMock()
)
managed_file_id = "managed-file-abc"
model_id = "deployment-xyz"
provider_file_id = "gs://bucket/path/to/file.jsonl"
# model_info is nested under litellm_metadata (batch path)
kwargs = {
"input_file_id": managed_file_id,
"model_file_id_mapping": {
managed_file_id: {model_id: provider_file_id},
},
"litellm_metadata": {
"model_info": {"id": model_id},
},
}
result = await proxy_managed_files.async_pre_call_deployment_hook(
kwargs=kwargs, call_type=CallTypes.acreate_batch
)
assert result["input_file_id"] == provider_file_id, (
f"Expected provider file ID '{provider_file_id}', got '{result['input_file_id']}'"
)
@pytest.mark.asyncio
async def test_async_pre_call_deployment_hook_prefers_top_level_model_info():
"""
When model_info exists at top-level kwargs, async_pre_call_deployment_hook
should use it without falling back to litellm_metadata.
"""
proxy_managed_files = _PROXY_LiteLLMManagedFiles(
DualCache(), prisma_client=MagicMock()
)
managed_file_id = "managed-file-abc"
top_level_model_id = "deployment-top"
nested_model_id = "deployment-nested"
top_level_provider_file = "file-top-123"
nested_provider_file = "file-nested-456"
kwargs = {
"input_file_id": managed_file_id,
"model_file_id_mapping": {
managed_file_id: {
top_level_model_id: top_level_provider_file,
nested_model_id: nested_provider_file,
},
},
"model_info": {"id": top_level_model_id},
"litellm_metadata": {
"model_info": {"id": nested_model_id},
},
}
result = await proxy_managed_files.async_pre_call_deployment_hook(
kwargs=kwargs, call_type=CallTypes.acreate_batch
)
assert result["input_file_id"] == top_level_provider_file, (
"Should prefer top-level model_info over litellm_metadata"
)
@pytest.mark.asyncio
async def test_async_pre_call_deployment_hook_no_model_info_leaves_file_id_unchanged():
"""
When model_info is absent from both top-level and litellm_metadata,
the managed file ID should remain unchanged.
"""
proxy_managed_files = _PROXY_LiteLLMManagedFiles(
DualCache(), prisma_client=MagicMock()
)
managed_file_id = "managed-file-abc"
kwargs = {
"input_file_id": managed_file_id,
"model_file_id_mapping": {
managed_file_id: {"some-model": "provider-file-xyz"},
},
}
result = await proxy_managed_files.async_pre_call_deployment_hook(
kwargs=kwargs, call_type=CallTypes.acreate_batch
)
assert result["input_file_id"] == managed_file_id, (
"File ID should remain unchanged when model_info is not available"
)
# def test_list_managed_files():
# proxy_managed_files = _PROXY_LiteLLMManagedFiles(DualCache())

3
tests/proxy_unit_tests/test_proxy_utils.py
View file

@ -8,6 +8,7 @@ from unittest.mock import Mock
import pytest
from fastapi import Request
from starlette.datastructures import State
from litellm.proxy.utils import _get_docs_url, _get_redoc_url
@ -32,6 +33,7 @@ def mock_request(monkeypatch):
mock_request = Mock(spec=Request)
mock_request.query_params = {} # Set mock query_params to an empty dictionary
mock_request.headers = {"traceparent": "test_traceparent"}
mock_request.state = State() # Real State so _safe_get_request_headers caching works
monkeypatch.setattr(
"litellm.proxy.litellm_pre_call_utils.add_litellm_data_to_request", mock_request
)
@ -810,6 +812,7 @@ async def test_add_litellm_data_to_request_duplicate_tags(
mock_request.url.path = "/chat/completions"
mock_request.query_params = {}
mock_request.headers = {}
mock_request.state = State()
# Setup key with tags in metadata
user_api_key_dict = UserAPIKeyAuth(

48
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py
View file

@ -33,8 +33,8 @@ sys.path.insert(
) # Adds the parent directory to the system path
from litellm.litellm_core_utils.llm_cost_calc.utils import (
_calculate_input_cost,
PromptTokensDetailsResult,
_calculate_input_cost,
calculate_cache_writing_cost,
generic_cost_per_token,
)
@ -127,6 +127,52 @@ def test_reasoning_tokens_gemini():
)
def test_reasoning_tokens_gemini_3_1_flash_lite():
"""Test cost calculation for gemini-3.1-flash-lite-preview with reasoning tokens"""
model = "gemini-3.1-flash-lite-preview"
custom_llm_provider = "gemini"
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
litellm.model_cost = litellm.get_model_cost_map(url="")
usage = Usage(
completion_tokens=1000,
prompt_tokens=500,
total_tokens=1500,
completion_tokens_details=CompletionTokensDetailsWrapper(
accepted_prediction_tokens=None,
audio_tokens=None,
reasoning_tokens=400,
rejected_prediction_tokens=None,
text_tokens=600,
),
prompt_tokens_details=PromptTokensDetailsWrapper(
audio_tokens=None, cached_tokens=None, text_tokens=500, image_tokens=None
),
)
model_cost_map = litellm.model_cost[model]
prompt_cost, completion_cost = generic_cost_per_token(
model=model,
usage=usage,
custom_llm_provider=custom_llm_provider,
)
assert round(prompt_cost, 10) == round(
model_cost_map["input_cost_per_token"] * usage.prompt_tokens,
10,
)
assert round(completion_cost, 10) == round(
(
model_cost_map["output_cost_per_token"]
* usage.completion_tokens_details.text_tokens
)
+ (
model_cost_map["output_cost_per_reasoning_token"]
* usage.completion_tokens_details.reasoning_tokens
),
10,
)
def test_image_tokens_with_custom_pricing():
"""Test that image_tokens in completion are properly costed with output_cost_per_image_token."""
from unittest.mock import patch

14
tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_content_after_stop_reason.py
View file

@ -23,7 +23,7 @@ sys.path.insert(0, os.path.abspath("../../../../.."))
from litellm.llms.anthropic.experimental_pass_through.adapters.streaming_iterator import (
AnthropicStreamWrapper,
)
from litellm.types.utils import Delta, ModelResponse, StreamingChoices, Usage
from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices, Usage
class MockCompletionStreamWithContentAfterStopReason:
@ -32,16 +32,14 @@ class MockCompletionStreamWithContentAfterStopReason:
def __init__(self):
self.responses = [
# Initial text content
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content="Hello"), index=0, finish_reason=None
)
],
),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=" world"), index=0, finish_reason=None
@ -49,8 +47,7 @@ class MockCompletionStreamWithContentAfterStopReason:
],
),
# Message delta with stop_reason AND usage (this is how it actually comes from the API)
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=""), index=0, finish_reason="stop"
@ -60,8 +57,7 @@ class MockCompletionStreamWithContentAfterStopReason:
),
# Additional content after the stop_reason - this simulates the scenario
# where there might be additional content blocks after the main response
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=" Additional content"),

26
tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_parallel_tool_calls.py
View file

@ -10,7 +10,7 @@ from litellm.llms.anthropic.experimental_pass_through.adapters.streaming_iterato
)
from litellm.types.utils import (
Delta,
ModelResponse,
ModelResponseStream,
StreamingChoices,
Usage,
ChatCompletionDeltaToolCall,
@ -19,7 +19,7 @@ from litellm.types.utils import (
class MockCompletionStream:
def __init__(self, responses: List[ModelResponse]):
def __init__(self, responses: List[ModelResponseStream]):
self.responses = responses
self.index = 0
@ -44,9 +44,8 @@ class MockCompletionStream:
return response
def construct_text_chunk(text: str) -> ModelResponse:
return ModelResponse(
stream=True,
def construct_text_chunk(text: str) -> ModelResponseStream:
return ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=text),
@ -59,11 +58,10 @@ def construct_text_chunk(text: str) -> ModelResponse:
def construct_split_tool_call(
id: str, function_name: str, function_arg_parts: List[str]
) -> List[ModelResponse]:
) -> List[ModelResponseStream]:
return [
# https://platform.openai.com/docs/guides/function-calling#streaming
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(
@ -82,8 +80,7 @@ def construct_split_tool_call(
],
),
*[
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(
@ -109,8 +106,7 @@ def construct_split_tool_call(
def test_anthropic_stream_wrapper_single_tool_call():
responses = [
*construct_split_tool_call("tooluse_foo", "get_weather", ['{"city":', '"NY"}']),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content="", stop_reason="tool_calls"),
@ -172,8 +168,7 @@ def test_anthropic_stream_wrapper_back_to_back_tool_calls():
responses = [
*construct_split_tool_call("tooluse_foo", "get_weather", ['{"city":', '"NY"}']),
*construct_split_tool_call("tooluse_bar", "get_weather", ['{"city":', '"SF"}']),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content="", stop_reason="tool_calls"),
@ -244,8 +239,7 @@ def test_anthropic_stream_wrapper_interleaved_tool_calls_and_text():
"tooluse_bar", "get_weather", ['{"city":', '"CHI"}']
),
construct_text_chunk("The weather is not so nice today."),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content="", stop_reason="tool_calls"),

17
tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_sse_wrapper.py
View file

@ -9,31 +9,28 @@ sys.path.insert(0, os.path.abspath("../../../../.."))
from litellm.llms.anthropic.experimental_pass_through.adapters.streaming_iterator import (
AnthropicStreamWrapper,
)
from litellm.types.utils import Delta, ModelResponse, StreamingChoices
from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices
# Create a simple test
class MockCompletionStream:
def __init__(self):
self.responses = [
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content="Hello"), index=0, finish_reason=None
)
],
),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=" World"), index=0, finish_reason=None
)
],
),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=""), index=0, finish_reason="stop"
@ -109,16 +106,14 @@ async def test_async_anthropic_sse_wrapper():
class AsyncMockCompletionStream:
def __init__(self):
self.responses = [
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content="Hello"), index=0, finish_reason=None
)
],
),
ModelResponse(
stream=True,
ModelResponseStream(
choices=[
StreamingChoices(
delta=Delta(content=" World"), index=0, finish_reason=None

57
tests/test_litellm/llms/azure/test_azure_exception_mapping.py
View file

@ -384,4 +384,59 @@ class TestAzureExceptionMapping:
model="azure/dall-e-3",
original_exception=mock_exception,
custom_llm_provider="azure",
)
)
def test_invalid_encrypted_content_error_with_helpful_message(self):
"""Test that invalid_encrypted_content errors include helpful guidance
about enabling encrypted_content_affinity."""
from litellm.exceptions import BadRequestError
mock_exception = Exception(
"The encrypted content gAAAAABpnW_yEYmSNEyOG... could not be verified. "
"Reason: Encrypted content organization_id did not match the target organization."
)
mock_exception.body = {
"error": {
"message": "The encrypted content could not be verified.",
"type": "invalid_request_error",
"code": "invalid_encrypted_content",
}
}
mock_response = MagicMock()
mock_response.status_code = 400
mock_exception.response = mock_response
with pytest.raises(BadRequestError) as exc_info:
exception_type(
model="azure/gpt-5.1-codex",
original_exception=mock_exception,
custom_llm_provider="azure",
)
error = exc_info.value
assert "encrypted_content_affinity" in error.message
assert "enable_pre_call_checks" in error.message
assert "optional_pre_call_checks" in error.message
assert "docs.litellm.ai" in error.message
def test_openai_invalid_encrypted_content_error(self):
"""Test that OpenAI invalid_encrypted_content errors also get helpful guidance."""
from litellm.exceptions import BadRequestError
mock_exception = Exception(
"The encrypted content could not be verified."
)
mock_response = MagicMock()
mock_response.status_code = 400
mock_exception.response = mock_response
with pytest.raises(BadRequestError) as exc_info:
exception_type(
model="gpt-5.1-codex",
original_exception=mock_exception,
custom_llm_provider="openai",
)
error = exc_info.value
assert "encrypted_content_affinity" in error.message
assert "enable_pre_call_checks" in error.message

112
tests/test_litellm/llms/openrouter/responses/test_openrouter_responses_transformation.py Normal file
View file

@ -0,0 +1,112 @@
"""
Tests for OpenRouter Responses API configuration.
Validates that OpenRouter is registered as a native Responses API provider,
routing requests directly to https://openrouter.ai/api/v1/responses instead
of falling back to the chat completion bridge. This is required to preserve
reasoning.encrypted_content for multi-turn stateless workflows.
Related issue: https://github.com/BerriAI/litellm/issues/22189
"""
import litellm
from litellm.llms.openrouter.responses.transformation import (
OpenRouterResponsesAPIConfig,
)
from litellm.types.utils import LlmProviders
from litellm.utils import ProviderConfigManager
class TestOpenRouterResponsesAPIConfig:
"""Test OpenRouter Responses API configuration."""
def test_custom_llm_provider(self):
"""custom_llm_provider should return OPENROUTER."""
config = OpenRouterResponsesAPIConfig()
assert config.custom_llm_provider == LlmProviders.OPENROUTER
def test_get_complete_url_default(self):
"""Default URL should point to OpenRouter's Responses API endpoint."""
config = OpenRouterResponsesAPIConfig()
url = config.get_complete_url(api_base=None, litellm_params={})
assert url == "https://openrouter.ai/api/v1/responses"
def test_get_complete_url_custom_base(self):
"""Custom api_base should be respected."""
config = OpenRouterResponsesAPIConfig()
url = config.get_complete_url(
api_base="https://custom.openrouter.ai/api/v1",
litellm_params={},
)
assert url == "https://custom.openrouter.ai/api/v1/responses"
def test_get_complete_url_strips_trailing_slash(self):
"""Trailing slashes on api_base should be stripped."""
config = OpenRouterResponsesAPIConfig()
url = config.get_complete_url(
api_base="https://openrouter.ai/api/v1/",
litellm_params={},
)
assert url == "https://openrouter.ai/api/v1/responses"
def test_validate_environment_sets_auth_header(self):
"""validate_environment should set the Authorization header."""
config = OpenRouterResponsesAPIConfig()
from litellm.types.router import GenericLiteLLMParams
params = GenericLiteLLMParams(api_key="sk-or-test-key")
headers = config.validate_environment(
headers={}, model="openai/o4-mini", litellm_params=params
)
assert headers["Authorization"] == "Bearer sk-or-test-key"
def test_validate_environment_raises_without_key(self):
"""validate_environment should raise when no API key is available."""
config = OpenRouterResponsesAPIConfig()
from litellm.types.router import GenericLiteLLMParams
try:
config.validate_environment(
headers={},
model="openai/o4-mini",
litellm_params=GenericLiteLLMParams(),
)
assert False, "Should have raised ValueError"
except ValueError as e:
assert "OpenRouter API key is required" in str(e)
class TestOpenRouterResponsesAPIRegistration:
"""Test that OpenRouter is properly registered as a native Responses API provider."""
def test_provider_config_manager_returns_openrouter_config(self):
"""
ProviderConfigManager.get_provider_responses_api_config should return
OpenRouterResponsesAPIConfig for the OPENROUTER provider, NOT None.
When it returns None, requests fall through to the completion bridge,
which loses encrypted_content (the bug in issue #22189).
"""
config = ProviderConfigManager.get_provider_responses_api_config(
provider=LlmProviders.OPENROUTER,
)
assert config is not None, (
"OpenRouter must be registered as a native Responses API provider "
"to preserve reasoning.encrypted_content"
)
assert isinstance(config, OpenRouterResponsesAPIConfig)
def test_openrouter_not_using_completion_bridge(self):
"""
Verify that OpenRouter does NOT fall through to the completion bridge.
The completion bridge drops encrypted_content because chat completions
use a different format (reasoning_details) than the Responses API.
"""
config = ProviderConfigManager.get_provider_responses_api_config(
provider=LlmProviders.OPENROUTER,
)
# If config is not None, the native Responses API path is used
assert config is not None
# The URL should point to OpenRouter's responses endpoint
url = config.get_complete_url(api_base=None, litellm_params={})
assert "/responses" in url

184
tests/test_litellm/llms/vertex_ai/files/test_vertex_ai_files_transformation.py Normal file
View file

@ -0,0 +1,184 @@
"""
Tests for VertexAIFilesConfig transformation methods (Issues 5-7).
"""
import json
import urllib.parse
import httpx
import pytest
from unittest.mock import MagicMock
from litellm.llms.vertex_ai.files.transformation import VertexAIFilesConfig
from litellm.types.llms.openai import OpenAIFileObject, HttpxBinaryResponseContent
from openai.types.file_deleted import FileDeleted
@pytest.fixture
def config():
return VertexAIFilesConfig()
class TestParseGcsUri:
"""Tests for the _parse_gcs_uri helper used by retrieve / content / delete."""
def test_should_parse_standard_gs_uri(self, config):
bucket, encoded = config._parse_gcs_uri(
"gs://my-bucket/path/to/object.jsonl"
)
assert bucket == "my-bucket"
assert encoded == urllib.parse.quote("path/to/object.jsonl", safe="")
def test_should_parse_uri_with_nested_publisher_path(self, config):
uri = "gs://litellm-local/litellm-vertex-files/publishers/google/models/gemini-2.0-flash-001/abc-123"
bucket, encoded = config._parse_gcs_uri(uri)
assert bucket == "litellm-local"
expected_path = "litellm-vertex-files/publishers/google/models/gemini-2.0-flash-001/abc-123"
assert encoded == urllib.parse.quote(expected_path, safe="")
def test_should_handle_url_encoded_input(self, config):
encoded_uri = urllib.parse.quote("gs://my-bucket/some/path", safe="")
bucket, encoded = config._parse_gcs_uri(encoded_uri)
assert bucket == "my-bucket"
assert encoded == urllib.parse.quote("some/path", safe="")
def test_should_handle_bucket_only(self, config):
bucket, encoded = config._parse_gcs_uri("gs://my-bucket")
assert bucket == "my-bucket"
assert encoded == ""
def test_should_handle_no_gs_prefix(self, config):
bucket, encoded = config._parse_gcs_uri("my-bucket/object.txt")
assert bucket == "my-bucket"
assert encoded == "object.txt"
class TestTransformRetrieveFile:
def test_should_build_correct_gcs_metadata_url(self, config):
file_id = "gs://my-bucket/path/to/file.jsonl"
url, params = config.transform_retrieve_file_request(
file_id=file_id, optional_params={}, litellm_params={}
)
expected_encoded = urllib.parse.quote("path/to/file.jsonl", safe="")
assert url == f"https://storage.googleapis.com/storage/v1/b/my-bucket/o/{expected_encoded}"
assert params == {}
def test_should_return_openai_file_object_from_gcs_response(self, config):
gcs_json = {
"id": "my-bucket/path/to/file.jsonl/123456",
"name": "path/to/file.jsonl",
"size": "4096",
"timeCreated": "2025-02-15T10:00:00.000Z",
"metadata": {"purpose": "batch"},
}
raw_response = MagicMock(spec=httpx.Response)
raw_response.json.return_value = gcs_json
result = config.transform_retrieve_file_response(
raw_response=raw_response,
logging_obj=MagicMock(),
litellm_params={},
)
assert isinstance(result, OpenAIFileObject)
assert result.id == "gs://my-bucket/path/to/file.jsonl"
assert result.filename == "path/to/file.jsonl"
assert result.bytes == 4096
assert result.object == "file"
assert result.status == "processed"
assert result.purpose == "batch"
def test_should_default_purpose_to_batch_when_metadata_missing(self, config):
gcs_json = {
"id": "bucket/obj/999",
"name": "obj",
"size": "0",
"timeCreated": "2025-01-01T00:00:00.000Z",
}
raw_response = MagicMock(spec=httpx.Response)
raw_response.json.return_value = gcs_json
result = config.transform_retrieve_file_response(
raw_response=raw_response,
logging_obj=MagicMock(),
litellm_params={},
)
assert result.purpose == "batch"
class TestTransformFileContent:
def test_should_build_gcs_media_download_url(self, config):
file_id = "gs://my-bucket/path/to/file.jsonl"
url, params = config.transform_file_content_request(
file_content_request={"file_id": file_id},
optional_params={},
litellm_params={},
)
encoded = urllib.parse.quote("path/to/file.jsonl", safe="")
assert url == f"https://storage.googleapis.com/storage/v1/b/my-bucket/o/{encoded}?alt=media"
assert params == {}
def test_should_return_binary_response_content(self, config):
raw_response = httpx.Response(
status_code=200,
content=b'{"line": 1}\n{"line": 2}\n',
headers={"content-type": "application/octet-stream"},
request=httpx.Request("GET", "https://example.com"),
)
result = config.transform_file_content_response(
raw_response=raw_response,
logging_obj=MagicMock(),
litellm_params={},
)
assert isinstance(result, HttpxBinaryResponseContent)
assert result.response.content == b'{"line": 1}\n{"line": 2}\n'
class TestTransformDeleteFile:
def test_should_build_correct_gcs_delete_url(self, config):
file_id = "gs://my-bucket/path/to/file.jsonl"
url, params = config.transform_delete_file_request(
file_id=file_id, optional_params={}, litellm_params={}
)
encoded = urllib.parse.quote("path/to/file.jsonl", safe="")
assert url == f"https://storage.googleapis.com/storage/v1/b/my-bucket/o/{encoded}"
assert params == {}
def test_should_return_file_deleted_with_reconstructed_id(self, config):
raw_response = MagicMock(spec=httpx.Response)
mock_request = MagicMock()
encoded_name = urllib.parse.quote(
"litellm-vertex-files/publishers/google/models/gemini-2.0-flash-001/abc", safe=""
)
mock_request.url = (
f"https://storage.googleapis.com/storage/v1/b/my-bucket/o/{encoded_name}"
)
raw_response.request = mock_request
result = config.transform_delete_file_response(
raw_response=raw_response,
logging_obj=MagicMock(),
litellm_params={},
)
assert isinstance(result, FileDeleted)
assert result.deleted is True
assert result.object == "file"
assert "litellm-vertex-files/publishers/google/models/gemini-2.0-flash-001/abc" in result.id
def test_should_fallback_to_deleted_id_when_no_request(self, config):
raw_response = MagicMock(spec=httpx.Response)
raw_response.request = None
result = config.transform_delete_file_response(
raw_response=raw_response,
logging_obj=MagicMock(),
litellm_params={},
)
assert isinstance(result, FileDeleted)
assert result.id == "deleted"
assert result.deleted is True

17
tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py
View file

@ -2130,7 +2130,7 @@ def test_reasoning_effort_dict_format_gemini_3():
assert result["thinkingConfig"]["thinkingLevel"] == "high"
assert result["thinkingConfig"]["includeThoughts"] is True
# Test dict format without effort key - should fall back to Gemini 3 default (low)
# Test dict format without effort key - no thinkingConfig should be set
optional_params = {}
non_default_params = {"reasoning_effort": {"summary": "auto"}}
result = v.map_openai_params(
@ -2139,8 +2139,8 @@ def test_reasoning_effort_dict_format_gemini_3():
model=model,
drop_params=False,
)
# Gemini 3 defaults to thinkingLevel="low" when no explicit effort is set
assert result["thinkingConfig"]["thinkingLevel"] == "low"
# No effort key in dict → no thinkingConfig set
assert "thinkingConfig" not in result
def test_temperature_default_for_gemini_3():
@ -2453,8 +2453,8 @@ def test_gemini_3_image_models_no_thinking_config():
def test_gemini_3_text_models_get_thinking_config():
"""
Test that Gemini 3 text models DO receive automatic thinkingConfig.
This ensures we didn't break the existing behavior for non-image models.
Test that Gemini 3 text models do NOT receive automatic thinkingConfig
when no reasoning_effort or thinking param is provided.
"""
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
@ -2462,7 +2462,7 @@ def test_gemini_3_text_models_get_thinking_config():
v = VertexGeminiConfig()
# Test gemini-3-pro-preview (text model, should get thinking)
# Test gemini-3-pro-preview (text model, no explicit thinking params)
model = "gemini-3-pro-preview"
optional_params = {}
non_default_params = {}
@ -2474,9 +2474,8 @@ def test_gemini_3_text_models_get_thinking_config():
drop_params=False,
)
# Should have thinkingConfig automatically added
assert "thinkingConfig" in result
assert result["thinkingConfig"]["thinkingLevel"] == "low"
# Should NOT have thinkingConfig automatically added when user provides no reasoning_effort
assert "thinkingConfig" not in result
assert result["temperature"] == 1.0

2
tests/test_litellm/llms/vertex_ai/test_vertex.py
View file

@ -393,6 +393,7 @@ def test_multiple_function_call():
],
},
{
"role": "user",
"parts": [
{
"function_response": {
@ -498,6 +499,7 @@ def test_multiple_function_call_changed_text_pos():
],
},
{
"role": "user",
"parts": [
{
"function_response": {

86
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py
View file

@ -2307,5 +2307,91 @@ class TestMCPServerManager:
assert resolved_server.server_name == "test_server" # server_name matches
class TestMCPServerTimestamps:
"""Regression tests: created_at/updated_at must be preserved, not overwritten with datetime.now()."""
@pytest.mark.asyncio
async def test_build_mcp_server_from_table_preserves_timestamps(self):
"""build_mcp_server_from_table must carry created_at and updated_at into MCPServer."""
manager = MCPServerManager()
created = datetime(2024, 1, 15, 10, 0, 0)
updated = datetime(2024, 6, 20, 12, 30, 0)
table_record = LiteLLM_MCPServerTable(
server_id="ts-server-1",
server_name="ts_server",
url="https://example.com/mcp",
transport=MCPTransport.http,
created_at=created,
updated_at=updated,
)
mcp_server = await manager.build_mcp_server_from_table(table_record)
assert mcp_server.created_at == created
assert mcp_server.updated_at == updated
def test_build_mcp_server_table_preserves_timestamps(self):
"""_build_mcp_server_table must use the MCPServer's stored timestamps, not datetime.now()."""
manager = MCPServerManager()
created = datetime(2024, 1, 15, 10, 0, 0)
updated = datetime(2024, 6, 20, 12, 30, 0)
server = MCPServer(
server_id="ts-server-2",
name="ts_server",
url="https://example.com/mcp",
transport=MCPTransport.http,
created_at=created,
updated_at=updated,
)
table = manager._build_mcp_server_table(server)
assert table.created_at == created
assert table.updated_at == updated
def test_build_mcp_server_table_none_timestamps_when_not_set(self):
"""_build_mcp_server_table must return None timestamps when not set on MCPServer."""
manager = MCPServerManager()
server = MCPServer(
server_id="ts-server-3",
name="ts_server",
url="https://example.com/mcp",
transport=MCPTransport.http,
)
table = manager._build_mcp_server_table(server)
assert table.created_at is None
assert table.updated_at is None
@pytest.mark.asyncio
async def test_round_trip_timestamps_preserved(self):
"""Timestamps survive the full round-trip: LiteLLM_MCPServerTable -> MCPServer -> LiteLLM_MCPServerTable."""
manager = MCPServerManager()
created = datetime(2023, 3, 10, 8, 0, 0)
updated = datetime(2023, 9, 5, 16, 45, 0)
table_record = LiteLLM_MCPServerTable(
server_id="ts-server-4",
server_name="ts_server_rt",
url="https://example.com/mcp",
transport=MCPTransport.http,
created_at=created,
updated_at=updated,
)
mcp_server = await manager.build_mcp_server_from_table(table_record)
rebuilt_table = manager._build_mcp_server_table(mcp_server)
assert rebuilt_table.created_at == created
assert rebuilt_table.updated_at == updated
if __name__ == "__main__":
pytest.main([__file__])

52
tests/test_litellm/proxy/auth/test_auth_checks.py
View file

@ -1519,3 +1519,55 @@ async def test_get_fuzzy_user_object_case_insensitive_email():
assert call_args.kwargs["where"]["user_email"]["equals"] == "test@example.com"
assert call_args.kwargs["where"]["user_email"]["mode"] == "insensitive"
assert call_args.kwargs["include"] == {"organization_memberships": True}
@pytest.mark.asyncio
async def test_common_checks_skip_route_check_for_custom_auth():
"""
Test that custom routes (e.g. /ldap/ngs/ready) pass common_checks when
skip_route_check=True, which is the case for custom auth flows.
Regression test for: custom user-added routes being rejected as admin-only
after _run_post_custom_auth_checks was introduced.
"""
from fastapi import Request
from litellm.proxy.auth.auth_checks import common_checks
mock_request = MagicMock(spec=Request)
valid_token = UserAPIKeyAuth(token="test-token")
# Without skip_route_check, a custom route with unknown user should fail
with pytest.raises(Exception):
await common_checks(
request_body={},
team_object=None,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/ldap/ngs/ready",
llm_router=None,
proxy_logging_obj=MagicMock(),
valid_token=valid_token,
request=mock_request,
skip_route_check=False,
)
# With skip_route_check=True (custom auth path), the same route should pass
result = await common_checks(
request_body={},
team_object=None,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/ldap/ngs/ready",
llm_router=None,
proxy_logging_obj=MagicMock(),
valid_token=valid_token,
request=mock_request,
skip_route_check=True,
)
assert result is True

29
tests/test_litellm/proxy/auth/test_handle_jwt.py
View file

@ -1514,7 +1514,7 @@ async def test_resolve_jwks_url_resolves_oidc_discovery_document():
A .well-known/openid-configuration URL should be fetched and its
jwks_uri returned.
"""
from unittest.mock import AsyncMock, MagicMock, patch
from unittest.mock import AsyncMock, MagicMock
from litellm.caching.dual_cache import DualCache
@ -1533,8 +1533,10 @@ async def test_resolve_jwks_url_resolves_oidc_discovery_document():
mock_response.status_code = 200
mock_response.json.return_value = {"jwks_uri": jwks_url, "issuer": "https://..."}
with patch.object(handler.http_handler, "get", new_callable=AsyncMock, return_value=mock_response) as mock_get:
result = await handler._resolve_jwks_url(discovery_url)
mock_get = AsyncMock(return_value=mock_response)
handler.http_handler.get = mock_get
result = await handler._resolve_jwks_url(discovery_url)
assert result == jwks_url
mock_get.assert_called_once_with(discovery_url)
@ -1543,7 +1545,7 @@ async def test_resolve_jwks_url_resolves_oidc_discovery_document():
@pytest.mark.asyncio
async def test_resolve_jwks_url_caches_resolved_jwks_uri():
"""Resolved jwks_uri is cached — second call does not hit the network."""
from unittest.mock import AsyncMock, MagicMock, patch
from unittest.mock import AsyncMock, MagicMock
from litellm.caching.dual_cache import DualCache
@ -1559,11 +1561,14 @@ async def test_resolve_jwks_url_caches_resolved_jwks_uri():
jwks_url = "https://login.microsoftonline.com/tenant/discovery/keys"
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"jwks_uri": jwks_url}
with patch.object(handler.http_handler, "get", new_callable=AsyncMock, return_value=mock_response) as mock_get:
first = await handler._resolve_jwks_url(discovery_url)
second = await handler._resolve_jwks_url(discovery_url)
mock_get = AsyncMock(return_value=mock_response)
handler.http_handler.get = mock_get
first = await handler._resolve_jwks_url(discovery_url)
second = await handler._resolve_jwks_url(discovery_url)
assert first == jwks_url
assert second == jwks_url
@ -1574,7 +1579,7 @@ async def test_resolve_jwks_url_caches_resolved_jwks_uri():
@pytest.mark.asyncio
async def test_resolve_jwks_url_raises_if_no_jwks_uri_in_discovery_doc():
"""Raise a helpful error if the discovery document has no jwks_uri."""
from unittest.mock import AsyncMock, MagicMock, patch
from unittest.mock import AsyncMock, MagicMock
from litellm.caching.dual_cache import DualCache
@ -1587,11 +1592,13 @@ async def test_resolve_jwks_url_raises_if_no_jwks_uri_in_discovery_doc():
discovery_url = "https://example.com/.well-known/openid-configuration"
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"issuer": "https://example.com"} # no jwks_uri
with patch.object(handler.http_handler, "get", new_callable=AsyncMock, return_value=mock_response):
with pytest.raises(Exception, match="jwks_uri"):
await handler._resolve_jwks_url(discovery_url)
handler.http_handler.get = AsyncMock(return_value=mock_response)
with pytest.raises(Exception, match="jwks_uri"):
await handler._resolve_jwks_url(discovery_url)
# ---------------------------------------------------------------------------

74
tests/test_litellm/proxy/auth/test_route_checks.py
View file

@ -1116,3 +1116,77 @@ def test_route_in_additional_public_routes_exact_match():
assert route_in_additonal_public_routes("/status") is True
# Non-matching routes should fail
assert route_in_additonal_public_routes("/other") is False
def test_internal_user_can_access_key_reset_spend_route():
"""
Regression test: team admins (role=internal_user) should pass the route-level
check for /key/{hash}/reset_spend. The endpoint itself enforces team admin status.
"""
user_obj = LiteLLM_UserTable(
user_id="team-admin-user",
user_email="teamadmin@example.com",
user_role=LitellmUserRoles.INTERNAL_USER.value,
)
valid_token = UserAPIKeyAuth(
user_id="team-admin-user",
user_role=LitellmUserRoles.INTERNAL_USER.value,
)
request = MagicMock(spec=Request)
request.query_params = {}
key_hash = "baec26d2901589fe9fec76610e6e2be4895cdd8e19b3ada9a4fa2eb85e1901ae"
route = f"/key/{key_hash}/reset_spend"
# Should not raise — the route-level check must pass for team admins
RouteChecks.non_proxy_admin_allowed_routes_check(
user_obj=user_obj,
_user_role=LitellmUserRoles.INTERNAL_USER.value,
route=route,
request=request,
valid_token=valid_token,
request_data={},
)
def test_non_admin_non_team_admin_cannot_access_config_update_but_can_attempt_reset_spend():
"""
An internal_user passes the route check for /key/{hash}/reset_spend
(authorization is deferred to the endpoint), but is still blocked from
admin-only routes like /config/update.
"""
user_obj = LiteLLM_UserTable(
user_id="regular-user",
user_email="user@example.com",
user_role=LitellmUserRoles.INTERNAL_USER.value,
)
valid_token = UserAPIKeyAuth(
user_id="regular-user",
user_role=LitellmUserRoles.INTERNAL_USER.value,
)
request = MagicMock(spec=Request)
request.query_params = {}
key_hash = "baec26d2901589fe9fec76610e6e2be4895cdd8e19b3ada9a4fa2eb85e1901ae"
# /key/{hash}/reset_spend passes the route check for internal_user
RouteChecks.non_proxy_admin_allowed_routes_check(
user_obj=user_obj,
_user_role=LitellmUserRoles.INTERNAL_USER.value,
route=f"/key/{key_hash}/reset_spend",
request=request,
valid_token=valid_token,
request_data={},
)
# /config/update is still blocked
with pytest.raises(Exception) as exc_info:
RouteChecks.non_proxy_admin_allowed_routes_check(
user_obj=user_obj,
_user_role=LitellmUserRoles.INTERNAL_USER.value,
route="/config/update",
request=request,
valid_token=valid_token,
request_data={},
)
assert "Only proxy admin can be used to generate" in str(exc_info.value)

93
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_generic_guardrail_api.py
View file

@ -13,8 +13,8 @@ import pytest
import litellm
from litellm import ModelResponse
from litellm.exceptions import GuardrailRaisedException, Timeout
from litellm._version import version as litellm_version
from litellm.exceptions import GuardrailRaisedException, Timeout
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.guardrails.guardrail_hooks.generic_guardrail_api import (
GenericGuardrailAPI,
@ -188,6 +188,97 @@ class TestGenericGuardrailAPIConfiguration:
)
assert "x-api-key" not in guardrail.headers
def test_init_with_extra_headers(self):
"""Test that extra_headers is stored for forwarding client headers to the guardrail"""
guardrail = GenericGuardrailAPI(
api_base="https://api.test.guardrail.com",
extra_headers=["x-request-id", "x-custom-auth"],
)
assert guardrail.extra_headers == ["x-request-id", "x-custom-auth"]
class TestExtraHeadersForwarding:
"""Test extra_headers: client headers allowed to be forwarded to the guardrail"""
@pytest.mark.asyncio
async def test_extra_headers_values_forwarded_to_guardrail(self):
"""When extra_headers is set, those client header values are sent to the guardrail."""
guardrail = GenericGuardrailAPI(
api_base="https://api.test.guardrail.com",
extra_headers=["x-my-header", "x-request-id"],
)
request_data = {
"proxy_server_request": {
"headers": {
"x-my-header": "my-value",
"x-request-id": "req-123",
"x-private": "secret",
},
},
}
mock_response = MagicMock()
mock_response.json.return_value = {
"action": "NONE",
"texts": ["test"],
}
mock_response.raise_for_status = MagicMock()
with patch.object(
guardrail.async_handler, "post", return_value=mock_response
) as mock_post:
await guardrail.apply_guardrail(
inputs={"texts": ["test"]},
request_data=request_data,
input_type="request",
)
call_args = mock_post.call_args
json_payload = call_args.kwargs["json"]
request_headers = json_payload.get("request_headers") or {}
# Headers in extra_headers have their values forwarded
assert request_headers.get("x-my-header") == "my-value"
assert request_headers.get("x-request-id") == "req-123"
# Headers not in allowlist are sent as placeholder
assert request_headers.get("x-private") == _HEADER_PRESENT_PLACEHOLDER
@pytest.mark.asyncio
async def test_without_extra_headers_custom_header_value_not_forwarded(self):
"""Without extra_headers, a custom client header is sent as [present] only."""
guardrail = GenericGuardrailAPI(
api_base="https://api.test.guardrail.com",
# no extra_headers
)
request_data = {
"proxy_server_request": {
"headers": {
"x-custom-auth": "bearer secret-token",
},
},
}
mock_response = MagicMock()
mock_response.json.return_value = {
"action": "NONE",
"texts": ["test"],
}
mock_response.raise_for_status = MagicMock()
with patch.object(
guardrail.async_handler, "post", return_value=mock_response
) as mock_post:
await guardrail.apply_guardrail(
inputs={"texts": ["test"]},
request_data=request_data,
input_type="request",
)
call_args = mock_post.call_args
json_payload = call_args.kwargs["json"]
request_headers = json_payload.get("request_headers") or {}
# x-custom-auth is not in default allowlist nor extra_headers, so value is not forwarded
assert request_headers.get("x-custom-auth") == _HEADER_PRESENT_PLACEHOLDER
class TestMetadataExtraction:
"""Test metadata extraction from request data"""

468
tests/test_litellm/proxy/guardrails/test_guardrail_endpoints.py
View file

@ -17,13 +17,19 @@ from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
from litellm.proxy.guardrails.guardrail_endpoints import (
CreateGuardrailRequest,
PatchGuardrailRequest,
RegisterGuardrailRequest,
UpdateGuardrailRequest,
apply_guardrail,
approve_guardrail_submission,
create_guardrail,
delete_guardrail,
get_guardrail_info,
get_guardrail_submission,
list_guardrail_submissions,
list_guardrails_v2,
patch_guardrail,
register_guardrail,
reject_guardrail_submission,
update_guardrail,
)
@ -1146,4 +1152,464 @@ class TestBuildFieldDict:
)
assert result["type"] == "bool"
assert result["required"] is True
assert result["required"] is True
# --- Team guardrail registration (register / submissions) ---
MOCK_REGISTER_REQUEST = RegisterGuardrailRequest(
guardrail_name="team-prompt-guard",
litellm_params={
"guardrail": "generic_guardrail_api",
"mode": "pre_call",
"api_base": "https://guardrails.example.com/validate",
},
guardrail_info={"description": "Team prompt injection detector"},
)
@pytest.mark.asyncio
async def test_register_guardrail_success(mocker):
"""Register creates a row with status pending_review and returns guardrail_id."""
mock_prisma = mocker.Mock()
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=None)
created_row = mocker.Mock(
guardrail_id="reg-123",
guardrail_name=MOCK_REGISTER_REQUEST.guardrail_name,
status="pending_review",
submitted_at=datetime.now(),
)
mock_prisma.db.litellm_guardrailstable.create = AsyncMock(return_value=created_row)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_id="u1", user_email="alice@co.com", team_id="team-1")
result = await register_guardrail(MOCK_REGISTER_REQUEST, user)
assert result.guardrail_id == "reg-123"
assert result.guardrail_name == MOCK_REGISTER_REQUEST.guardrail_name
assert result.status == "pending_review"
mock_prisma.db.litellm_guardrailstable.create.assert_called_once()
call_data = mock_prisma.db.litellm_guardrailstable.create.call_args[1]["data"]
assert call_data["status"] == "pending_review"
assert call_data["guardrail_name"] == MOCK_REGISTER_REQUEST.guardrail_name
@pytest.mark.asyncio
async def test_register_guardrail_rejects_non_generic_api(mocker):
"""Register returns 400 when litellm_params.guardrail is not generic_guardrail_api."""
mocker.patch("litellm.proxy.proxy_server.prisma_client", mocker.Mock())
req = RegisterGuardrailRequest(
guardrail_name="other-guard",
litellm_params={"guardrail": "bedrock", "mode": "pre_call", "api_base": "https://x.com"},
)
user = UserAPIKeyAuth(user_id="u1", user_email="a@b.com", team_id="team-1")
with pytest.raises(HTTPException) as exc_info:
await register_guardrail(req, user)
assert exc_info.value.status_code == 400
assert "generic_guardrail_api" in exc_info.value.detail
@pytest.mark.asyncio
async def test_register_guardrail_requires_team_id(mocker):
"""Register returns 400 when API key has no associated team_id."""
mocker.patch("litellm.proxy.proxy_server.prisma_client", mocker.Mock())
user = UserAPIKeyAuth(user_id="u1", user_email="a@b.com", team_id=None)
with pytest.raises(HTTPException) as exc_info:
await register_guardrail(MOCK_REGISTER_REQUEST, user)
assert exc_info.value.status_code == 400
assert "team" in exc_info.value.detail.lower()
@pytest.mark.asyncio
async def test_register_guardrail_duplicate_name(mocker):
"""Register returns 400 when guardrail_name already exists."""
mock_prisma = mocker.Mock()
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(
return_value={"guardrail_name": MOCK_REGISTER_REQUEST.guardrail_name}
)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_id="u1", user_email="a@b.com", team_id="team-1")
with pytest.raises(HTTPException) as exc_info:
await register_guardrail(MOCK_REGISTER_REQUEST, user)
assert exc_info.value.status_code == 400
assert "already exists" in exc_info.value.detail
@pytest.mark.asyncio
async def test_list_guardrail_submissions_requires_admin(mocker):
"""List submissions returns 403 when user is not admin."""
mocker.patch("litellm.proxy.proxy_server.prisma_client", mocker.Mock())
user = UserAPIKeyAuth(user_role=LitellmUserRoles.INTERNAL_USER)
with pytest.raises(HTTPException) as exc_info:
await list_guardrail_submissions(user_api_key_dict=user)
assert exc_info.value.status_code == 403
@pytest.mark.asyncio
async def test_list_guardrail_submissions_success(mocker):
"""List submissions returns list and summary for admin."""
mock_prisma = mocker.Mock()
row = mocker.Mock(
guardrail_id="sub-1",
guardrail_name="pending-guard",
status="pending_review",
team_id="t1",
litellm_params={"guardrail": "generic_guardrail_api", "api_base": "https://x.com"},
guardrail_info={
"description": "A guard",
"submitted_by_user_id": "u1",
"submitted_by_email": "alice@co.com",
},
submitted_at=datetime.now(),
reviewed_at=None,
created_at=datetime.now(),
updated_at=datetime.now(),
)
mock_prisma.db.litellm_guardrailstable.find_many = AsyncMock(return_value=[row])
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
result = await list_guardrail_submissions(user_api_key_dict=user)
assert len(result.submissions) == 1
assert result.submissions[0].guardrail_id == "sub-1"
assert result.submissions[0].status == "pending_review"
assert result.submissions[0].team_guardrail is True # team_id is set
assert result.summary.total >= 1
assert result.summary.pending_review >= 1
@pytest.mark.asyncio
async def test_list_guardrail_submissions_returns_only_team_guardrails(mocker):
"""List submissions only returns team guardrails (team_id not null)."""
mock_prisma = mocker.Mock()
find_many = AsyncMock(return_value=[])
mock_prisma.db.litellm_guardrailstable.find_many = find_many
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
await list_guardrail_submissions(user_api_key_dict=user)
calls = find_many.call_args_list
assert len(calls) >= 1
first_where = calls[0].kwargs.get("where", {})
assert first_where.get("team_id") == {"not": None}
@pytest.mark.asyncio
async def test_list_guardrail_submissions_team_id_filter(mocker):
"""List submissions with team_id filter returns only that team's guardrails."""
mock_prisma = mocker.Mock()
row_abc = mocker.Mock(
guardrail_id="team-1",
guardrail_name="team-guard",
status="active",
team_id="team-abc",
litellm_params={},
guardrail_info={},
submitted_at=None,
reviewed_at=None,
created_at=datetime.now(),
updated_at=datetime.now(),
)
row_other = mocker.Mock(
guardrail_id="team-2",
guardrail_name="other-guard",
status="active",
team_id="team-xyz",
litellm_params={},
guardrail_info={},
submitted_at=None,
reviewed_at=None,
created_at=datetime.now(),
updated_at=datetime.now(),
)
find_many = AsyncMock(return_value=[row_abc, row_other])
mock_prisma.db.litellm_guardrailstable.find_many = find_many
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
result = await list_guardrail_submissions(
user_api_key_dict=user, team_id="team-abc"
)
assert len(result.submissions) == 1
assert result.submissions[0].guardrail_id == "team-1"
assert result.submissions[0].team_guardrail is True
assert result.summary.total == 2 # summary counts all team guardrails
@pytest.mark.asyncio
async def test_get_guardrail_submission_not_found(mocker):
"""Get submission returns 404 when guardrail_id does not exist."""
mock_prisma = mocker.Mock()
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=None)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
with pytest.raises(HTTPException) as exc_info:
await get_guardrail_submission("nonexistent-id", user)
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_approve_guardrail_submission_success(mocker):
"""Approve sets status to active and initializes guardrail in memory."""
mock_prisma = mocker.Mock()
row = mocker.Mock(
guardrail_id="approve-me",
guardrail_name="my-guard",
status="pending_review",
litellm_params={"guardrail": "generic_guardrail_api", "mode": "pre_call", "api_base": "https://g.com"},
guardrail_info={},
)
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=row)
mock_prisma.db.litellm_guardrailstable.update = AsyncMock()
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
mock_handler = mocker.Mock()
mock_handler.initialize_guardrail = mocker.Mock()
mocker.patch(
"litellm.proxy.guardrails.guardrail_registry.IN_MEMORY_GUARDRAIL_HANDLER",
mock_handler,
)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
result = await approve_guardrail_submission("approve-me", user)
assert result["status"] == "active"
assert result["guardrail_id"] == "approve-me"
mock_prisma.db.litellm_guardrailstable.update.assert_called_once()
call_data = mock_prisma.db.litellm_guardrailstable.update.call_args[1]["data"]
assert call_data["status"] == "active"
@pytest.mark.asyncio
async def test_approve_guardrail_submission_not_pending(mocker):
"""Approve returns 400 when status is not pending_review."""
mock_prisma = mocker.Mock()
row = mocker.Mock(guardrail_id="x", guardrail_name="y", status="active")
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=row)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
with pytest.raises(HTTPException) as exc_info:
await approve_guardrail_submission("x", user)
assert exc_info.value.status_code == 400
@pytest.mark.asyncio
async def test_reject_guardrail_submission_success(mocker):
"""Reject sets status to rejected."""
mock_prisma = mocker.Mock()
row = mocker.Mock(guardrail_id="rej-1", guardrail_name="r", status="pending_review")
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=row)
mock_prisma.db.litellm_guardrailstable.update = AsyncMock()
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
result = await reject_guardrail_submission("rej-1", user)
assert result["status"] == "rejected"
mock_prisma.db.litellm_guardrailstable.update.assert_called_once()
call_data = mock_prisma.db.litellm_guardrailstable.update.call_args[1]["data"]
assert call_data["status"] == "rejected"
@pytest.mark.asyncio
async def test_reject_guardrail_submission_not_pending(mocker):
"""Reject returns 400 when status is not pending_review (e.g. already active)."""
mock_prisma = mocker.Mock()
row = mocker.Mock(guardrail_id="already-active", guardrail_name="g", status="active")
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=row)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
with pytest.raises(HTTPException) as exc_info:
await reject_guardrail_submission("already-active", user)
assert exc_info.value.status_code == 400
assert "not pending review" in exc_info.value.detail.lower()
# --- Tests for review fixes ---
@pytest.mark.asyncio
@pytest.mark.parametrize(
"api_base,expected_detail",
[
("file:///etc/passwd", "http or https scheme"),
("ftp://internal.host/data", "http or https scheme"),
("javascript:alert(1)", "http or https scheme"),
("://missing-scheme", "http or https scheme"),
("https://", "valid hostname"),
],
ids=[
"file_scheme",
"ftp_scheme",
"javascript_scheme",
"no_scheme",
"no_hostname",
],
)
async def test_register_guardrail_rejects_bad_api_base(mocker, api_base, expected_detail):
"""Register returns 400 when api_base has invalid scheme or missing hostname."""
mocker.patch("litellm.proxy.proxy_server.prisma_client", mocker.Mock())
req = RegisterGuardrailRequest(
guardrail_name="bad-url-guard",
litellm_params={
"guardrail": "generic_guardrail_api",
"mode": "pre_call",
"api_base": api_base,
},
)
user = UserAPIKeyAuth(user_id="u1", user_email="a@b.com", team_id="team-1")
with pytest.raises(HTTPException) as exc_info:
await register_guardrail(req, user)
assert exc_info.value.status_code == 400
assert expected_detail in exc_info.value.detail
@pytest.mark.asyncio
async def test_register_guardrail_accepts_valid_https_url(mocker):
"""Register accepts valid https api_base URLs."""
mock_prisma = mocker.Mock()
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=None)
created_row = mocker.Mock(
guardrail_id="valid-url-123",
guardrail_name="valid-guard",
status="pending_review",
submitted_at=datetime.now(),
)
mock_prisma.db.litellm_guardrailstable.create = AsyncMock(return_value=created_row)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
req = RegisterGuardrailRequest(
guardrail_name="valid-guard",
litellm_params={
"guardrail": "generic_guardrail_api",
"mode": "pre_call",
"api_base": "https://guardrails.example.com/v1/check",
},
)
user = UserAPIKeyAuth(user_id="u1", user_email="a@b.com", team_id="team-1")
result = await register_guardrail(req, user)
assert result.guardrail_id == "valid-url-123"
assert result.status == "pending_review"
@pytest.mark.asyncio
async def test_approve_guardrail_init_failure_returns_warning(mocker):
"""Approve returns a warning field when in-memory initialization fails."""
mock_prisma = mocker.Mock()
row = mocker.Mock(
guardrail_id="warn-me",
guardrail_name="fragile-guard",
status="pending_review",
litellm_params={
"guardrail": "generic_guardrail_api",
"mode": "pre_call",
"api_base": "https://g.com",
},
guardrail_info={},
)
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=row)
mock_prisma.db.litellm_guardrailstable.update = AsyncMock()
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
mock_handler = mocker.Mock()
mock_handler.initialize_guardrail = mocker.Mock(
side_effect=Exception("missing dependency")
)
mocker.patch(
"litellm.proxy.guardrails.guardrail_registry.IN_MEMORY_GUARDRAIL_HANDLER",
mock_handler,
)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
result = await approve_guardrail_submission("warn-me", user)
assert result["status"] == "active"
assert "warning" in result
assert "failed to initialize" in result["warning"].lower()
assert "missing dependency" in result["warning"]
@pytest.mark.asyncio
async def test_approve_guardrail_no_warning_on_success(mocker):
"""Approve does NOT include a warning field when init succeeds."""
mock_prisma = mocker.Mock()
row = mocker.Mock(
guardrail_id="ok-guard",
guardrail_name="good-guard",
status="pending_review",
litellm_params={
"guardrail": "generic_guardrail_api",
"mode": "pre_call",
"api_base": "https://g.com",
},
guardrail_info={},
)
mock_prisma.db.litellm_guardrailstable.find_unique = AsyncMock(return_value=row)
mock_prisma.db.litellm_guardrailstable.update = AsyncMock()
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
mock_handler = mocker.Mock()
mock_handler.initialize_guardrail = mocker.Mock() # no exception
mocker.patch(
"litellm.proxy.guardrails.guardrail_registry.IN_MEMORY_GUARDRAIL_HANDLER",
mock_handler,
)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
result = await approve_guardrail_submission("ok-guard", user)
assert result["status"] == "active"
assert "warning" not in result
@pytest.mark.asyncio
async def test_list_submissions_single_db_query(mocker):
"""List submissions makes exactly one find_many call (no redundant query)."""
mock_prisma = mocker.Mock()
find_many = AsyncMock(return_value=[])
mock_prisma.db.litellm_guardrailstable.find_many = find_many
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
await list_guardrail_submissions(user_api_key_dict=user)
assert find_many.call_count == 1
@pytest.mark.asyncio
async def test_list_submissions_summary_counts_unaffected_by_filters(mocker):
"""Summary counts reflect all team guardrails regardless of status filter."""
mock_prisma = mocker.Mock()
pending_row = mocker.Mock(
guardrail_id="p1", guardrail_name="p", status="pending_review",
team_id="t1", litellm_params={}, guardrail_info={},
submitted_at=None, reviewed_at=None,
created_at=datetime.now(), updated_at=datetime.now(),
)
active_row = mocker.Mock(
guardrail_id="a1", guardrail_name="a", status="active",
team_id="t1", litellm_params={}, guardrail_info={},
submitted_at=None, reviewed_at=None,
created_at=datetime.now(), updated_at=datetime.now(),
)
all_rows = [pending_row, active_row]
mock_prisma.db.litellm_guardrailstable.find_many = AsyncMock(return_value=all_rows)
mocker.patch("litellm.proxy.proxy_server.prisma_client", mock_prisma)
user = UserAPIKeyAuth(user_role=LitellmUserRoles.PROXY_ADMIN)
# Filter to only pending, but summary should still show both
result = await list_guardrail_submissions(status="pending_review", user_api_key_dict=user)
assert len(result.submissions) == 1 # filtered
assert result.summary.total == 2 # unfiltered
assert result.summary.pending_review == 1
assert result.summary.active == 1

120
tests/test_litellm/proxy/management_endpoints/test_cost_tracking_settings.py
View file

@ -270,3 +270,123 @@ class TestCostTrackingSettings:
assert "error" in response_data["detail"]
assert "STORE_MODEL_IN_DB" in response_data["detail"]["error"]
class TestResolveModelForCostLookup:
"""Tests for _resolve_model_for_cost_lookup base_model resolution."""
def test_resolves_base_model_for_azure_deployment(self):
"""
When a model group has base_model set in model_info,
_resolve_model_for_cost_lookup should return the base_model
instead of the raw litellm_params.model (Azure deployment name).
"""
from litellm.proxy.management_endpoints.cost_tracking_settings import (
_resolve_model_for_cost_lookup,
)
mock_router = MagicMock()
mock_router.get_model_list.return_value = [
{
"model_name": "gpt-5.3-codex",
"litellm_params": {
"model": "azure/openai/gpt-5.3-codex",
"api_base": "https://fake.openai.azure.com/",
"api_key": "fake-key",
},
"model_info": {
"id": "test-id",
"base_model": "azure/gpt-4o",
},
}
]
with patch(
"litellm.proxy.proxy_server.llm_router",
mock_router,
):
resolved_model, provider = _resolve_model_for_cost_lookup("gpt-5.3-codex")
assert resolved_model == "azure/gpt-4o"
mock_router.get_model_list.assert_called_once_with(model_name="gpt-5.3-codex")
def test_falls_back_to_litellm_params_model_when_no_base_model(self):
"""
When no base_model is set, should fall back to litellm_params.model.
"""
from litellm.proxy.management_endpoints.cost_tracking_settings import (
_resolve_model_for_cost_lookup,
)
mock_router = MagicMock()
mock_router.get_model_list.return_value = [
{
"model_name": "gpt-4",
"litellm_params": {
"model": "openai/gpt-4",
},
"model_info": {
"id": "test-id",
},
}
]
with patch(
"litellm.proxy.proxy_server.llm_router",
mock_router,
):
resolved_model, provider = _resolve_model_for_cost_lookup("gpt-4")
assert resolved_model == "openai/gpt-4"
def test_resolves_base_model_from_litellm_params(self):
"""
When base_model is in litellm_params (not model_info),
it should still be resolved.
"""
from litellm.proxy.management_endpoints.cost_tracking_settings import (
_resolve_model_for_cost_lookup,
)
mock_router = MagicMock()
mock_router.get_model_list.return_value = [
{
"model_name": "my-azure-model",
"litellm_params": {
"model": "azure/my-custom-deployment",
"base_model": "azure/gpt-4o-mini",
},
"model_info": {
"id": "test-id",
},
}
]
with patch(
"litellm.proxy.proxy_server.llm_router",
mock_router,
):
resolved_model, provider = _resolve_model_for_cost_lookup(
"my-azure-model"
)
assert resolved_model == "azure/gpt-4o-mini"
def test_returns_original_model_when_no_router(self):
"""
When no router is available, should return the original model name.
"""
from litellm.proxy.management_endpoints.cost_tracking_settings import (
_resolve_model_for_cost_lookup,
)
with patch(
"litellm.proxy.proxy_server.llm_router",
None,
):
resolved_model, provider = _resolve_model_for_cost_lookup(
"azure/openai/gpt-5.3-codex"
)
assert resolved_model == "azure/openai/gpt-5.3-codex"
assert provider is None

135
tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py
View file

@ -1168,3 +1168,138 @@ def test_create_file_with_deep_nested_litellm_metadata(
assert captured_litellm_metadata["config"]["database"]["port"] == "5432"
assert "cache" in captured_litellm_metadata["config"]
assert captured_litellm_metadata["config"]["cache"]["enabled"] == "true"
# ---------------------------------------------------------------------------
# Team-level enforced_file_expires_after tests
# ---------------------------------------------------------------------------
def _make_capturing_managed_files():
"""Create a DummyManagedFiles that captures the expires_after from the request."""
from litellm.llms.base_llm.files.transformation import BaseFileEndpoints
captured = {}
class CapturingManagedFiles(BaseFileEndpoints):
async def acreate_file(
self,
llm_router,
create_file_request,
target_model_names_list,
litellm_parent_otel_span,
user_api_key_dict,
):
if isinstance(create_file_request, dict):
captured["expires_after"] = create_file_request.get("expires_after")
else:
captured["expires_after"] = getattr(
create_file_request, "expires_after", None
)
return OpenAIFileObject(
id="file-abc123",
object="file",
bytes=100,
created_at=1234567890,
filename="mydata.jsonl",
purpose="batch",
status="uploaded",
)
async def afile_retrieve(self, file_id, litellm_parent_otel_span, llm_router):
raise NotImplementedError
async def afile_list(self, purpose, litellm_parent_otel_span):
raise NotImplementedError
async def afile_delete(
self, file_id, litellm_parent_otel_span, llm_router, **data
):
raise NotImplementedError
async def afile_content(
self, file_id, litellm_parent_otel_span, llm_router, **data
):
raise NotImplementedError
return CapturingManagedFiles(), captured
def _post_file_with_team_metadata(
monkeypatch,
llm_router: Router,
team_metadata: dict,
form_data: dict,
):
"""POST /v1/files with given team_metadata, return captured expires_after."""
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
proxy_logging_obj = ProxyLogging(
user_api_key_cache=DualCache(default_in_memory_ttl=1)
)
dummy, captured = _make_capturing_managed_files()
proxy_logging_obj.proxy_hook_mapping["managed_files"] = dummy
monkeypatch.setattr("litellm.proxy.proxy_server.llm_router", llm_router)
monkeypatch.setattr(
"litellm.proxy.proxy_server.proxy_logging_obj", proxy_logging_obj
)
user_key = UserAPIKeyAuth(api_key="test-key", team_metadata=team_metadata)
app.dependency_overrides[user_api_key_auth] = lambda: user_key
test_file = ("mydata.jsonl", b'{"prompt": "Hello"}', "application/json")
try:
response = client.post(
"/v1/files",
files={"file": test_file},
data=form_data,
headers={"Authorization": "Bearer test-key"},
)
assert response.status_code == 200
finally:
app.dependency_overrides.clear()
return captured["expires_after"]
def test_file_team_override_overrides_caller(
mocker: MockerFixture, monkeypatch, llm_router: Router
):
"""Team enforced_file_expires_after wins over caller-provided value."""
expires_after = _post_file_with_team_metadata(
monkeypatch,
llm_router,
team_metadata={
"enforced_file_expires_after": {
"anchor": "created_at",
"seconds": 3600,
}
},
form_data={
"purpose": "batch",
"target_model_names": "gpt-3.5-turbo",
"expires_after[anchor]": "created_at",
"expires_after[seconds]": "86400",
},
)
assert expires_after["anchor"] == "created_at"
assert expires_after["seconds"] == 3600
def test_file_no_team_setting_preserves_caller(
mocker: MockerFixture, monkeypatch, llm_router: Router
):
"""No team setting = caller-provided expires_after passes through."""
expires_after = _post_file_with_team_metadata(
monkeypatch,
llm_router,
team_metadata={},
form_data={
"purpose": "batch",
"target_model_names": "gpt-3.5-turbo",
"expires_after[anchor]": "created_at",
"expires_after[seconds]": "86400",
},
)
assert expires_after["anchor"] == "created_at"
assert expires_after["seconds"] == 86400

219
tests/test_litellm/proxy/pass_through_endpoints/test_vertex_ai_batch_passthrough.py
View file

@ -227,52 +227,29 @@ class TestVertexAIBatchPassthroughHandler:
mock_managed_files_hook.store_unified_object_id.assert_called_once()
def test_batch_cost_calculation_integration(self):
"""Test integration with batch cost calculation"""
"""Single Vertex AI response → non-zero cost with correct token counts."""
from litellm.batches.batch_utils import calculate_vertex_ai_batch_cost_and_usage
# Mock Vertex AI batch responses
vertex_ai_batch_responses = [
{
"status": "JOB_STATE_SUCCEEDED",
"response": {
"candidates": [
{
"content": {
"parts": [
{"text": "Hello, world!"}
]
}
}
],
"usageMetadata": {
"promptTokenCount": 10,
"candidatesTokenCount": 5,
"totalTokenCount": 15
"totalTokenCount": 15,
}
}
}
]
with patch('litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini.VertexGeminiConfig') as mock_config:
with patch('litellm.completion_cost') as mock_completion_cost:
# Setup mocks
mock_config.return_value._transform_google_generate_content_to_openai_model_response.return_value = Mock(
usage=Mock(total_tokens=15, prompt_tokens=10, completion_tokens=5)
)
mock_completion_cost.return_value = 0.001
# Test the cost calculation
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
vertex_ai_batch_responses,
model_name="gemini-1.5-flash"
)
# Verify results
assert total_cost == 0.001
assert usage.total_tokens == 15
assert usage.prompt_tokens == 10
assert usage.completion_tokens == 5
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
vertex_ai_batch_responses, model_name="gemini-1.5-flash-001"
)
assert usage.total_tokens == 15
assert usage.prompt_tokens == 10
assert usage.completion_tokens == 5
assert total_cost > 0, "batch_cost_calculator should return a non-zero cost"
def test_batch_response_transformation(self):
"""Test transformation of Vertex AI batch responses to OpenAI format"""
@ -385,155 +362,107 @@ class TestVertexAIBatchPassthroughHandler:
class TestVertexAIBatchCostCalculation:
"""Test cases for Vertex AI batch cost calculation functionality"""
"""Test cases for Vertex AI batch cost calculation functionality.
def test_calculate_vertex_ai_batch_cost_and_usage_success(self):
"""Test successful batch cost and usage calculation"""
The function under test (calculate_vertex_ai_batch_cost_and_usage) extracts
usageMetadata directly from Vertex AI response dicts and calls
batch_cost_calculator — no VertexGeminiConfig transformation involved.
"""
def test_should_aggregate_cost_and_usage_across_responses(self):
"""Two successful responses → costs and token counts are summed."""
from litellm.batches.batch_utils import calculate_vertex_ai_batch_cost_and_usage
# Mock successful batch responses
vertex_ai_batch_responses = [
responses = [
{
"status": "JOB_STATE_SUCCEEDED",
"response": {
"candidates": [
{
"content": {
"parts": [
{"text": "Hello, world!"}
]
}
}
],
"usageMetadata": {
"promptTokenCount": 10,
"candidatesTokenCount": 5,
"totalTokenCount": 15
"totalTokenCount": 15,
}
}
},
{
"status": "JOB_STATE_SUCCEEDED",
"response": {
"candidates": [
{
"content": {
"parts": [
{"text": "How are you?"}
]
}
}
],
"usageMetadata": {
"promptTokenCount": 8,
"candidatesTokenCount": 3,
"totalTokenCount": 11
"totalTokenCount": 11,
}
}
}
},
]
with patch('litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini.VertexGeminiConfig') as mock_config:
with patch('litellm.completion_cost') as mock_completion_cost:
# Setup mocks
mock_model_response = Mock()
mock_model_response.usage = Mock(total_tokens=15, prompt_tokens=10, completion_tokens=5)
mock_config.return_value._transform_google_generate_content_to_openai_model_response.return_value = mock_model_response
mock_completion_cost.return_value = 0.001
# Test the calculation
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
vertex_ai_batch_responses,
model_name="gemini-1.5-flash"
)
# Verify results
assert total_cost == 0.002 # 2 responses * 0.001 each
assert usage.total_tokens == 30 # 15 + 15
assert usage.prompt_tokens == 20 # 10 + 10
assert usage.completion_tokens == 10 # 5 + 5
def test_calculate_vertex_ai_batch_cost_and_usage_with_failed_responses(self):
"""Test batch cost calculation with some failed responses"""
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
responses, model_name="gemini-1.5-flash-001"
)
assert usage.prompt_tokens == 18
assert usage.completion_tokens == 8
assert usage.total_tokens == 26
assert total_cost > 0, "batch_cost_calculator should return a non-zero cost"
def test_should_skip_responses_with_null_response_body(self):
"""Failed lines (response: None) are skipped without error."""
from litellm.batches.batch_utils import calculate_vertex_ai_batch_cost_and_usage
# Mock batch responses with some failures
vertex_ai_batch_responses = [
responses = [
{
"status": "JOB_STATE_SUCCEEDED",
"response": {
"candidates": [
{
"content": {
"parts": [
{"text": "Hello, world!"}
]
}
}
],
"usageMetadata": {
"promptTokenCount": 10,
"candidatesTokenCount": 5,
"totalTokenCount": 15
"totalTokenCount": 15,
}
}
},
{"status": "JOB_STATE_FAILED", "response": None},
{
"status": "JOB_STATE_FAILED", # Failed response
"response": None
},
{
"status": "JOB_STATE_SUCCEEDED",
"response": {
"candidates": [
{
"content": {
"parts": [
{"text": "How are you?"}
]
}
}
],
"usageMetadata": {
"promptTokenCount": 8,
"candidatesTokenCount": 3,
"totalTokenCount": 11
"totalTokenCount": 11,
}
}
}
},
]
with patch('litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini.VertexGeminiConfig') as mock_config:
with patch('litellm.completion_cost') as mock_completion_cost:
# Setup mocks
mock_model_response = Mock()
mock_model_response.usage = Mock(total_tokens=15, prompt_tokens=10, completion_tokens=5)
mock_config.return_value._transform_google_generate_content_to_openai_model_response.return_value = mock_model_response
mock_completion_cost.return_value = 0.001
# Test the calculation
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
vertex_ai_batch_responses,
model_name="gemini-1.5-flash"
)
# Verify results - should only process successful responses
assert total_cost == 0.002 # 2 successful responses * 0.001 each
assert usage.total_tokens == 30 # 15 + 15
assert usage.prompt_tokens == 20 # 10 + 10
assert usage.completion_tokens == 10 # 5 + 5
def test_calculate_vertex_ai_batch_cost_and_usage_empty_responses(self):
"""Test batch cost calculation with empty response list"""
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
responses, model_name="gemini-1.5-flash-001"
)
assert usage.prompt_tokens == 18
assert usage.completion_tokens == 8
assert usage.total_tokens == 26
assert total_cost > 0
def test_should_return_zeros_for_empty_response_list(self):
"""Empty input → zero cost and zero usage."""
from litellm.batches.batch_utils import calculate_vertex_ai_batch_cost_and_usage
# Test with empty list
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage([], model_name="gemini-1.5-flash")
# Verify results
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
[], model_name="gemini-1.5-flash-001"
)
assert total_cost == 0.0
assert usage.total_tokens == 0
assert usage.prompt_tokens == 0
assert usage.completion_tokens == 0
def test_should_handle_missing_usage_metadata_gracefully(self):
"""Response without usageMetadata → 0 tokens, 0 cost for that line."""
from litellm.batches.batch_utils import calculate_vertex_ai_batch_cost_and_usage
responses = [
{"response": {"candidates": [{"content": {"parts": [{"text": "hi"}]}}]}},
]
total_cost, usage = calculate_vertex_ai_batch_cost_and_usage(
responses, model_name="gemini-1.5-flash-001"
)
assert usage.prompt_tokens == 0
assert usage.completion_tokens == 0
assert usage.total_tokens == 0

162
tests/test_litellm/proxy/test_batch_expiry.py Normal file
View file

@ -0,0 +1,162 @@
"""
Tests for batch output_expires_after passthrough and team-level expiry enforcement.
"""
import os
import sys
from unittest.mock import MagicMock, patch
import pytest
sys.path.insert(
0, os.path.abspath("../../..")
) # Adds the parent directory to the system path
import litellm
from litellm.caching.caching import DualCache
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.proxy_server import app
from litellm.proxy.utils import ProxyLogging
from litellm.router import Router
from litellm.types.utils import LiteLLMBatch
from fastapi.testclient import TestClient
client = TestClient(app)
TEAM_EXPIRY = {"anchor": "created_at", "seconds": 3600}
CALLER_EXPIRY = {"anchor": "created_at", "seconds": 86400}
@pytest.fixture
def llm_router() -> Router:
return Router(
model_list=[
{
"model_name": "gpt-3.5-turbo",
"litellm_params": {
"model": "openai/gpt-3.5-turbo",
"api_key": "test-key",
},
"model_info": {"id": "gpt-3.5-turbo-id"},
},
]
)
def _setup_proxy(monkeypatch, llm_router: Router):
proxy_logging_obj = ProxyLogging(
user_api_key_cache=DualCache(default_in_memory_ttl=1)
)
monkeypatch.setattr("litellm.proxy.proxy_server.llm_router", llm_router)
monkeypatch.setattr(
"litellm.proxy.proxy_server.proxy_logging_obj", proxy_logging_obj
)
def _make_batch_response() -> LiteLLMBatch:
return LiteLLMBatch(
id="batch_abc123",
completion_window="24h",
created_at=1234567890,
endpoint="/v1/chat/completions",
input_file_id="file-abc123",
object="batch",
status="validating",
)
def test_output_expires_after_passthrough():
"""output_expires_after flows through create_batch to the provider."""
captured = {}
def capturing_create(**kwargs):
captured.update(kwargs)
mock_response = MagicMock()
mock_response.id = "batch_123"
return mock_response
with patch("litellm.batches.main.openai_batches_instance") as mock_instance:
mock_instance.create_batch.side_effect = capturing_create
litellm.create_batch(
completion_window="24h",
endpoint="/v1/chat/completions",
input_file_id="file-abc123",
output_expires_after=CALLER_EXPIRY,
custom_llm_provider="openai",
)
assert captured["create_batch_data"]["output_expires_after"] == CALLER_EXPIRY
class TestBatchEndpointTeamOverride:
"""Verify team-level enforced_batch_output_expires_after in the proxy endpoint."""
def _post_batch(
self,
monkeypatch,
llm_router: Router,
team_metadata: dict,
request_body: dict,
) -> dict:
"""POST /v1/batches with given team_metadata and body, return captured kwargs."""
_setup_proxy(monkeypatch, llm_router)
user_key = UserAPIKeyAuth(
api_key="test-key",
team_metadata=team_metadata,
)
app.dependency_overrides[user_api_key_auth] = lambda: user_key
captured_kwargs = {}
async def mock_acreate_batch(**kwargs):
captured_kwargs.update(kwargs)
return _make_batch_response()
monkeypatch.setattr(litellm, "acreate_batch", mock_acreate_batch)
try:
response = client.post(
"/v1/batches",
json=request_body,
headers={"Authorization": "Bearer test-key"},
)
assert response.status_code == 200
finally:
app.dependency_overrides.clear()
return captured_kwargs
def test_team_override_overrides_caller(self, monkeypatch, llm_router):
"""Team enforcement wins over caller-provided value."""
kwargs = self._post_batch(
monkeypatch,
llm_router,
team_metadata={
"enforced_batch_output_expires_after": TEAM_EXPIRY,
},
request_body={
"input_file_id": "file-abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"output_expires_after": CALLER_EXPIRY,
},
)
assert kwargs["output_expires_after"] == TEAM_EXPIRY
def test_no_team_setting_preserves_caller(self, monkeypatch, llm_router):
"""No team setting = caller value passes through."""
kwargs = self._post_batch(
monkeypatch,
llm_router,
team_metadata={},
request_body={
"input_file_id": "file-abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"output_expires_after": CALLER_EXPIRY,
},
)
assert kwargs["output_expires_after"] == CALLER_EXPIRY

59
tests/test_litellm/proxy/test_spend_log_cleanup.py
View file

@ -233,6 +233,65 @@ async def test_cleanup_old_spend_logs_no_retention_period():
mock_prisma_client.db.execute_raw.assert_not_called()
@pytest.mark.asyncio
async def test_lock_not_released_when_not_acquired():
"""
Lock release should be skipped when _should_delete_spend_logs returns False
before the lock is ever acquired.
"""
mock_prisma_client = MagicMock()
mock_prisma_client.db.execute_raw = AsyncMock()
mock_redis_cache = MagicMock()
mock_pod_lock_manager = MagicMock()
mock_pod_lock_manager.redis_cache = mock_redis_cache
mock_pod_lock_manager.acquire_lock = AsyncMock(return_value=True)
mock_pod_lock_manager.release_lock = AsyncMock()
# No retention setting → _should_delete_spend_logs() returns False before lock is acquired
cleaner = SpendLogCleanup(general_settings={})
cleaner.pod_lock_manager = mock_pod_lock_manager
await cleaner.cleanup_old_spend_logs(mock_prisma_client)
mock_pod_lock_manager.acquire_lock.assert_not_called()
mock_pod_lock_manager.release_lock.assert_not_called()
@pytest.mark.asyncio
async def test_integer_retention_treated_as_days():
"""
An integer value for maximum_spend_logs_retention_period should be treated
as days (e.g., 3 → '3d' → 259200 seconds).
"""
cleaner = SpendLogCleanup(
general_settings={"maximum_spend_logs_retention_period": 3}
)
result = cleaner._should_delete_spend_logs()
assert result is True
assert cleaner.retention_seconds == 3 * 86400 # 3 days in seconds
def test_string_retention_still_works():
"""
String values like '3d', '24h', '3600s' should continue to parse correctly.
"""
cases = [
("3d", 3 * 86400),
("24h", 24 * 3600),
("3600s", 3600),
("2w", 2 * 604800),
]
for setting, expected_seconds in cases:
cleaner = SpendLogCleanup(
general_settings={"maximum_spend_logs_retention_period": setting}
)
assert cleaner._should_delete_spend_logs() is True, f"Failed for {setting}"
assert cleaner.retention_seconds == expected_seconds, (
f"Expected {expected_seconds} for {setting}, got {cleaner.retention_seconds}"
)
def test_cleanup_batch_size_env_var(monkeypatch):
"""Ensure batch size is configurable via environment variable"""
import importlib

713
tests/test_litellm/router_utils/pre_call_checks/test_encrypted_content_affinity_check.py Normal file
View file

@ -0,0 +1,713 @@
"""
Tests for encrypted_content_affinity pre-call check.
The mechanism works without any cache and supports two encoding strategies:
1. **Items with IDs**: item IDs for output items with `encrypted_content` are rewritten to
`encitem_{base64("litellm:model_id:{model_id};item_id:{original_id}")}`.
2. **Items without IDs** (Codex): encrypted_content itself is wrapped with model_id metadata:
`litellm_enc:{base64("model_id:{model_id}")};{original_encrypted_content}`.
- On routing: `EncryptedContentAffinityCheck` decodes from either item IDs or wrapped
encrypted_content to extract `model_id` and pins the request to that deployment.
- Before forwarding: `_restore_encrypted_content_item_ids_in_input` decodes IDs and unwraps
encrypted_content back to their original forms before sending to the upstream provider.
"""
import os
import sys
from unittest.mock import AsyncMock, patch
import pytest
sys.path.insert(0, os.path.abspath("../.."))
import json
import litellm
from litellm.responses.utils import ResponsesAPIRequestUtils
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
class MockResponse:
def __init__(self, json_data, status_code):
self._json_data = json_data
self.status_code = status_code
self.text = json.dumps(json_data)
self.headers = {}
def json(self):
return self._json_data
def _get_item_id(item) -> str:
"""Extract item ID from either a Pydantic model or a dict."""
if isinstance(item, dict):
return item.get("id", "")
return getattr(item, "id", "") or ""
def _has_encrypted_content(item) -> bool:
"""Check whether an output item carries encrypted_content."""
if isinstance(item, dict):
return "encrypted_content" in item
return hasattr(item, "encrypted_content") and getattr(item, "encrypted_content") is not None
def _extract_encoded_item_id(response) -> str:
"""
Walk the response output and return the first litellm-encoded item ID
(i.e. one that starts with ``encitem_``).
"""
for item in response.output or []:
item_id = _get_item_id(item)
if item_id.startswith("encitem_"):
return item_id
return ""
# ---------------------------------------------------------------------------
# Unit tests for encoding / decoding utilities
# ---------------------------------------------------------------------------
class TestEncryptedItemIdCodec:
def test_roundtrip(self):
model_id = "deployment-1"
original_item_id = "rs_abc123def456"
encoded = ResponsesAPIRequestUtils._build_encrypted_item_id(model_id, original_item_id)
assert encoded.startswith("encitem_")
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(encoded)
assert decoded is not None
assert decoded["model_id"] == model_id
assert decoded["item_id"] == original_item_id
def test_decode_without_padding(self):
"""Decoding must succeed even if base64 padding (=) was stripped in transit."""
model_id = "gpt-5.1-codex-openai-2"
original_item_id = "rs_0efb96cb222403210069a01d5d52588196a9dc394ffdb89d00"
encoded = ResponsesAPIRequestUtils._build_encrypted_item_id(model_id, original_item_id)
# Strip any trailing '=' to simulate what happens in transit
stripped = encoded.rstrip("=")
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(stripped)
assert decoded is not None
assert decoded["model_id"] == model_id
assert decoded["item_id"] == original_item_id
def test_non_encoded_id_returns_none(self):
assert ResponsesAPIRequestUtils._decode_encrypted_item_id("rs_abc123") is None
assert ResponsesAPIRequestUtils._decode_encrypted_item_id("msg_abc") is None
assert ResponsesAPIRequestUtils._decode_encrypted_item_id("") is None
def test_semicolon_in_item_id(self):
"""item_id values containing ';' must survive the roundtrip."""
model_id = "deployment-1"
original_item_id = "rs_part1;part2;part3"
encoded = ResponsesAPIRequestUtils._build_encrypted_item_id(model_id, original_item_id)
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(encoded)
assert decoded is not None
assert decoded["item_id"] == original_item_id
class TestUpdateEncryptedContentItemIds:
def test_rewrites_encrypted_items_in_dict_response(self):
model_id = "deployment-1"
response = {
"id": "resp_123",
"output": [
{"id": "msg_abc", "type": "message", "content": []},
{"id": "rs_xyz", "type": "reasoning", "encrypted_content": "secret"},
],
}
result = ResponsesAPIRequestUtils._update_encrypted_content_item_ids_in_response(
response, model_id
)
# Plain message item untouched
assert result["output"][0]["id"] == "msg_abc"
# Reasoning item with encrypted_content gets encoded
encoded_id = result["output"][1]["id"]
assert encoded_id.startswith("encitem_")
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(encoded_id)
assert decoded["model_id"] == model_id
assert decoded["item_id"] == "rs_xyz"
def test_no_op_when_model_id_is_none(self):
response = {
"output": [{"id": "rs_xyz", "type": "reasoning", "encrypted_content": "secret"}]
}
result = ResponsesAPIRequestUtils._update_encrypted_content_item_ids_in_response(
response, None
)
assert result["output"][0]["id"] == "rs_xyz"
class TestEncryptedContentWrapping:
def test_wrap_and_unwrap_encrypted_content(self):
"""Test wrapping encrypted_content with model_id metadata."""
model_id = "deployment-1"
original_content = "gAAAAABpnW_yEYmSNEyOG_original_encrypted_data"
wrapped = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
original_content, model_id
)
assert wrapped.startswith("litellm_enc:")
assert wrapped != original_content
unwrapped_model_id, unwrapped_content = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(wrapped)
)
assert unwrapped_model_id == model_id
assert unwrapped_content == original_content
def test_unwrap_plain_encrypted_content(self):
"""Unwrapping plain encrypted_content returns None for model_id."""
plain_content = "gAAAAABpnW_yEYmSNEyOG_plain_content"
model_id, content = ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(
plain_content
)
assert model_id is None
assert content == plain_content
def test_update_response_wraps_encrypted_content_without_id(self):
"""Items with encrypted_content but no ID get the content wrapped."""
model_id = "deployment-1"
response = {
"id": "resp_123",
"output": [
{"type": "message", "content": []},
{
"type": "reasoning",
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG_secret",
},
],
}
result = ResponsesAPIRequestUtils._update_encrypted_content_item_ids_in_response(
response, model_id
)
assert result["output"][0].get("encrypted_content") is None
wrapped = result["output"][1]["encrypted_content"]
assert wrapped.startswith("litellm_enc:")
model_id_extracted, unwrapped = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(wrapped)
)
assert model_id_extracted == model_id
assert unwrapped == "gAAAAABpnW_yEYmSNEyOG_secret"
class TestRestoreEncryptedContentItemIds:
def test_restores_encoded_ids(self):
model_id = "deployment-1"
original_id = "rs_encrypted_item_456"
encoded_id = ResponsesAPIRequestUtils._build_encrypted_item_id(model_id, original_id)
request_input = [
{"type": "message", "id": "msg_abc123", "role": "assistant"},
{"type": "reasoning", "id": encoded_id, "encrypted_content": "secret"},
]
restored = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(
request_input
)
assert restored[0]["id"] == "msg_abc123"
assert restored[1]["id"] == original_id
def test_unwraps_encrypted_content(self):
"""Test that wrapped encrypted_content is unwrapped before forwarding."""
model_id = "deployment-1"
original_content = "gAAAAABpnW_yEYmSNEyOG_original"
wrapped_content = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
original_content, model_id
)
request_input = [
{"type": "reasoning", "encrypted_content": wrapped_content},
]
restored = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(
request_input
)
assert restored[0]["encrypted_content"] == original_content
def test_no_op_for_plain_string_input(self):
result = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(
"Hello world"
)
assert result == "Hello world"
def test_no_op_for_unencoded_ids(self):
request_input = [{"type": "message", "id": "msg_plain"}]
result = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(
request_input
)
assert result[0]["id"] == "msg_plain"
# ---------------------------------------------------------------------------
# Integration tests (router-level)
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_encrypted_content_affinity_tracks_and_routes():
"""
The first response rewrites encrypted-content item IDs to encoded form.
The follow-up request with those encoded IDs is pinned to the same deployment.
"""
mock_response_data = {
"id": "resp_mock-123",
"object": "response",
"created_at": 1741476542,
"status": "completed",
"model": "openai/gpt-5.1-codex",
"output": [
{
"type": "message",
"id": "msg_abc123",
"status": "completed",
"role": "assistant",
"content": [{"type": "output_text", "text": "Hello!", "annotations": []}],
},
{
"type": "reasoning",
"id": "rs_encrypted_item_456",
"status": "completed",
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG...",
},
],
"parallel_tool_calls": True,
"usage": {"input_tokens": 5, "output_tokens": 10, "total_tokens": 15},
"error": None,
}
router = litellm.Router(
model_list=[
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-1",
},
"model_info": {"id": "deployment-1"},
},
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-2",
},
"model_info": {"id": "deployment-2"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
)
selected_deployments = []
def deterministic_choice(seq):
if len(selected_deployments) == 0:
return seq[0]
return seq[1] if len(seq) > 1 else seq[0]
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post, patch(
"litellm.router_strategy.simple_shuffle.random.choice",
side_effect=deterministic_choice,
):
mock_post.return_value = MockResponse(mock_response_data, 200)
# First request — goes to deployment-1 via deterministic_choice
first_response = await router.aresponses(
model="openai.gpt-5.1-codex",
input="Hello, how are you?",
)
first_model_id = first_response._hidden_params["model_id"]
selected_deployments.append(first_model_id)
# The response must have rewritten the encrypted item's ID to encoded form
encoded_item_id = _extract_encoded_item_id(first_response)
assert encoded_item_id.startswith("encitem_"), (
f"Expected output item ID to be rewritten to encitem_... but got {encoded_item_id!r}"
)
# Verify the encoded ID decodes back to the correct deployment + original ID
decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(encoded_item_id)
assert decoded is not None
assert decoded["model_id"] == first_model_id
assert decoded["item_id"] == "rs_encrypted_item_456"
# Second request: use the encoded item IDs from the first response
second_response = await router.aresponses(
model="openai.gpt-5.1-codex",
input=[
{"type": "message", "id": "msg_abc123", "role": "assistant"},
{
"type": "reasoning",
"id": encoded_item_id,
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG...",
},
],
)
second_model_id = second_response._hidden_params["model_id"]
assert second_model_id == first_model_id, (
f"Expected affinity to route to {first_model_id}, but got {second_model_id}"
)
@pytest.mark.asyncio
async def test_encrypted_content_affinity_no_effect_on_chat_completions():
"""
Encrypted content affinity should not affect regular chat completions.
"""
router = litellm.Router(
model_list=[
{
"model_name": "gpt-3.5-turbo",
"litellm_params": {
"model": "gpt-3.5-turbo",
"api_key": "test-key",
"mock_response": "Hello from chat completion!",
},
"model_info": {"id": "chat-deployment-1"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
)
response1 = await router.acompletion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello"}],
)
response2 = await router.acompletion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello again"}],
)
assert response1.id is not None
assert response2.id is not None
@pytest.mark.asyncio
async def test_encrypted_content_affinity_bypasses_rpm_limits():
"""
When encrypted content affinity pins to a deployment, the request
goes through even if normal routing would avoid it.
"""
mock_response_data = {
"id": "resp_mock-rpm-test",
"object": "response",
"created_at": 1741476542,
"status": "completed",
"model": "openai/gpt-5.1-codex",
"output": [
{
"type": "reasoning",
"id": "rs_encrypted_must_pin",
"status": "completed",
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG...",
},
],
"usage": {"input_tokens": 5, "output_tokens": 10, "total_tokens": 15},
"error": None,
}
router = litellm.Router(
model_list=[
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-1",
},
"model_info": {"id": "deployment-alpha"},
},
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-2",
},
"model_info": {"id": "deployment-beta"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
routing_strategy="usage-based-routing-v2",
)
selected_deployments = []
def deterministic_choice(seq):
if len(selected_deployments) == 0:
return seq[0]
return seq[1] if len(seq) > 1 else seq[0]
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post, patch(
"litellm.router_strategy.simple_shuffle.random.choice",
side_effect=deterministic_choice,
):
mock_post.return_value = MockResponse(mock_response_data, 200)
first_response = await router.aresponses(
model="openai.gpt-5.1-codex",
input="Initial request",
)
first_model_id = first_response._hidden_params["model_id"]
selected_deployments.append(first_model_id)
# Extract encoded item ID from the first response output
encoded_item_id = _extract_encoded_item_id(first_response)
assert encoded_item_id.startswith("encitem_"), (
f"Expected encitem_... but got {encoded_item_id!r}"
)
# Follow-up with the encoded item ID — should pin to same deployment
second_response = await router.aresponses(
model="openai.gpt-5.1-codex",
input=[
{
"type": "reasoning",
"id": encoded_item_id,
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG...",
},
],
)
second_model_id = second_response._hidden_params["model_id"]
assert second_model_id == first_model_id
@pytest.mark.asyncio
async def test_encrypted_content_affinity_no_match_normal_routing():
"""
Input items with non-encoded IDs (no encitem_ prefix) fall through to
normal load balancing.
"""
mock_response_data = {
"id": "resp_mock-no-match",
"object": "response",
"created_at": 1741476542,
"status": "completed",
"model": "openai/gpt-5.1-codex",
"output": [
{
"type": "message",
"id": "msg_new",
"status": "completed",
"role": "assistant",
"content": [{"type": "output_text", "text": "Response"}],
},
],
"usage": {"input_tokens": 5, "output_tokens": 10, "total_tokens": 15},
"error": None,
}
router = litellm.Router(
model_list=[
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-1",
},
"model_info": {"id": "deployment-a"},
},
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-2",
},
"model_info": {"id": "deployment-b"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
)
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
mock_post.return_value = MockResponse(mock_response_data, 200)
# Non-encoded item ID — no affinity should kick in
response = await router.aresponses(
model="openai.gpt-5.1-codex",
input=[
{"type": "message", "id": "unknown_item_id_12345"},
],
)
assert response.id is not None
@pytest.mark.asyncio
async def test_encrypted_content_affinity_with_wrapped_content_no_id():
"""
Test affinity routing when items have wrapped encrypted_content but no ID.
This simulates Codex client behavior where IDs are omitted.
"""
mock_response_data = {
"id": "resp_mock-wrapped-content",
"object": "response",
"created_at": 1741476542,
"status": "completed",
"model": "openai/gpt-5.1-codex",
"output": [
{
"type": "reasoning",
"status": "completed",
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG_original_content",
},
],
"usage": {"input_tokens": 5, "output_tokens": 10, "total_tokens": 15},
"error": None,
}
router = litellm.Router(
model_list=[
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-1",
},
"model_info": {"id": "deployment-1"},
},
{
"model_name": "openai.gpt-5.1-codex",
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-2",
},
"model_info": {"id": "deployment-2"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
)
selected_deployments = []
def deterministic_choice(seq):
if len(selected_deployments) == 0:
return seq[0]
return seq[1] if len(seq) > 1 else seq[0]
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post, patch(
"litellm.router_strategy.simple_shuffle.random.choice",
side_effect=deterministic_choice,
):
mock_post.return_value = MockResponse(mock_response_data, 200)
# First request — goes to deployment-1
first_response = await router.aresponses(
model="openai.gpt-5.1-codex",
input="Hello, how are you?",
)
first_model_id = first_response._hidden_params["model_id"]
selected_deployments.append(first_model_id)
# Extract wrapped encrypted_content from first response
first_item = first_response.output[0]
wrapped_content = (
first_item.encrypted_content
if hasattr(first_item, "encrypted_content")
else first_item.get("encrypted_content")
)
assert wrapped_content.startswith("litellm_enc:"), (
f"Expected wrapped content but got {wrapped_content[:50]}..."
)
# Verify we can extract model_id from wrapped content
extracted_model_id, _ = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(
wrapped_content
)
)
assert extracted_model_id == first_model_id
# Second request: use wrapped encrypted_content WITHOUT an ID (Codex behavior)
second_response = await router.aresponses(
model="openai.gpt-5.1-codex",
input=[
{
"type": "reasoning",
"encrypted_content": wrapped_content,
},
],
)
second_model_id = second_response._hidden_params["model_id"]
assert second_model_id == first_model_id, (
f"Expected affinity to route to {first_model_id}, but got {second_model_id}"
)
def test_encrypted_content_wrapping_preserves_original_content():
"""
Test that wrapping and unwrapping encrypted_content preserves the original content.
This is critical for streaming responses where content must round-trip correctly.
"""
model_id = "test-deployment-1"
original_encrypted_content = "gAAAAABpnW_yEYmSNEyOG_streaming_test_content_with_special_chars==+/"
wrapped = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
original_encrypted_content, model_id
)
assert wrapped.startswith("litellm_enc:")
assert wrapped != original_encrypted_content
extracted_model_id, unwrapped_content = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(wrapped)
)
assert extracted_model_id == model_id
assert unwrapped_content == original_encrypted_content
def test_encrypted_content_wrapping_with_multiple_semicolons():
"""
Test that encrypted_content containing semicolons is handled correctly.
"""
model_id = "deployment-with-semicolons"
original_content = "gAAAAAB;some;content;with;semicolons"
wrapped = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
original_content, model_id
)
extracted_model_id, unwrapped = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(wrapped)
)
assert extracted_model_id == model_id
assert unwrapped == original_content
def test_encrypted_content_wrapping_empty_string():
"""
Test that empty encrypted_content is handled gracefully.
"""
model_id = "test-deployment"
original_content = ""
wrapped = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
original_content, model_id
)
assert wrapped.startswith("litellm_enc:")
extracted_model_id, unwrapped = (
ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(wrapped)
)
assert extracted_model_id == model_id
assert unwrapped == original_content

25
tests/test_litellm/test_utils.py
View file

@ -1139,20 +1139,25 @@ def test_pre_process_non_default_params(model, custom_llm_provider):
provider_config=provider_config,
)
print(processed_non_default_params)
# Vertex AI / Gemini uses Pydantic's model_json_schema() which doesn't
# include additionalProperties: False (Gemini rejects it). Other
# providers use OpenAI's to_strict_json_schema() which does.
expected_schema = {
"properties": {
"x": {"title": "X", "type": "string"},
"y": {"title": "Y", "type": "string"},
},
"required": ["x", "y"],
"title": "ResponseFormat",
"type": "object",
}
if custom_llm_provider not in ("vertex_ai", "vertex_ai_beta", "gemini"):
expected_schema["additionalProperties"] = False
assert processed_non_default_params == {
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"properties": {
"x": {"title": "X", "type": "string"},
"y": {"title": "Y", "type": "string"},
},
"required": ["x", "y"],
"title": "ResponseFormat",
"type": "object",
"additionalProperties": False,
},
"schema": expected_schema,
"name": "ResponseFormat",
"strict": True,
},

3
tests/test_litellm/types/test_types_utils.py
View file

@ -120,7 +120,8 @@ def test_usage_completion_tokens_details_text_tokens():
'reasoning_tokens': 65,
'rejected_prediction_tokens': None,
'text_tokens': 12,
'image_tokens': None
'image_tokens': None,
'video_tokens': None
}
assert dump_result['completion_tokens_details'] == expected_completion_details

66
ui/litellm-dashboard/src/app/(dashboard)/hooks/keys/useResetKeySpend.ts Normal file
View file

@ -0,0 +1,66 @@
import { useMutation, useQueryClient } from "@tanstack/react-query";
import {
getProxyBaseUrl,
getGlobalLitellmHeaderName,
deriveErrorMessage,
handleError,
} from "@/components/networking";
import useAuthorized from "@/app/(dashboard)/hooks/useAuthorized";
import { keyKeys } from "./useKeys";
// ── Types ─────────────────────────────────────────────────────────────────────
export interface ResetKeySpendResponse {
key_hash: string;
spend: number;
previous_spend: number;
max_budget: number | null;
budget_reset_at: string | null;
}
// ── Fetch function ────────────────────────────────────────────────────────────
export const resetKeySpend = async (
accessToken: string,
keyToken: string,
): Promise<ResetKeySpendResponse> => {
const baseUrl = getProxyBaseUrl();
const url = `${baseUrl ? `${baseUrl}/key/${keyToken}/reset_spend` : `/key/${keyToken}/reset_spend`}`;
const response = await fetch(url, {
method: "POST",
headers: {
[getGlobalLitellmHeaderName()]: `Bearer ${accessToken}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ reset_to: 0 }),
});
if (!response.ok) {
const errorData = await response.json();
const errorMessage = deriveErrorMessage(errorData);
handleError(errorMessage);
throw new Error(errorMessage);
}
return response.json();
};
// ── Hook ──────────────────────────────────────────────────────────────────────
export const useResetKeySpend = () => {
const { accessToken } = useAuthorized();
const queryClient = useQueryClient();
return useMutation<ResetKeySpendResponse, Error, string>({
mutationFn: async (keyToken) => {
if (!accessToken) {
throw new Error("Access token is required");
}
return resetKeySpend(accessToken, keyToken);
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: keyKeys.all });
},
});
};

111
ui/litellm-dashboard/src/app/(dashboard)/hooks/projects/useCreateProject.test.ts Normal file
View file

@ -0,0 +1,111 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import React, { ReactNode } from "react";
import { useCreateProject, ProjectCreateParams } from "./useCreateProject";
import { projectKeys, ProjectResponse } from "./useProjects";
vi.mock("@/components/networking", () => ({
getProxyBaseUrl: vi.fn(() => ""),
getGlobalLitellmHeaderName: vi.fn(() => "Authorization"),
deriveErrorMessage: vi.fn((data: any) => data?.error || "Error"),
handleError: vi.fn(),
}));
const mockUseAuthorized = vi.fn();
vi.mock("@/app/(dashboard)/hooks/useAuthorized", () => ({
default: () => mockUseAuthorized(),
}));
const mockProject: ProjectResponse = {
project_id: "proj-1",
project_alias: "Test Project",
description: "A test project",
team_id: "team-1",
budget_id: null,
metadata: null,
models: ["gpt-4"],
spend: 25.0,
model_spend: null,
model_rpm_limit: null,
model_tpm_limit: null,
blocked: false,
object_permission_id: null,
created_at: "2024-01-01T00:00:00Z",
created_by: "user-1",
updated_at: "2024-01-02T00:00:00Z",
updated_by: "user-1",
litellm_budget_table: null,
};
function makeWrapper(queryClient: QueryClient) {
return ({ children }: { children: ReactNode }) =>
React.createElement(QueryClientProvider, { client: queryClient }, children);
}
describe("useCreateProject", () => {
let queryClient: QueryClient;
beforeEach(() => {
queryClient = new QueryClient({
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
});
vi.clearAllMocks();
global.fetch = vi.fn();
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Admin" });
});
it("should render", () => {
const { result } = renderHook(() => useCreateProject(), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.mutate).toBeDefined();
});
it("should POST to /project/new and return the created project", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
const { result } = renderHook(() => useCreateProject(), {
wrapper: makeWrapper(queryClient),
});
const params: ProjectCreateParams = { team_id: "team-1", project_alias: "New Project" };
const data = await result.current.mutateAsync(params);
expect(data).toEqual(mockProject);
const [url, init] = (global.fetch as any).mock.calls[0];
expect(url).toContain("/project/new");
expect(init.method).toBe("POST");
expect(JSON.parse(init.body)).toMatchObject(params);
});
it("should invalidate project queries on success", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
const invalidateSpy = vi.spyOn(queryClient, "invalidateQueries");
const { result } = renderHook(() => useCreateProject(), {
wrapper: makeWrapper(queryClient),
});
await result.current.mutateAsync({ team_id: "team-1" });
expect(invalidateSpy).toHaveBeenCalledWith({ queryKey: projectKeys.all });
});
it("should set isError when the request fails", async () => {
(global.fetch as any).mockResolvedValue({
ok: false,
json: async () => ({ error: "Server error" }),
});
const { result } = renderHook(() => useCreateProject(), {
wrapper: makeWrapper(queryClient),
});
result.current.mutateAsync({ team_id: "team-1" }).catch(() => {});
await waitFor(() => expect(result.current.isError).toBe(true));
});
it("should throw when accessToken is missing", async () => {
mockUseAuthorized.mockReturnValue({ accessToken: null, userRole: "Admin" });
const { result } = renderHook(() => useCreateProject(), {
wrapper: makeWrapper(queryClient),
});
await expect(result.current.mutateAsync({ team_id: "team-1" })).rejects.toThrow(
"Access token is required"
);
expect(global.fetch).not.toHaveBeenCalled();
});
});

88
ui/litellm-dashboard/src/app/(dashboard)/hooks/projects/useDeleteProject.test.ts Normal file
View file

@ -0,0 +1,88 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import React, { ReactNode } from "react";
import { useDeleteProject } from "./useDeleteProject";
import { projectKeys } from "./useProjects";
vi.mock("@/components/networking", () => ({
getProxyBaseUrl: vi.fn(() => ""),
getGlobalLitellmHeaderName: vi.fn(() => "Authorization"),
deriveErrorMessage: vi.fn((data: any) => data?.error || "Error"),
handleError: vi.fn(),
}));
const mockUseAuthorized = vi.fn();
vi.mock("@/app/(dashboard)/hooks/useAuthorized", () => ({
default: () => mockUseAuthorized(),
}));
function makeWrapper(queryClient: QueryClient) {
return ({ children }: { children: ReactNode }) =>
React.createElement(QueryClientProvider, { client: queryClient }, children);
}
describe("useDeleteProject", () => {
let queryClient: QueryClient;
beforeEach(() => {
queryClient = new QueryClient({
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
});
vi.clearAllMocks();
global.fetch = vi.fn();
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Admin" });
});
it("should render", () => {
const { result } = renderHook(() => useDeleteProject(), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.mutate).toBeDefined();
});
it("should send DELETE to /project/delete with the given project IDs", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => ({}) });
const { result } = renderHook(() => useDeleteProject(), {
wrapper: makeWrapper(queryClient),
});
await result.current.mutateAsync(["proj-1", "proj-2"]);
const [url, init] = (global.fetch as any).mock.calls[0];
expect(url).toContain("/project/delete");
expect(init.method).toBe("DELETE");
expect(JSON.parse(init.body)).toEqual({ project_ids: ["proj-1", "proj-2"] });
});
it("should invalidate project queries on success", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => ({}) });
const invalidateSpy = vi.spyOn(queryClient, "invalidateQueries");
const { result } = renderHook(() => useDeleteProject(), {
wrapper: makeWrapper(queryClient),
});
await result.current.mutateAsync(["proj-1"]);
expect(invalidateSpy).toHaveBeenCalledWith({ queryKey: projectKeys.all });
});
it("should set isError when the request fails", async () => {
(global.fetch as any).mockResolvedValue({
ok: false,
json: async () => ({ error: "Not found" }),
});
const { result } = renderHook(() => useDeleteProject(), {
wrapper: makeWrapper(queryClient),
});
result.current.mutateAsync(["proj-1"]).catch(() => {});
await waitFor(() => expect(result.current.isError).toBe(true));
});
it("should throw when accessToken is missing", async () => {
mockUseAuthorized.mockReturnValue({ accessToken: null, userRole: "Admin" });
const { result } = renderHook(() => useDeleteProject(), {
wrapper: makeWrapper(queryClient),
});
await expect(result.current.mutateAsync(["proj-1"])).rejects.toThrow(
"Access token is required"
);
expect(global.fetch).not.toHaveBeenCalled();
});
});

144
ui/litellm-dashboard/src/app/(dashboard)/hooks/projects/useProjectDetails.test.ts Normal file
View file

@ -0,0 +1,144 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import React, { ReactNode } from "react";
import { useProjectDetails } from "./useProjectDetails";
import { projectKeys, ProjectResponse } from "./useProjects";
vi.mock("@/components/networking", () => ({
getProxyBaseUrl: vi.fn(() => ""),
getGlobalLitellmHeaderName: vi.fn(() => "Authorization"),
deriveErrorMessage: vi.fn((data: any) => data?.error || "Error"),
handleError: vi.fn(),
}));
const mockUseAuthorized = vi.fn();
vi.mock("@/app/(dashboard)/hooks/useAuthorized", () => ({
default: () => mockUseAuthorized(),
}));
const mockProject: ProjectResponse = {
project_id: "proj-1",
project_alias: "Test Project",
description: "A test project",
team_id: "team-1",
budget_id: null,
metadata: null,
models: ["gpt-4"],
spend: 25.0,
model_spend: null,
model_rpm_limit: null,
model_tpm_limit: null,
blocked: false,
object_permission_id: null,
created_at: "2024-01-01T00:00:00Z",
created_by: "user-1",
updated_at: "2024-01-02T00:00:00Z",
updated_by: "user-1",
litellm_budget_table: null,
};
const mockProjects: ProjectResponse[] = [
mockProject,
{ ...mockProject, project_id: "proj-2", project_alias: "Test Project 2" },
];
function makeWrapper(queryClient: QueryClient) {
return ({ children }: { children: ReactNode }) =>
React.createElement(QueryClientProvider, { client: queryClient }, children);
}
describe("useProjectDetails", () => {
let queryClient: QueryClient;
beforeEach(() => {
queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
vi.clearAllMocks();
global.fetch = vi.fn();
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Admin" });
});
it("should render", () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
const { result } = renderHook(() => useProjectDetails("proj-1"), {
wrapper: makeWrapper(queryClient),
});
expect(result.current).toBeDefined();
});
it("should return project details when the request succeeds", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
const { result } = renderHook(() => useProjectDetails("proj-1"), {
wrapper: makeWrapper(queryClient),
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
expect(result.current.data).toEqual(mockProject);
});
it("should call /project/info with the projectId encoded as a query param", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
renderHook(() => useProjectDetails("proj-1"), { wrapper: makeWrapper(queryClient) });
await waitFor(() => expect(global.fetch).toHaveBeenCalled());
const [url] = (global.fetch as any).mock.calls[0];
expect(url).toContain("/project/info");
expect(url).toContain("project_id=proj-1");
});
it("should not fetch when projectId is missing", () => {
const { result } = renderHook(() => useProjectDetails(undefined), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.isFetched).toBe(false);
expect(global.fetch).not.toHaveBeenCalled();
});
it("should not fetch when accessToken is missing", () => {
mockUseAuthorized.mockReturnValue({ accessToken: null, userRole: "Admin" });
const { result } = renderHook(() => useProjectDetails("proj-1"), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.isFetched).toBe(false);
expect(global.fetch).not.toHaveBeenCalled();
});
it("should not fetch when userRole is not an admin role", () => {
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Internal User" });
const { result } = renderHook(() => useProjectDetails("proj-1"), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.isFetched).toBe(false);
expect(global.fetch).not.toHaveBeenCalled();
});
it("should seed initialData from the projects list cache", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
queryClient.setQueryData(projectKeys.list({}), mockProjects);
const { result } = renderHook(() => useProjectDetails("proj-1"), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.data).toEqual(mockProject);
expect(result.current.isLoading).toBe(false);
await waitFor(() => expect(result.current.isFetching).toBe(false));
});
it("should return undefined initialData when projectId is not in the cache", () => {
queryClient.setQueryData(projectKeys.list({}), mockProjects);
const { result } = renderHook(() => useProjectDetails("non-existent"), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.data).toBeUndefined();
});
it("should set isError when the request fails", async () => {
(global.fetch as any).mockResolvedValue({
ok: false,
json: async () => ({ error: "Not found" }),
});
const { result } = renderHook(() => useProjectDetails("proj-1"), {
wrapper: makeWrapper(queryClient),
});
await waitFor(() => expect(result.current.isError).toBe(true));
});
});

124
ui/litellm-dashboard/src/app/(dashboard)/hooks/projects/useProjects.test.ts Normal file
View file

@ -0,0 +1,124 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import React, { ReactNode } from "react";
import { useProjects, ProjectResponse } from "./useProjects";
vi.mock("@/components/networking", () => ({
getProxyBaseUrl: vi.fn(() => ""),
getGlobalLitellmHeaderName: vi.fn(() => "Authorization"),
deriveErrorMessage: vi.fn((data: any) => data?.error || "Error"),
handleError: vi.fn(),
}));
const mockUseAuthorized = vi.fn();
vi.mock("@/app/(dashboard)/hooks/useAuthorized", () => ({
default: () => mockUseAuthorized(),
}));
const mockProjects: ProjectResponse[] = [
{
project_id: "proj-1",
project_alias: "Test Project",
description: "A test project",
team_id: "team-1",
budget_id: null,
metadata: null,
models: ["gpt-4"],
spend: 25.0,
model_spend: null,
model_rpm_limit: null,
model_tpm_limit: null,
blocked: false,
object_permission_id: null,
created_at: "2024-01-01T00:00:00Z",
created_by: "user-1",
updated_at: "2024-01-02T00:00:00Z",
updated_by: "user-1",
litellm_budget_table: null,
},
{
project_id: "proj-2",
project_alias: "Test Project 2",
description: null,
team_id: "team-1",
budget_id: null,
metadata: null,
models: [],
spend: 0,
model_spend: null,
model_rpm_limit: null,
model_tpm_limit: null,
blocked: false,
object_permission_id: null,
created_at: "2024-01-03T00:00:00Z",
created_by: "user-1",
updated_at: "2024-01-03T00:00:00Z",
updated_by: "user-1",
litellm_budget_table: null,
},
];
function makeWrapper(queryClient: QueryClient) {
return ({ children }: { children: ReactNode }) =>
React.createElement(QueryClientProvider, { client: queryClient }, children);
}
describe("useProjects", () => {
let queryClient: QueryClient;
beforeEach(() => {
queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
vi.clearAllMocks();
global.fetch = vi.fn();
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Admin" });
});
it("should render", () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProjects });
const { result } = renderHook(() => useProjects(), { wrapper: makeWrapper(queryClient) });
expect(result.current).toBeDefined();
});
it("should return projects when the request succeeds", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProjects });
const { result } = renderHook(() => useProjects(), { wrapper: makeWrapper(queryClient) });
await waitFor(() => expect(result.current.isSuccess).toBe(true));
expect(result.current.data).toEqual(mockProjects);
});
it("should call GET /project/list with the auth header", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProjects });
renderHook(() => useProjects(), { wrapper: makeWrapper(queryClient) });
await waitFor(() => expect(global.fetch).toHaveBeenCalled());
const [url, init] = (global.fetch as any).mock.calls[0];
expect(url).toContain("/project/list");
expect(init.headers["Authorization"]).toBe("Bearer test-token");
});
it("should set isError when the request fails", async () => {
(global.fetch as any).mockResolvedValue({
ok: false,
json: async () => ({ error: "Not authorized" }),
});
const { result } = renderHook(() => useProjects(), { wrapper: makeWrapper(queryClient) });
await waitFor(() => expect(result.current.isError).toBe(true));
expect(result.current.data).toBeUndefined();
});
it("should not fetch when accessToken is missing", () => {
mockUseAuthorized.mockReturnValue({ accessToken: null, userRole: "Admin" });
const { result } = renderHook(() => useProjects(), { wrapper: makeWrapper(queryClient) });
expect(result.current.isFetched).toBe(false);
expect(global.fetch).not.toHaveBeenCalled();
});
it("should not fetch when userRole is not an admin role", () => {
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Internal User" });
const { result } = renderHook(() => useProjects(), { wrapper: makeWrapper(queryClient) });
expect(result.current.isFetched).toBe(false);
expect(global.fetch).not.toHaveBeenCalled();
});
});

116
ui/litellm-dashboard/src/app/(dashboard)/hooks/projects/useUpdateProject.test.ts Normal file
View file

@ -0,0 +1,116 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import React, { ReactNode } from "react";
import { useUpdateProject } from "./useUpdateProject";
import { projectKeys, ProjectResponse } from "./useProjects";
vi.mock("@/components/networking", () => ({
getProxyBaseUrl: vi.fn(() => ""),
getGlobalLitellmHeaderName: vi.fn(() => "Authorization"),
deriveErrorMessage: vi.fn((data: any) => data?.error || "Error"),
handleError: vi.fn(),
}));
const mockUseAuthorized = vi.fn();
vi.mock("@/app/(dashboard)/hooks/useAuthorized", () => ({
default: () => mockUseAuthorized(),
}));
const mockProject: ProjectResponse = {
project_id: "proj-1",
project_alias: "Test Project",
description: "A test project",
team_id: "team-1",
budget_id: null,
metadata: null,
models: ["gpt-4"],
spend: 25.0,
model_spend: null,
model_rpm_limit: null,
model_tpm_limit: null,
blocked: false,
object_permission_id: null,
created_at: "2024-01-01T00:00:00Z",
created_by: "user-1",
updated_at: "2024-01-02T00:00:00Z",
updated_by: "user-1",
litellm_budget_table: null,
};
function makeWrapper(queryClient: QueryClient) {
return ({ children }: { children: ReactNode }) =>
React.createElement(QueryClientProvider, { client: queryClient }, children);
}
describe("useUpdateProject", () => {
let queryClient: QueryClient;
beforeEach(() => {
queryClient = new QueryClient({
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
});
vi.clearAllMocks();
global.fetch = vi.fn();
mockUseAuthorized.mockReturnValue({ accessToken: "test-token", userRole: "Admin" });
});
it("should render", () => {
const { result } = renderHook(() => useUpdateProject(), {
wrapper: makeWrapper(queryClient),
});
expect(result.current.mutate).toBeDefined();
});
it("should POST to /project/update and return the updated project", async () => {
const updated = { ...mockProject, project_alias: "Updated Name" };
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => updated });
const { result } = renderHook(() => useUpdateProject(), {
wrapper: makeWrapper(queryClient),
});
const data = await result.current.mutateAsync({
projectId: "proj-1",
params: { project_alias: "Updated Name" },
});
expect(data).toEqual(updated);
const [url, init] = (global.fetch as any).mock.calls[0];
expect(url).toContain("/project/update");
expect(JSON.parse(init.body)).toMatchObject({
project_id: "proj-1",
project_alias: "Updated Name",
});
});
it("should invalidate project queries on success", async () => {
(global.fetch as any).mockResolvedValue({ ok: true, json: async () => mockProject });
const invalidateSpy = vi.spyOn(queryClient, "invalidateQueries");
const { result } = renderHook(() => useUpdateProject(), {
wrapper: makeWrapper(queryClient),
});
await result.current.mutateAsync({ projectId: "proj-1", params: {} });
expect(invalidateSpy).toHaveBeenCalledWith({ queryKey: projectKeys.all });
});
it("should set isError when the request fails", async () => {
(global.fetch as any).mockResolvedValue({
ok: false,
json: async () => ({ error: "Server error" }),
});
const { result } = renderHook(() => useUpdateProject(), {
wrapper: makeWrapper(queryClient),
});
result.current.mutateAsync({ projectId: "proj-1", params: {} }).catch(() => {});
await waitFor(() => expect(result.current.isError).toBe(true));
});
it("should throw when accessToken is missing", async () => {
mockUseAuthorized.mockReturnValue({ accessToken: null, userRole: "Admin" });
const { result } = renderHook(() => useUpdateProject(), {
wrapper: makeWrapper(queryClient),
});
await expect(
result.current.mutateAsync({ projectId: "proj-1", params: {} })
).rejects.toThrow("Access token is required");
expect(global.fetch).not.toHaveBeenCalled();
});
});

1
ui/litellm-dashboard/src/components/add_model/AddModelForm.tsx
View file

@ -358,6 +358,7 @@ const AddModelForm: React.FC<AddModelFormProps> = ({
teams={teams}
guardrailsList={guardrailsList || []}
tagsList={tagsList || {}}
accessToken={accessToken || ""}
/>
</>
)}

Some files were not shown because too many files have changed in this diff Show more