mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-06 08:16:43 +00:00
Add docs for encrypted_content_affinity
This commit is contained in:
Sameer Kankute
parent
7ab577f836
commit
863867c4fe
3 changed files with 189 additions and 1 deletions
|
|
@ -359,7 +359,7 @@ router_settings:
|
|||
| redis_url | str | URL for Redis server. **Known performance issue with Redis URL.** |
|
||||
| cache_responses | boolean | Flag to enable caching LLM Responses, if cache set under `router_settings`. If true, caches responses. Defaults to False. |
|
||||
| router_general_settings | RouterGeneralSettings | [SDK-Only] Router general settings - contains optimizations like 'async_only_mode'. [Docs](../routing.md#router-general-settings) |
|
||||
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `deployment_affinity`, `forward_client_headers_by_model_group` |
|
||||
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `encrypted_content_affinity`, `deployment_affinity`, `session_affinity`, `forward_client_headers_by_model_group` |
|
||||
| deployment_affinity_ttl_seconds | int | TTL (seconds) for user-key → deployment affinity mapping when `deployment_affinity` is enabled (configured at Router init / proxy startup). Defaults to `3600` (1 hour). |
|
||||
| ignore_invalid_deployments | boolean | If true, ignores invalid deployments. Default for proxy is True - to prevent invalid models from blocking other models from being loaded. |
|
||||
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search.md) |
|
||||
|
|
|
|||
|
|
@ -347,3 +347,36 @@ If `order=1` deployment is unavailable (e.g., rate-limited), the router falls ba
|
|||
- **Higher throughput**: More requests handled simultaneously across deployments
|
||||
- **Improved reliability**: If one deployment fails, traffic automatically routes to healthy ones
|
||||
- **Better resource utilization**: Load spread evenly across all available deployments
|
||||
|
||||
## Special Considerations for Responses API
|
||||
|
||||
When load balancing OpenAI's Responses API across deployments with **different API keys** (e.g., different Azure regions or organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the originating API key.
|
||||
|
||||
**Solution:** Use the `encrypted_content_affinity` pre-call check to automatically route follow-up requests containing encrypted items to the correct deployment:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-5.1-codex
|
||||
litellm_params:
|
||||
model: azure/gpt-5.1-codex
|
||||
api_base: https://eastus.openai.azure.com/
|
||||
api_key: os.environ/AZURE_API_KEY_EASTUS
|
||||
model_info:
|
||||
id: "deployment-eastus"
|
||||
|
||||
- model_name: gpt-5.1-codex
|
||||
litellm_params:
|
||||
model: azure/gpt-5.1-codex
|
||||
api_base: https://westeurope.openai.azure.com/
|
||||
api_key: os.environ/AZURE_API_KEY_WESTEUROPE
|
||||
model_info:
|
||||
id: "deployment-westeurope"
|
||||
|
||||
router_settings:
|
||||
optional_pre_call_checks:
|
||||
- encrypted_content_affinity # 👈 Prevents invalid_encrypted_content errors
|
||||
```
|
||||
|
||||
This ensures requests containing encrypted content are routed to the deployment that created them, while other requests continue to load balance normally.
|
||||
|
||||
**[Learn more about Encrypted Content Affinity →](../response_api.md#encrypted-content-affinity-multi-region-load-balancing)**
|
||||
|
|
|
|||
|
|
@ -920,9 +920,14 @@ follow_up = await router.aresponses(
|
|||
To enable session continuity for Responses API in your LiteLLM proxy, set `optional_pre_call_checks` in your proxy config.yaml.
|
||||
|
||||
- `responses_api_deployment_check`: high priority routing when `previous_response_id` is provided
|
||||
- `encrypted_content_affinity`: **[Recommended]** content-aware routing for encrypted items (e.g., `rs_...` reasoning items)
|
||||
- `session_affinity`: sticky sessions based on session id (takes priority over `deployment_affinity`)
|
||||
- `deployment_affinity`: sticky sessions based on user key (applies even without `previous_response_id`)
|
||||
|
||||
:::tip Recommended: Use `encrypted_content_affinity`
|
||||
For Responses API with load balancing across deployments with **different API keys**, use `encrypted_content_affinity` instead of `deployment_affinity`. It only pins requests that contain encrypted content, avoiding quota reduction while preventing `invalid_encrypted_content` errors.
|
||||
:::
|
||||
|
||||
Notes:
|
||||
- User-key affinity is keyed on `metadata.user_api_key_hash` (the API key hash). The OpenAI `user` request parameter is an end-user identifier and is intentionally not used for deployment affinity.
|
||||
- Session-ID affinity is keyed on `metadata.session_id`. For proxy requests, this can be passed via the `x-litellm-session-id` HTTP header. For Python SDK requests, you can pass it via `litellm_metadata={"session_id": "value"}` in request args.
|
||||
|
|
@ -983,6 +988,156 @@ follow_up = client.responses.create(
|
|||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Encrypted Content Affinity (Multi-Region Load Balancing)
|
||||
|
||||
When load balancing Responses API across deployments with **different API keys** (e.g., different Azure regions or OpenAI organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the API key that created them.
|
||||
|
||||
### The Problem
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.",
|
||||
"type": "invalid_request_error",
|
||||
"code": "invalid_encrypted_content"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This error occurs when:
|
||||
1. Initial request goes to Deployment A (API Key 1) → produces encrypted item `rs_xyz`
|
||||
2. Follow-up request with `rs_xyz` in input gets load balanced to Deployment B (API Key 2)
|
||||
3. Deployment B cannot decrypt content created by Deployment A → **request fails**
|
||||
|
||||
### The Solution: `encrypted_content_affinity`
|
||||
|
||||
The `encrypted_content_affinity` pre-call check intelligently tracks encrypted content and routes follow-up requests to the originating deployment **only when necessary**.
|
||||
|
||||
**Key Benefits:**
|
||||
- ✅ **No quota reduction**: Unlike `deployment_affinity`, only pins requests that contain tracked encrypted items
|
||||
- ✅ **Bypasses rate limits**: When encrypted content requires a specific deployment, RPM/TPM limits are bypassed (the request would fail on any other deployment anyway)
|
||||
- ✅ **No `previous_response_id` required**: Works by tracking item IDs in response output and matching them in request input
|
||||
- ✅ **Globally safe**: Can be enabled for all models; non-Responses-API calls (chat, embeddings) are unaffected
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Tracking Phase** (after successful response):
|
||||
- Extracts all item IDs from response `output` (e.g., `msg_abc`, `rs_xyz`)
|
||||
- Caches mapping: `item_id` → `deployment_id` (default TTL: 24 hours)
|
||||
|
||||
2. **Routing Phase** (before request):
|
||||
- Scans request `input` for item IDs
|
||||
- If tracked item found → pins to originating deployment, bypasses rate limits
|
||||
- If no tracked items → normal load balancing
|
||||
|
||||
### Configuration
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="Python SDK">
|
||||
|
||||
```python
|
||||
from litellm import Router
|
||||
|
||||
router = Router(
|
||||
model_list=[
|
||||
{
|
||||
"model_name": "gpt-5.1-codex",
|
||||
"litellm_params": {
|
||||
"model": "openai/gpt-5.1-codex",
|
||||
"api_key": "org-1-api-key", # Different API key
|
||||
},
|
||||
"model_info": {"id": "deployment-us-east"},
|
||||
},
|
||||
{
|
||||
"model_name": "gpt-5.1-codex",
|
||||
"litellm_params": {
|
||||
"model": "openai/gpt-5.1-codex",
|
||||
"api_key": "org-2-api-key", # Different API key
|
||||
},
|
||||
"model_info": {"id": "deployment-eu-west"},
|
||||
},
|
||||
],
|
||||
optional_pre_call_checks=["encrypted_content_affinity"],
|
||||
deployment_affinity_ttl_seconds=86400, # 24 hours (default)
|
||||
)
|
||||
|
||||
# Initial request - routes to any deployment
|
||||
response1 = await router.aresponses(
|
||||
model="gpt-5.1-codex",
|
||||
input="Explain quantum computing",
|
||||
)
|
||||
|
||||
# Follow-up with encrypted items - automatically routes to same deployment
|
||||
response2 = await router.aresponses(
|
||||
model="gpt-5.1-codex",
|
||||
input=response1.output, # Contains encrypted items from response1
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="proxy" label="Proxy Server">
|
||||
|
||||
```yaml showLineNumbers title="config.yaml"
|
||||
model_list:
|
||||
- model_name: gpt-5.1-codex
|
||||
litellm_params:
|
||||
model: azure/gpt-5.1-codex
|
||||
api_base: https://eastus.openai.azure.com/
|
||||
api_key: os.environ/AZURE_API_KEY_EASTUS
|
||||
rpm: 600
|
||||
tpm: 100000
|
||||
model_info:
|
||||
id: "gpt-5.1-codex-eastus"
|
||||
|
||||
- model_name: gpt-5.1-codex
|
||||
litellm_params:
|
||||
model: azure/gpt-5.1-codex
|
||||
api_base: https://westeurope.openai.azure.com/
|
||||
api_key: os.environ/AZURE_API_KEY_WESTEUROPE
|
||||
rpm: 600
|
||||
tpm: 100000
|
||||
model_info:
|
||||
id: "gpt-5.1-codex-westeurope"
|
||||
|
||||
router_settings:
|
||||
routing_strategy: usage-based-routing-v2
|
||||
enable_pre_call_checks: true
|
||||
optional_pre_call_checks:
|
||||
- encrypted_content_affinity
|
||||
deployment_affinity_ttl_seconds: 86400 # Optional, default is 86400 (24 hours)
|
||||
```
|
||||
|
||||
**Start proxy:**
|
||||
```bash
|
||||
litellm --config config.yaml
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### When to Use Each Affinity Type
|
||||
|
||||
| Affinity Type | Use Case | Scope | Quota Impact |
|
||||
|---------------|----------|-------|--------------|
|
||||
| **`encrypted_content_affinity`** | **[Recommended]** Multi-region Responses API with different API keys | Only requests with tracked encrypted items | ✅ None (surgical pinning) |
|
||||
| `responses_api_deployment_check` | When `previous_response_id` is available | Requests with `previous_response_id` | ✅ None |
|
||||
| `session_affinity` | Session-based applications | All requests with same `session_id` | ⚠️ Reduces quota by # of sessions |
|
||||
| `deployment_affinity` | Simple sticky sessions | All requests from same API key | ❌ Reduces quota by # of users |
|
||||
|
||||
### Multi-Instance Deployment (Redis)
|
||||
|
||||
For multiple LiteLLM proxy instances, use Redis to share affinity state:
|
||||
|
||||
```yaml
|
||||
router_settings:
|
||||
optional_pre_call_checks:
|
||||
- encrypted_content_affinity
|
||||
redis_host: redis.example.com
|
||||
redis_port: 6379
|
||||
redis_password: your-password
|
||||
```
|
||||
|
||||
|
||||
## Calling non-Responses API endpoints (`/responses` to `/chat/completions` Bridge)
|
||||
|
||||
LiteLLM allows you to call non-Responses API models via a bridge to LiteLLM's `/chat/completions` endpoint. This is useful for calling Anthropic, Gemini and even non-Responses API OpenAI models.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue