diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index 8dbebad884e..095d1acc035 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -359,7 +359,7 @@ router_settings: | redis_url | str | URL for Redis server. **Known performance issue with Redis URL.** | | cache_responses | boolean | Flag to enable caching LLM Responses, if cache set under `router_settings`. If true, caches responses. Defaults to False. | | router_general_settings | RouterGeneralSettings | [SDK-Only] Router general settings - contains optimizations like 'async_only_mode'. [Docs](../routing.md#router-general-settings) | -| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `deployment_affinity`, `forward_client_headers_by_model_group` | +| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `encrypted_content_affinity`, `deployment_affinity`, `session_affinity`, `forward_client_headers_by_model_group` | | deployment_affinity_ttl_seconds | int | TTL (seconds) for user-key → deployment affinity mapping when `deployment_affinity` is enabled (configured at Router init / proxy startup). Defaults to `3600` (1 hour). | | ignore_invalid_deployments | boolean | If true, ignores invalid deployments. Default for proxy is True - to prevent invalid models from blocking other models from being loaded. | | search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search.md) | diff --git a/docs/my-website/docs/proxy/load_balancing.md b/docs/my-website/docs/proxy/load_balancing.md index 186307d6498..5bf39d179f6 100644 --- a/docs/my-website/docs/proxy/load_balancing.md +++ b/docs/my-website/docs/proxy/load_balancing.md @@ -347,3 +347,36 @@ If `order=1` deployment is unavailable (e.g., rate-limited), the router falls ba - **Higher throughput**: More requests handled simultaneously across deployments - **Improved reliability**: If one deployment fails, traffic automatically routes to healthy ones - **Better resource utilization**: Load spread evenly across all available deployments + +## Special Considerations for Responses API + +When load balancing OpenAI's Responses API across deployments with **different API keys** (e.g., different Azure regions or organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the originating API key. + +**Solution:** Use the `encrypted_content_affinity` pre-call check to automatically route follow-up requests containing encrypted items to the correct deployment: + +```yaml +model_list: + - model_name: gpt-5.1-codex + litellm_params: + model: azure/gpt-5.1-codex + api_base: https://eastus.openai.azure.com/ + api_key: os.environ/AZURE_API_KEY_EASTUS + model_info: + id: "deployment-eastus" + + - model_name: gpt-5.1-codex + litellm_params: + model: azure/gpt-5.1-codex + api_base: https://westeurope.openai.azure.com/ + api_key: os.environ/AZURE_API_KEY_WESTEUROPE + model_info: + id: "deployment-westeurope" + +router_settings: + optional_pre_call_checks: + - encrypted_content_affinity # 👈 Prevents invalid_encrypted_content errors +``` + +This ensures requests containing encrypted content are routed to the deployment that created them, while other requests continue to load balance normally. + +**[Learn more about Encrypted Content Affinity →](../response_api.md#encrypted-content-affinity-multi-region-load-balancing)** diff --git a/docs/my-website/docs/response_api.md b/docs/my-website/docs/response_api.md index b37be2b5bc2..85144a80245 100644 --- a/docs/my-website/docs/response_api.md +++ b/docs/my-website/docs/response_api.md @@ -920,9 +920,14 @@ follow_up = await router.aresponses( To enable session continuity for Responses API in your LiteLLM proxy, set `optional_pre_call_checks` in your proxy config.yaml. - `responses_api_deployment_check`: high priority routing when `previous_response_id` is provided +- `encrypted_content_affinity`: **[Recommended]** content-aware routing for encrypted items (e.g., `rs_...` reasoning items) - `session_affinity`: sticky sessions based on session id (takes priority over `deployment_affinity`) - `deployment_affinity`: sticky sessions based on user key (applies even without `previous_response_id`) +:::tip Recommended: Use `encrypted_content_affinity` +For Responses API with load balancing across deployments with **different API keys**, use `encrypted_content_affinity` instead of `deployment_affinity`. It only pins requests that contain encrypted content, avoiding quota reduction while preventing `invalid_encrypted_content` errors. +::: + Notes: - User-key affinity is keyed on `metadata.user_api_key_hash` (the API key hash). The OpenAI `user` request parameter is an end-user identifier and is intentionally not used for deployment affinity. - Session-ID affinity is keyed on `metadata.session_id`. For proxy requests, this can be passed via the `x-litellm-session-id` HTTP header. For Python SDK requests, you can pass it via `litellm_metadata={"session_id": "value"}` in request args. @@ -983,6 +988,156 @@ follow_up = client.responses.create( +## Encrypted Content Affinity (Multi-Region Load Balancing) + +When load balancing Responses API across deployments with **different API keys** (e.g., different Azure regions or OpenAI organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the API key that created them. + +### The Problem + +```json +{ + "error": { + "message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.", + "type": "invalid_request_error", + "code": "invalid_encrypted_content" + } +} +``` + +This error occurs when: +1. Initial request goes to Deployment A (API Key 1) → produces encrypted item `rs_xyz` +2. Follow-up request with `rs_xyz` in input gets load balanced to Deployment B (API Key 2) +3. Deployment B cannot decrypt content created by Deployment A → **request fails** + +### The Solution: `encrypted_content_affinity` + +The `encrypted_content_affinity` pre-call check intelligently tracks encrypted content and routes follow-up requests to the originating deployment **only when necessary**. + +**Key Benefits:** +- ✅ **No quota reduction**: Unlike `deployment_affinity`, only pins requests that contain tracked encrypted items +- ✅ **Bypasses rate limits**: When encrypted content requires a specific deployment, RPM/TPM limits are bypassed (the request would fail on any other deployment anyway) +- ✅ **No `previous_response_id` required**: Works by tracking item IDs in response output and matching them in request input +- ✅ **Globally safe**: Can be enabled for all models; non-Responses-API calls (chat, embeddings) are unaffected + +### How It Works + +1. **Tracking Phase** (after successful response): + - Extracts all item IDs from response `output` (e.g., `msg_abc`, `rs_xyz`) + - Caches mapping: `item_id` → `deployment_id` (default TTL: 24 hours) + +2. **Routing Phase** (before request): + - Scans request `input` for item IDs + - If tracked item found → pins to originating deployment, bypasses rate limits + - If no tracked items → normal load balancing + +### Configuration + + + + +```python +from litellm import Router + +router = Router( + model_list=[ + { + "model_name": "gpt-5.1-codex", + "litellm_params": { + "model": "openai/gpt-5.1-codex", + "api_key": "org-1-api-key", # Different API key + }, + "model_info": {"id": "deployment-us-east"}, + }, + { + "model_name": "gpt-5.1-codex", + "litellm_params": { + "model": "openai/gpt-5.1-codex", + "api_key": "org-2-api-key", # Different API key + }, + "model_info": {"id": "deployment-eu-west"}, + }, + ], + optional_pre_call_checks=["encrypted_content_affinity"], + deployment_affinity_ttl_seconds=86400, # 24 hours (default) +) + +# Initial request - routes to any deployment +response1 = await router.aresponses( + model="gpt-5.1-codex", + input="Explain quantum computing", +) + +# Follow-up with encrypted items - automatically routes to same deployment +response2 = await router.aresponses( + model="gpt-5.1-codex", + input=response1.output, # Contains encrypted items from response1 +) +``` + + + + +```yaml showLineNumbers title="config.yaml" +model_list: + - model_name: gpt-5.1-codex + litellm_params: + model: azure/gpt-5.1-codex + api_base: https://eastus.openai.azure.com/ + api_key: os.environ/AZURE_API_KEY_EASTUS + rpm: 600 + tpm: 100000 + model_info: + id: "gpt-5.1-codex-eastus" + + - model_name: gpt-5.1-codex + litellm_params: + model: azure/gpt-5.1-codex + api_base: https://westeurope.openai.azure.com/ + api_key: os.environ/AZURE_API_KEY_WESTEUROPE + rpm: 600 + tpm: 100000 + model_info: + id: "gpt-5.1-codex-westeurope" + +router_settings: + routing_strategy: usage-based-routing-v2 + enable_pre_call_checks: true + optional_pre_call_checks: + - encrypted_content_affinity + deployment_affinity_ttl_seconds: 86400 # Optional, default is 86400 (24 hours) +``` + +**Start proxy:** +```bash +litellm --config config.yaml +``` + + + + +### When to Use Each Affinity Type + +| Affinity Type | Use Case | Scope | Quota Impact | +|---------------|----------|-------|--------------| +| **`encrypted_content_affinity`** | **[Recommended]** Multi-region Responses API with different API keys | Only requests with tracked encrypted items | ✅ None (surgical pinning) | +| `responses_api_deployment_check` | When `previous_response_id` is available | Requests with `previous_response_id` | ✅ None | +| `session_affinity` | Session-based applications | All requests with same `session_id` | ⚠️ Reduces quota by # of sessions | +| `deployment_affinity` | Simple sticky sessions | All requests from same API key | ❌ Reduces quota by # of users | + +### Multi-Instance Deployment (Redis) + +For multiple LiteLLM proxy instances, use Redis to share affinity state: + +```yaml +router_settings: + optional_pre_call_checks: + - encrypted_content_affinity + redis_host: redis.example.com + redis_port: 6379 + redis_password: your-password +``` + + ## Calling non-Responses API endpoints (`/responses` to `/chat/completions` Bridge) LiteLLM allows you to call non-Responses API models via a bridge to LiteLLM's `/chat/completions` endpoint. This is useful for calling Anthropic, Gemini and even non-Responses API OpenAI models.