From 6935d446cc8f8ee2168778cea3aab9aa5b69c294 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Thu, 9 Apr 2026 14:27:42 +0000 Subject: [PATCH] docs: add prompt caching based routing to routing docs - Add 'Prompt Caching Based Routing' section to routing.md with SDK and Proxy examples, including Bedrock multi-account config - Add prompt caching routing section to proxy/load_balancing.md - Add prompt cache routing tutorial to Routing & Load Balancing sidebar - Add sidebar_label frontmatter to the Claude Code tutorial for cleaner display in the routing nav section Co-authored-by: Krrish Dholakia --- docs/my-website/docs/proxy/load_balancing.md | 31 +++++ docs/my-website/docs/routing.md | 117 ++++++++++++++++++ .../claude_code_prompt_cache_routing.md | 4 + docs/my-website/sidebars.js | 1 + 4 files changed, 153 insertions(+) diff --git a/docs/my-website/docs/proxy/load_balancing.md b/docs/my-website/docs/proxy/load_balancing.md index 93f3d944340..1def5e95319 100644 --- a/docs/my-website/docs/proxy/load_balancing.md +++ b/docs/my-website/docs/proxy/load_balancing.md @@ -134,6 +134,37 @@ router_settings: Detailed information about [routing strategies can be found here](../routing) ::: +## Prompt Caching Based Routing + +Route requests to deployments where a prompt cache already exists. When load balancing across multiple deployments of models that support [prompt caching](../completion/prompt_caching.md) (e.g., Anthropic, Bedrock with Claude), this ensures subsequent requests reuse cached prompts on the same deployment — maximizing cache hits and reducing costs. + +This is especially useful with tools like **Claude Code** that rely on prompt caching for performance. + +```yaml +model_list: + - model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-20250514 + api_key: os.environ/ANTHROPIC_API_KEY_1 + - model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-20250514 + api_key: os.environ/ANTHROPIC_API_KEY_2 + +router_settings: + optional_pre_call_checks: ["prompt_caching"] # 👈 Enable prompt caching routing +``` + +**How it works:** +1. On a successful completion, LiteLLM checks if the prompt is eligible for caching. +2. If eligible, it stores a mapping of the prompt hash → deployment `model_id`. +3. On subsequent requests with the same prompt prefix, LiteLLM routes to the same deployment. +4. If no cached mapping exists, standard routing strategies apply. + +:::tip +For a detailed setup guide with Claude Code, see [Claude Code - Prompt Cache Routing](../tutorials/claude_code_prompt_cache_routing.md). For full documentation on prompt caching routing (including SDK usage), see [Routing - Prompt Caching](../routing#prompt-caching-based-routing). +::: + #### Step 2: Start Proxy with config ```shell diff --git a/docs/my-website/docs/routing.md b/docs/my-website/docs/routing.md index 5aa655ae212..a728e6725f1 100644 --- a/docs/my-website/docs/routing.md +++ b/docs/my-website/docs/routing.md @@ -830,6 +830,123 @@ asyncio.run(router_acompletion()) +## Prompt Caching Based Routing + +Route requests to deployments where a prompt cache already exists. This is useful when using providers that support [prompt caching](./completion/prompt_caching.md) (e.g., Anthropic, Bedrock with Claude), especially with tools like Claude Code that rely on prompt caching for performance. + +When enabled, LiteLLM remembers which deployment handled a request that resulted in a prompt cache write. Subsequent requests with the same prompt prefix are routed to that same deployment, maximizing cache hits and reducing costs/latency. + +**How it works:** +1. On a successful completion call, LiteLLM checks if the prompt is eligible for caching (e.g., prompt > 1024 tokens for Anthropic). +2. If eligible, LiteLLM stores a mapping of the prompt hash → deployment `model_id`. +3. On subsequent requests, LiteLLM checks if the prompt matches a cached mapping and routes to the same deployment. +4. If no cached mapping exists, standard routing strategies apply. + + + + +**1. Set `optional_pre_call_checks` in config** + +```yaml +model_list: + - model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-20250514 + api_key: os.environ/ANTHROPIC_API_KEY_1 + - model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-20250514 + api_key: os.environ/ANTHROPIC_API_KEY_2 + +router_settings: + optional_pre_call_checks: ["prompt_caching"] # 👈 Enable prompt caching routing +``` + +**2. Start proxy** + +```bash +litellm --config /path/to/config.yaml +``` + +**3. Test it!** + +```bash +curl --location 'http://localhost:4000/v1/chat/completions' \ +--header 'Content-Type: application/json' \ +--header 'Authorization: Bearer sk-1234' \ +--data '{ + "model": "claude-sonnet", + "messages": [{"role": "user", "content": "Hey, how is it going?"}] +}' +``` + + + + +```python +from litellm import Router + +model_list = [ + { + "model_name": "claude-sonnet", + "litellm_params": { + "model": "anthropic/claude-sonnet-4-20250514", + "api_key": os.getenv("ANTHROPIC_API_KEY_1"), + }, + }, + { + "model_name": "claude-sonnet", + "litellm_params": { + "model": "anthropic/claude-sonnet-4-20250514", + "api_key": os.getenv("ANTHROPIC_API_KEY_2"), + }, + }, +] + +router = Router( + model_list=model_list, + optional_pre_call_checks=["prompt_caching"], # 👈 Enable prompt caching routing +) + +response = await router.acompletion( + model="claude-sonnet", + messages=[{"role": "user", "content": "Hey, how is it going?"}], +) +print(response) +``` + + + + +### Using with AWS Bedrock (Multiple Accounts) + +Prompt caching routing is especially useful with Bedrock when load balancing across multiple AWS accounts: + +```yaml +router_settings: + optional_pre_call_checks: ["prompt_caching"] + +model_list: + - model_name: claude-sonnet + litellm_params: + model: us.anthropic.claude-sonnet-4-5-20250929-v1:0 + aws_profile_name: account-1 + aws_region_name: us-west-2 + model_info: + litellm_provider: bedrock + - model_name: claude-sonnet + litellm_params: + model: us.anthropic.claude-sonnet-4-5-20250929-v1:0 + aws_profile_name: account-2 + aws_region_name: us-west-2 + model_info: + litellm_provider: bedrock +``` + +:::info +For a detailed setup guide with Claude Code, see [Claude Code - Prompt Cache Routing](./tutorials/claude_code_prompt_cache_routing.md). +::: + ## Traffic Mirroring / Silent Experiments Traffic mirroring allows you to "mimic" production traffic to a secondary (silent) model for evaluation purposes. The silent model's response is gathered in the background and does not affect the latency or result of the primary request. diff --git a/docs/my-website/docs/tutorials/claude_code_prompt_cache_routing.md b/docs/my-website/docs/tutorials/claude_code_prompt_cache_routing.md index bbb29489856..a25765f25ad 100644 --- a/docs/my-website/docs/tutorials/claude_code_prompt_cache_routing.md +++ b/docs/my-website/docs/tutorials/claude_code_prompt_cache_routing.md @@ -1,3 +1,7 @@ +--- +sidebar_label: Prompt Cache Routing +--- + # Claude Code - Prompt Cache Routing Claude's [Prompt Caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) feature helps to optimize API usage through attempting to cache prompts and re-use cached prompts during subsequent API calls. This feature is used by Claude Code. diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index ab8f257c7d5..bb83ee7d5da 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -1046,6 +1046,7 @@ const sidebars = { "scheduler", "proxy/auto_routing", "proxy/load_balancing", + "tutorials/claude_code_prompt_cache_routing", "proxy/keys_teams_router_settings", "proxy/provider_budget_routing", "proxy/reliability",