docs: add prompt caching based routing to routing docs

- Add 'Prompt Caching Based Routing' section to routing.md with SDK and
  Proxy examples, including Bedrock multi-account config
- Add prompt caching routing section to proxy/load_balancing.md
- Add prompt cache routing tutorial to Routing & Load Balancing sidebar
- Add sidebar_label frontmatter to the Claude Code tutorial for cleaner
  display in the routing nav section

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
This commit is contained in:
Cursor Agent 2026-04-09 14:27:42 +00:00
parent 97f722f558
commit 6935d446cc
No known key found for this signature in database
4 changed files with 153 additions and 0 deletions

View file

@ -134,6 +134,37 @@ router_settings:
Detailed information about [routing strategies can be found here](../routing)
:::
## Prompt Caching Based Routing
Route requests to deployments where a prompt cache already exists. When load balancing across multiple deployments of models that support [prompt caching](../completion/prompt_caching.md) (e.g., Anthropic, Bedrock with Claude), this ensures subsequent requests reuse cached prompts on the same deployment — maximizing cache hits and reducing costs.
This is especially useful with tools like **Claude Code** that rely on prompt caching for performance.
```yaml
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY_1
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY_2
router_settings:
optional_pre_call_checks: ["prompt_caching"] # 👈 Enable prompt caching routing
```
**How it works:**
1. On a successful completion, LiteLLM checks if the prompt is eligible for caching.
2. If eligible, it stores a mapping of the prompt hash → deployment `model_id`.
3. On subsequent requests with the same prompt prefix, LiteLLM routes to the same deployment.
4. If no cached mapping exists, standard routing strategies apply.
:::tip
For a detailed setup guide with Claude Code, see [Claude Code - Prompt Cache Routing](../tutorials/claude_code_prompt_cache_routing.md). For full documentation on prompt caching routing (including SDK usage), see [Routing - Prompt Caching](../routing#prompt-caching-based-routing).
:::
#### Step 2: Start Proxy with config
```shell

View file

@ -830,6 +830,123 @@ asyncio.run(router_acompletion())
</TabItem>
</Tabs>
## Prompt Caching Based Routing
Route requests to deployments where a prompt cache already exists. This is useful when using providers that support [prompt caching](./completion/prompt_caching.md) (e.g., Anthropic, Bedrock with Claude), especially with tools like Claude Code that rely on prompt caching for performance.
When enabled, LiteLLM remembers which deployment handled a request that resulted in a prompt cache write. Subsequent requests with the same prompt prefix are routed to that same deployment, maximizing cache hits and reducing costs/latency.
**How it works:**
1. On a successful completion call, LiteLLM checks if the prompt is eligible for caching (e.g., prompt > 1024 tokens for Anthropic).
2. If eligible, LiteLLM stores a mapping of the prompt hash → deployment `model_id`.
3. On subsequent requests, LiteLLM checks if the prompt matches a cached mapping and routes to the same deployment.
4. If no cached mapping exists, standard routing strategies apply.
<Tabs>
<TabItem value="proxy" label="Proxy">
**1. Set `optional_pre_call_checks` in config**
```yaml
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY_1
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY_2
router_settings:
optional_pre_call_checks: ["prompt_caching"] # 👈 Enable prompt caching routing
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
```
**3. Test it!**
```bash
curl --location 'http://localhost:4000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
"model": "claude-sonnet",
"messages": [{"role": "user", "content": "Hey, how is it going?"}]
}'
```
</TabItem>
<TabItem value="sdk" label="SDK">
```python
from litellm import Router
model_list = [
{
"model_name": "claude-sonnet",
"litellm_params": {
"model": "anthropic/claude-sonnet-4-20250514",
"api_key": os.getenv("ANTHROPIC_API_KEY_1"),
},
},
{
"model_name": "claude-sonnet",
"litellm_params": {
"model": "anthropic/claude-sonnet-4-20250514",
"api_key": os.getenv("ANTHROPIC_API_KEY_2"),
},
},
]
router = Router(
model_list=model_list,
optional_pre_call_checks=["prompt_caching"], # 👈 Enable prompt caching routing
)
response = await router.acompletion(
model="claude-sonnet",
messages=[{"role": "user", "content": "Hey, how is it going?"}],
)
print(response)
```
</TabItem>
</Tabs>
### Using with AWS Bedrock (Multiple Accounts)
Prompt caching routing is especially useful with Bedrock when load balancing across multiple AWS accounts:
```yaml
router_settings:
optional_pre_call_checks: ["prompt_caching"]
model_list:
- model_name: claude-sonnet
litellm_params:
model: us.anthropic.claude-sonnet-4-5-20250929-v1:0
aws_profile_name: account-1
aws_region_name: us-west-2
model_info:
litellm_provider: bedrock
- model_name: claude-sonnet
litellm_params:
model: us.anthropic.claude-sonnet-4-5-20250929-v1:0
aws_profile_name: account-2
aws_region_name: us-west-2
model_info:
litellm_provider: bedrock
```
:::info
For a detailed setup guide with Claude Code, see [Claude Code - Prompt Cache Routing](./tutorials/claude_code_prompt_cache_routing.md).
:::
## Traffic Mirroring / Silent Experiments
Traffic mirroring allows you to "mimic" production traffic to a secondary (silent) model for evaluation purposes. The silent model's response is gathered in the background and does not affect the latency or result of the primary request.

View file

@ -1,3 +1,7 @@
---
sidebar_label: Prompt Cache Routing
---
# Claude Code - Prompt Cache Routing
Claude's [Prompt Caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) feature helps to optimize API usage through attempting to cache prompts and re-use cached prompts during subsequent API calls. This feature is used by Claude Code.

View file

@ -1046,6 +1046,7 @@ const sidebars = {
"scheduler",
"proxy/auto_routing",
"proxy/load_balancing",
"tutorials/claude_code_prompt_cache_routing",
"proxy/keys_teams_router_settings",
"proxy/provider_budget_routing",
"proxy/reliability",