mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-11 22:51:28 +00:00
Update documentation for azure 4 feats
This commit is contained in:
Sameer Kankute
parent
dba9946b98
commit
c7ef668d78
6 changed files with 67 additions and 318 deletions
|
|
@ -897,14 +897,13 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
|
||||
## Effort Parameter: Control Token Usage {#effort-parameter}
|
||||
|
||||
Controls aspects like how much effort the model puts into its response, via `output_config={"effort": ..}`.
|
||||
Control how much effort Claude puts into its response using the `reasoning_effort` parameter. This allows you to trade off between response thoroughness and token efficiency.
|
||||
|
||||
:::info
|
||||
|
||||
Soon, we will map OpenAI's `reasoning_effort` parameter to this.
|
||||
LiteLLM automatically maps `reasoning_effort` to Anthropic's `output_config` format and adds the required `effort-2025-11-24` beta header for Claude Opus 4.5.
|
||||
:::
|
||||
|
||||
Potential Values for `effort` parameter: `"high"`, `"medium"`, `"low"`.
|
||||
Potential values for `reasoning_effort` parameter: `"high"`, `"medium"`, `"low"`.
|
||||
|
||||
### Usage Example
|
||||
|
||||
|
|
@ -920,7 +919,7 @@ message = "Analyze the trade-offs between microservices and monolithic architect
|
|||
response_high = litellm.completion(
|
||||
model="anthropic/claude-opus-4-5-20251101",
|
||||
messages=[{"role": "user", "content": message}],
|
||||
output_config={"effort": "high"}
|
||||
reasoning_effort="high"
|
||||
)
|
||||
|
||||
print("High effort response:")
|
||||
|
|
@ -931,7 +930,7 @@ print(f"Tokens used: {response_high.usage.completion_tokens}\n")
|
|||
response_medium = litellm.completion(
|
||||
model="anthropic/claude-opus-4-5-20251101",
|
||||
messages=[{"role": "user", "content": message}],
|
||||
output_config={"effort": "medium"}
|
||||
reasoning_effort="medium"
|
||||
)
|
||||
|
||||
print("Medium effort response:")
|
||||
|
|
@ -942,7 +941,7 @@ print(f"Tokens used: {response_medium.usage.completion_tokens}\n")
|
|||
response_low = litellm.completion(
|
||||
model="anthropic/claude-opus-4-5-20251101",
|
||||
messages=[{"role": "user", "content": message}],
|
||||
output_config={"effort": "low"}
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
print("Low effort response:")
|
||||
|
|
@ -987,295 +986,9 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
"role": "user",
|
||||
"content": "Analyze the trade-offs between microservices and monolithic architectures"
|
||||
}],
|
||||
"output_config": {
|
||||
"effort": "high"
|
||||
}
|
||||
"reasoning_effort": "high"
|
||||
}
|
||||
'
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Cost Tracking: Monitor Tool Search Usage {#cost-tracking}
|
||||
|
||||
### Understanding Tool Search Costs
|
||||
|
||||
Tool search operations are tracked separately in the usage object, allowing you to monitor and optimize costs.
|
||||
|
||||
It is available in the `usage` object, under `server_tool_use.tool_search_requests`.
|
||||
|
||||
Anthropic charges $0.0001 per tool search request.
|
||||
|
||||
### Tracking Example
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="LiteLLM Python SDK">
|
||||
|
||||
```python
|
||||
import litellm
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "tool_search_tool_regex_20251119",
|
||||
"name": "tool_search_tool_regex"
|
||||
},
|
||||
# ... 100 deferred tools
|
||||
]
|
||||
|
||||
response = litellm.completion(
|
||||
model="anthropic/claude-sonnet-4-5-20250929",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Find and use the weather tool for San Francisco"
|
||||
}],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
# Standard token usage
|
||||
print("Token Usage:")
|
||||
print(f" Input tokens: {response.usage.prompt_tokens}")
|
||||
print(f" Output tokens: {response.usage.completion_tokens}")
|
||||
print(f" Total tokens: {response.usage.total_tokens}")
|
||||
|
||||
# Tool search specific usage
|
||||
if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use:
|
||||
print(f"\nTool Search Usage:")
|
||||
print(f" Search requests: {response.usage.server_tool_use.tool_search_requests}")
|
||||
|
||||
# Calculate cost (example pricing)
|
||||
input_cost = response.usage.prompt_tokens * 0.000003 # $3 per 1M tokens
|
||||
output_cost = response.usage.completion_tokens * 0.000015 # $15 per 1M tokens
|
||||
search_cost = response.usage.server_tool_use.tool_search_requests * 0.0001 # Example
|
||||
|
||||
total_cost = input_cost + output_cost + search_cost
|
||||
|
||||
print(f"\nCost Breakdown:")
|
||||
print(f" Input tokens: ${input_cost:.6f}")
|
||||
print(f" Output tokens: ${output_cost:.6f}")
|
||||
print(f" Tool searches: ${search_cost:.6f}")
|
||||
print(f" Total: ${total_cost:.6f}")
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="proxy" label="LiteLLM Proxy">
|
||||
|
||||
1. Setup config.yaml
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: claude-4
|
||||
litellm_params:
|
||||
model: anthropic/claude-opus-4-5-20251101
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
```
|
||||
|
||||
2. Start the proxy
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
3. Test it!
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'Authorization: Bearer $LITELLM_KEY' \
|
||||
--data ' {
|
||||
"model": "claude-4",
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": "Find and use the weather tool for San Francisco"
|
||||
}],
|
||||
"tools": [
|
||||
{
|
||||
"type": "tool_search_tool_regex_20251119",
|
||||
"name": "tool_search_tool_regex"
|
||||
},
|
||||
# ... 100 deferred tools
|
||||
]
|
||||
}
|
||||
'
|
||||
```
|
||||
|
||||
Expected Response:
|
||||
|
||||
```json
|
||||
{
|
||||
...,
|
||||
"usage": {
|
||||
...,
|
||||
"server_tool_use": {
|
||||
"tool_search_requests": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Cost Optimization Tips
|
||||
|
||||
1. **Keep frequently used tools non-deferred** (3-5 tools)
|
||||
2. **Use tool search for large catalogs** (10+ tools)
|
||||
3. **Monitor search requests** to identify optimization opportunities
|
||||
4. **Combine with effort parameter** for maximum efficiency
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Combining Features {#combining-features}
|
||||
|
||||
### The Power of Integration
|
||||
|
||||
These features work together seamlessly. Here's a real-world example combining all of them:
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="LiteLLM Python SDK">
|
||||
|
||||
```python
|
||||
import litellm
|
||||
import json
|
||||
|
||||
# Large tool catalog with search, programmatic calling, and examples
|
||||
tools = [
|
||||
# Enable tool search
|
||||
{
|
||||
"type": "tool_search_tool_regex_20251119",
|
||||
"name": "tool_search_tool_regex"
|
||||
},
|
||||
# Enable programmatic calling
|
||||
{
|
||||
"type": "code_execution_20250825",
|
||||
"name": "code_execution"
|
||||
},
|
||||
# Database tool with all features
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "query_database",
|
||||
"description": "Execute SQL queries against the analytics database. Returns JSON array of results.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"sql": {
|
||||
"type": "string",
|
||||
"description": "SQL SELECT statement"
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": "Maximum rows to return"
|
||||
}
|
||||
},
|
||||
"required": ["sql"]
|
||||
}
|
||||
},
|
||||
"defer_loading": True, # Tool search
|
||||
"allowed_callers": ["code_execution_20250825"], # Programmatic calling
|
||||
"input_examples": [ # Input examples
|
||||
{
|
||||
"sql": "SELECT region, SUM(revenue) as total FROM sales GROUP BY region",
|
||||
"limit": 100
|
||||
}
|
||||
]
|
||||
},
|
||||
# ... 50 more tools with defer_loading
|
||||
]
|
||||
|
||||
# Make request with effort control
|
||||
response = litellm.completion(
|
||||
model="anthropic/claude-opus-4-5-20251101",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Analyze sales by region for the last quarter and identify top performers"
|
||||
}],
|
||||
tools=tools,
|
||||
output_config={"effort": "medium"} # Balanced efficiency
|
||||
)
|
||||
|
||||
# Track comprehensive usage
|
||||
print("Complete Usage Metrics:")
|
||||
print(f" Input tokens: {response.usage.prompt_tokens}")
|
||||
print(f" Output tokens: {response.usage.completion_tokens}")
|
||||
print(f" Total tokens: {response.usage.total_tokens}")
|
||||
|
||||
if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use:
|
||||
print(f" Tool searches: {response.usage.server_tool_use.tool_search_requests}")
|
||||
|
||||
print(f"\nResponse: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="proxy" label="LiteLLM Proxy">
|
||||
|
||||
1. Setup config.yaml
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: claude-4
|
||||
litellm_params:
|
||||
model: anthropic/claude-opus-4-5-20251101
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
```
|
||||
|
||||
2. Start the proxy
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
3. Test it!
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'Authorization: Bearer $LITELLM_KEY' \
|
||||
--data ' {
|
||||
"model": "claude-4",
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": "Analyze sales by region for the last quarter and identify top performers"
|
||||
}],
|
||||
"tools": [
|
||||
{
|
||||
"type": "tool_search_tool_regex_20251119",
|
||||
"name": "tool_search_tool_regex"
|
||||
},
|
||||
# ... 100 deferred tools
|
||||
],
|
||||
"output_config": {
|
||||
"effort": "medium"
|
||||
}
|
||||
}
|
||||
'
|
||||
```
|
||||
|
||||
Expected Response:
|
||||
|
||||
```json
|
||||
{
|
||||
...,
|
||||
"usage": {
|
||||
...,
|
||||
"server_tool_use": {
|
||||
"tool_search_requests": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Real-World Benefits
|
||||
|
||||
This combination enables:
|
||||
|
||||
1. **Massive scale** - Handle 1000+ tools efficiently
|
||||
2. **Low latency** - Programmatic calling reduces round trips
|
||||
3. **High accuracy** - Input examples ensure correct tool usage
|
||||
4. **Cost control** - Effort parameter optimizes token spend
|
||||
5. **Full visibility** - Track all usage metrics
|
||||
|
||||
|
|
|
|||
|
|
@ -41,7 +41,8 @@ Check this in code, [here](../completion/input.md#translated-openai-params)
|
|||
"extra_headers",
|
||||
"parallel_tool_calls",
|
||||
"response_format",
|
||||
"user"
|
||||
"user",
|
||||
"reasoning_effort",
|
||||
```
|
||||
|
||||
:::info
|
||||
|
|
@ -49,6 +50,7 @@ Check this in code, [here](../completion/input.md#translated-openai-params)
|
|||
**Notes:**
|
||||
- Anthropic API fails requests when `max_tokens` are not passed. Due to this litellm passes `max_tokens=4096` when no `max_tokens` are passed.
|
||||
- `response_format` is fully supported for Claude Sonnet 4.5 and Opus 4.1 models (see [Structured Outputs](#structured-outputs) section)
|
||||
- `reasoning_effort` is automatically mapped to `output_config={"effort": ...}` for Claude Opus 4.5 models (see [Effort Parameter](./anthropic_effort.md))
|
||||
|
||||
:::
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,10 @@ Control how many tokens Claude uses when responding with the `effort` parameter,
|
|||
|
||||
The `effort` parameter allows you to control how eager Claude is about spending tokens when responding to requests. This gives you the ability to trade off between response thoroughness and token efficiency, all with a single model.
|
||||
|
||||
**Note**: The effort parameter is currently in beta and only supported by Claude Opus 4.5. You must include the beta header `effort-2025-11-24` when using this feature (LiteLLM automatically adds this header when `output_config` with `effort` is detected).
|
||||
**Note**: The effort parameter is currently in beta and only supported by Claude Opus 4.5. LiteLLM automatically adds the `effort-2025-11-24` beta header when:
|
||||
- `reasoning_effort` parameter is provided (for Claude Opus 4.5 only)
|
||||
|
||||
For Claude Opus 4.5, `reasoning_effort="medium"`—both are automatically mapped to the correct format.
|
||||
|
||||
## How Effort Works
|
||||
|
||||
|
|
@ -52,9 +55,7 @@ response = litellm.completion(
|
|||
"role": "user",
|
||||
"content": "Analyze the trade-offs between microservices and monolithic architectures"
|
||||
}],
|
||||
output_config={
|
||||
"effort": "medium"
|
||||
}
|
||||
reasoning_effort="medium" # Automatically mapped to output_config for Opus 4.5
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
|
|
@ -217,11 +218,14 @@ response = litellm.completion(
|
|||
|
||||
The effort parameter is supported across all Anthropic-compatible providers:
|
||||
|
||||
- **Standard Anthropic**: ✅ Supported (Claude Opus 4.5)
|
||||
- **Azure Anthropic**: ✅ Supported (Claude Opus 4.5)
|
||||
- **Vertex AI Anthropic**: ✅ Supported (Claude Opus 4.5)
|
||||
- **Standard Anthropic API**: ✅ Supported (Claude Opus 4.5)
|
||||
- **Azure Anthropic / Microsoft Foundry**: ✅ Supported (Claude Opus 4.5)
|
||||
- **Amazon Bedrock**: ✅ Supported (Claude Opus 4.5)
|
||||
- **Google Cloud Vertex AI**: ✅ Supported (Claude Opus 4.5)
|
||||
|
||||
LiteLLM automatically handles the beta header injection for all providers.
|
||||
LiteLLM automatically handles:
|
||||
- Beta header injection (`effort-2025-11-24`) for all providers
|
||||
- Parameter mapping: `reasoning_effort` → `output_config={"effort": ...}` for Claude Opus 4.5
|
||||
|
||||
## Usage and Pricing
|
||||
|
||||
|
|
@ -242,9 +246,12 @@ print(f"Total tokens: {response.usage.total_tokens}")
|
|||
|
||||
### Beta header not being added
|
||||
|
||||
LiteLLM automatically adds the `effort-2025-11-24` beta header when `output_config` with `effort` is detected. If you're not seeing the header:
|
||||
LiteLLM automatically adds the `effort-2025-11-24` beta header when:
|
||||
- `reasoning_effort` parameter is provided (for Claude Opus 4.5 only)
|
||||
|
||||
1. Ensure you're using `output_config` with an `effort` field
|
||||
If you're not seeing the header:
|
||||
|
||||
1. Ensure you're using `reasoning_effort` parameter
|
||||
2. Verify the model is Claude Opus 4.5
|
||||
3. Check that LiteLLM version supports this feature
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,11 @@
|
|||
Programmatic tool calling allows Claude to write code that calls your tools programmatically within a code execution container, rather than requiring round trips through the model for each tool invocation. This reduces latency for multi-tool workflows and decreases token consumption by allowing Claude to filter or process data before it reaches the model's context window.
|
||||
|
||||
:::info
|
||||
Programmatic tool calling is currently in public beta. LiteLLM automatically adds the required `advanced-tool-use-2025-11-20` beta header when it detects tools with the `allowed_callers` field.
|
||||
Programmatic tool calling is currently in public beta. LiteLLM automatically detects tools with the `allowed_callers` field and adds the appropriate beta header based on your provider:
|
||||
|
||||
- **Anthropic API & Microsoft Foundry**: `advanced-tool-use-2025-11-20`
|
||||
- **Amazon Bedrock**: `advanced-tool-use-2025-11-20`
|
||||
- **Google Cloud Vertex AI**: Not supported
|
||||
|
||||
This feature requires the code execution tool to be enabled.
|
||||
:::
|
||||
|
|
@ -380,13 +384,14 @@ For example, calling 10 tools directly uses ~10x the tokens of calling them prog
|
|||
|
||||
## Provider Support
|
||||
|
||||
LiteLLM supports programmatic tool calling across all Anthropic-compatible providers:
|
||||
LiteLLM supports programmatic tool calling across the following Anthropic-compatible providers:
|
||||
|
||||
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`)
|
||||
- **Azure Anthropic** (`azure/claude-sonnet-4-5-20250929`)
|
||||
- **Vertex AI Anthropic** (`vertex_ai/claude-sonnet-4-5-20250929`)
|
||||
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`) ✅
|
||||
- **Azure Anthropic / Microsoft Foundry** (`azure/claude-sonnet-4-5-20250929`) ✅
|
||||
- **Amazon Bedrock** (`bedrock/invoke/anthropic.claude-sonnet-4-5-20250929-v1:0`) ✅
|
||||
- **Google Cloud Vertex AI** (`vertex_ai/claude-sonnet-4-5-20250929`) ❌ Not supported
|
||||
|
||||
The beta header is automatically added when LiteLLM detects tools with `allowed_callers` field.
|
||||
The beta header (`advanced-tool-use-2025-11-20`) is automatically added when LiteLLM detects tools with the `allowed_callers` field.
|
||||
|
||||
## Limitations
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,13 @@
|
|||
Provide concrete examples of valid tool inputs to help Claude understand how to use your tools more effectively. This is particularly useful for complex tools with nested objects, optional parameters, or format-sensitive inputs.
|
||||
|
||||
:::info
|
||||
Tool input examples is a beta feature. LiteLLM automatically adds the required `advanced-tool-use-2025-11-20` beta header when it detects tools with the `input_examples` field.
|
||||
Tool input examples is a beta feature. LiteLLM automatically detects tools with the `input_examples` field and adds the appropriate beta header based on your provider:
|
||||
|
||||
- **Anthropic API & Microsoft Foundry**: `advanced-tool-use-2025-11-20`
|
||||
- **Amazon Bedrock**: `advanced-tool-use-2025-11-20` (Claude Opus 4.5 only)
|
||||
- **Google Cloud Vertex AI**: Not supported
|
||||
|
||||
You don't need to manually specify beta headers—LiteLLM handles this automatically.
|
||||
:::
|
||||
|
||||
## When to Use Input Examples
|
||||
|
|
@ -378,13 +384,14 @@ Input examples work seamlessly with other Anthropic tool features:
|
|||
|
||||
## Provider Support
|
||||
|
||||
LiteLLM supports input examples across all Anthropic-compatible providers:
|
||||
LiteLLM supports input examples across the following Anthropic-compatible providers:
|
||||
|
||||
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`)
|
||||
- **Azure Anthropic** (`azure/claude-sonnet-4-5-20250929`)
|
||||
- **Vertex AI Anthropic** (`vertex_ai/claude-sonnet-4-5-20250929`)
|
||||
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`) ✅
|
||||
- **Azure Anthropic / Microsoft Foundry** (`azure/claude-sonnet-4-5-20250929`) ✅
|
||||
- **Amazon Bedrock** (`bedrock/invoke/anthropic.claude-opus-4-5-20251101-v1:0`) ✅ (Opus 4.5 only)
|
||||
- **Google Cloud Vertex AI** (`vertex_ai/claude-sonnet-4-5-20250929`) ❌ Not supported
|
||||
|
||||
The beta header is automatically added when LiteLLM detects tools with `input_examples` field.
|
||||
The beta header (`advanced-tool-use-2025-11-20`) is automatically added when LiteLLM detects tools with the `input_examples` field.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
|
|
|||
|
|
@ -290,7 +290,13 @@ response = client.chat.completions.create(
|
|||
|
||||
### Beta Header
|
||||
|
||||
LiteLLM automatically adds the `advanced-tool-use-2025-11-20` beta header when tool search tools are detected. You don't need to manually specify it.
|
||||
LiteLLM automatically detects tool search tools and adds the appropriate beta header based on your provider:
|
||||
|
||||
- **Anthropic API & Microsoft Foundry**: `advanced-tool-use-2025-11-20`
|
||||
- **Google Cloud Vertex AI**: `tool-search-tool-2025-10-19`
|
||||
- **Amazon Bedrock** (Invoke API, Opus 4.5 only): `tool-search-tool-2025-10-19`
|
||||
|
||||
You don't need to manually specify beta headers—LiteLLM handles this automatically.
|
||||
|
||||
### Deferred Loading
|
||||
|
||||
|
|
@ -387,9 +393,18 @@ If Claude references a tool that isn't in your deferred tools list, you'll get a
|
|||
- Not compatible with tool use examples
|
||||
- Requires Claude Opus 4.5 or Sonnet 4.5
|
||||
- On Bedrock, only available via invoke API (not converse API)
|
||||
- On Bedrock, only supported for Claude Opus 4.5 (not Sonnet 4.5)
|
||||
- BM25 variant (`tool_search_tool_bm25_20251119`) is not supported on Bedrock
|
||||
- Maximum 10,000 tools in catalog
|
||||
- Returns 3-5 most relevant tools per search
|
||||
|
||||
### Bedrock-Specific Notes
|
||||
|
||||
When using Bedrock's Invoke API:
|
||||
- The regex variant (`tool_search_tool_regex_20251119`) is automatically normalized to `tool_search_tool_regex`
|
||||
- The BM25 variant (`tool_search_tool_bm25_20251119`) is automatically filtered out as it's not supported
|
||||
- Tool search is only available for Claude Opus 4.5 models
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Anthropic Tool Search Documentation](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/tool-search)
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue