scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

docs: cleanup launch post

Browse source
This commit is contained in:
Krrish Dholakia 2025-11-25 15:58:51 -08:00
parent 70a1325847
commit 8ee6812edf
2 changed files with 470 additions and 48 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

503
docs/my-website/blog/anthropic_opus_4_5_and_advanced_features/index.md
View file

@ -329,8 +329,13 @@ curl --location 'http://0.0.0.0:4000/v1/messages' \
## Tool Search {#tool-search}
This lets Claude work with thousands of tools, by dynamically loading tools on-demand, instead of loading all tools into the context window upfront.
### Usage Example
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
import os
@ -407,7 +412,7 @@ tools = [
# Make a request - Claude will search for and use relevant tools
response = litellm.completion(
model="anthropic/claude-sonnet-4-5-20250929",
model="anthropic/claude-opus-4-5-20251101",
messages=[{
"role": "user",
"content": "What's the weather like in San Francisco?"
@ -422,6 +427,108 @@ print("Tool calls:", response.choices[0].message.tool_calls)
if hasattr(response.usage, 'server_tool_use'):
print(f"Tool searches performed: {response.usage.server_tool_use.tool_search_requests}")
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "What's the weather like in San Francisco?"
}],
"tools": [
# Tool search tool (regex variant)
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# Deferred tools - loaded on-demand
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a given location. Returns temperature and conditions.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
},
"defer_loading": True # Load on-demand
},
{
"type": "function",
"function": {
"name": "search_files",
"description": "Search through files in the workspace using keywords",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"file_types": {
"type": "array",
"items": {"type": "string"}
}
},
"required": ["query"]
}
},
"defer_loading": True
},
{
"type": "function",
"function": {
"name": "query_database",
"description": "Execute SQL queries against the database",
"parameters": {
"type": "object",
"properties": {
"sql": {"type": "string"}
},
"required": ["sql"]
}
},
"defer_loading": True
}
]
}
'
```
</TabItem>
</Tabs>
### BM25 Variant (Natural Language Search)
@ -441,6 +548,11 @@ tools = [
## Programmatic Tool Calling {#programmatic-tool-calling}
Programmatic tool calling allows Claude to write code that calls your tools programmatically. [Learn more](https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling)
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
import json
@ -527,10 +639,80 @@ final_response = litellm.completion(
print("\nFinal answer:", final_response.choices[0].message.content)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Query sales data for West, East, and Central regions, then tell me which had the highest revenue"
}],
"tools": [
# Code execution tool (required for programmatic calling)
{
"type": "code_execution_20250825",
"name": "code_execution"
},
# Tool that can be called from code
{
"type": "function",
"function": {
"name": "query_database",
"description": "Execute a SQL query against the sales database. Returns a list of rows as JSON objects.",
"parameters": {
"type": "object",
"properties": {
"sql": {
"type": "string",
"description": "SQL query to execute"
}
},
"required": ["sql"]
}
},
"allowed_callers": ["code_execution_20250825"] # Enable programmatic calling
}
]
}
'
```
</TabItem>
</Tabs>
---
## Tool Input Examples {#tool-input-examples}
You can now provide Claude with examples of how to use your tools. [Learn more](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-input-examples)
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
@ -609,12 +791,124 @@ response = litellm.completion(
print("Tool call:", response.choices[0].message.tool_calls[0].function.arguments)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Schedule a team meeting for tomorrow at 2pm for 45 minutes with john@company.com and sarah@company.com"
}],
"tools": [
{
"type": "function",
"function": {
"name": "create_calendar_event",
"description": "Create a new calendar event with attendees and reminders",
"parameters": {
"type": "object",
"properties": {
"title": {"type": "string"},
"start_time": {
"type": "string",
"description": "ISO 8601 format: YYYY-MM-DDTHH:MM:SS"
},
"duration_minutes": {"type": "integer"},
"attendees": {
"type": "array",
"items": {
"type": "object",
"properties": {
"email": {"type": "string"},
"optional": {"type": "boolean"}
}
}
},
"reminders": {
"type": "array",
"items": {
"type": "object",
"properties": {
"minutes_before": {"type": "integer"},
"method": {"type": "string", "enum": ["email", "popup"]}
}
}
}
},
"required": ["title", "start_time", "duration_minutes"]
}
},
# Provide concrete examples
"input_examples": [
{
"title": "Team Standup",
"start_time": "2025-01-15T09:00:00",
"duration_minutes": 30,
"attendees": [
{"email": "alice@company.com", "optional": False},
{"email": "bob@company.com", "optional": False}
],
"reminders": [
{"minutes_before": 15, "method": "popup"}
]
},
{
"title": "Lunch Break",
"start_time": "2025-01-15T12:00:00",
"duration_minutes": 60
# Demonstrates optional fields can be omitted
}
]
}
]
}
'
```
</TabItem>
</Tabs>
---
## Effort Parameter: Control Token Usage {#effort-parameter}
Controls aspects like how much effort the model puts into its response, via `output_config={"effort": ..}`.
:::info
Soon, we will map OpenAI's `reasoning_effort` parameter to this.
:::
Potential Values for `effort` parameter: `"high"`, `"medium"`, `"low"`.
### Usage Example
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
@ -660,41 +954,46 @@ print(f"Medium: {response_medium.usage.completion_tokens} tokens")
print(f"Low: {response_low.usage.completion_tokens} tokens")
```
### Effort with Tool Use
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
Lower effort affects both explanations and tool calls:
1. Setup config.yaml
```python
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}
}
}
]
# Low effort = fewer tool calls, more direct
response = litellm.completion(
model="anthropic/claude-opus-4-5-20251101",
messages=[{
"role": "user",
"content": "Check weather in San Francisco, New York, and London"
}],
tools=tools,
output_config={"effort": "low"} # May combine into fewer calls
)
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
---
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Analyze the trade-offs between microservices and monolithic architectures"
}],
"output_config": {
"effort": "high"
}
}
'
```
</TabItem>
</Tabs>
## Cost Tracking: Monitor Tool Search Usage {#cost-tracking}
@ -702,8 +1001,15 @@ response = litellm.completion(
Tool search operations are tracked separately in the usage object, allowing you to monitor and optimize costs.
It is available in the `usage` object, under `server_tool_use.tool_search_requests`.
Anthropic charges $0.0001 per tool search request.
### Tracking Example
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
@ -749,6 +1055,65 @@ if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use
print(f" Total: ${total_cost:.6f}")
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Find and use the weather tool for San Francisco"
}],
"tools": [
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# ... 100 deferred tools
]
}
'
```
Expected Response:
```json
{
...,
"usage": {
...,
"server_tool_use": {
"tool_search_requests": 1
}
}
}
```
</TabItem>
</Tabs>
### Cost Optimization Tips
1. **Keep frequently used tools non-deferred** (3-5 tools)
@ -756,15 +1121,6 @@ if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use
3. **Monitor search requests** to identify optimization opportunities
4. **Combine with effort parameter** for maximum efficiency
```python
# Optimized for cost
response = litellm.completion(
model="anthropic/claude-sonnet-4-5-20250929",
messages=[{"role": "user", "content": "Simple query"}],
tools=tools_with_search,
output_config={"effort": "low"} # Reduce output tokens
)
```
---
@ -774,6 +1130,9 @@ response = litellm.completion(
These features work together seamlessly. Here's a real-world example combining all of them:
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
import json
@ -846,6 +1205,68 @@ if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use
print(f"\nResponse: {response.choices[0].message.content}")
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Analyze sales by region for the last quarter and identify top performers"
}],
"tools": [
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# ... 100 deferred tools
],
"output_config": {
"effort": "medium"
}
}
'
```
Expected Response:
```json
{
...,
"usage": {
...,
"server_tool_use": {
"tool_search_requests": 1
}
}
}
```
</TabItem>
</Tabs>
### Real-World Benefits
This combination enables:

15
litellm/model_prices_and_context_window_backup.json
View file

@ -2687,7 +2687,7 @@
"/v1/audio/transcriptions"
]
},
"azure/gpt-5.1-2025-11-13": {
"azure/gpt-5.1-2025-11-13": {
"cache_read_input_token_cost": 1.25e-07,
"cache_read_input_token_cost_priority": 2.5e-07,
"input_cost_per_token": 1.25e-06,
@ -2723,7 +2723,7 @@
"supports_service_tier": true,
"supports_vision": true
},
"azure/gpt-5.1-chat-2025-11-13": {
"azure/gpt-5.1-chat-2025-11-13": {
"cache_read_input_token_cost": 1.25e-07,
"cache_read_input_token_cost_priority": 2.5e-07,
"input_cost_per_token": 1.25e-06,
@ -2758,7 +2758,7 @@
"supports_tool_choice": false,
"supports_vision": true
},
"azure/gpt-5.1-codex-2025-11-13": {
"azure/gpt-5.1-codex-2025-11-13": {
"cache_read_input_token_cost": 1.25e-07,
"cache_read_input_token_cost_priority": 2.5e-07,
"input_cost_per_token": 1.25e-06,
@ -19521,7 +19521,7 @@
"output_cost_per_token": 0.0,
"supports_function_calling": true
},
"ollama/deepseek-v3.1:671b-cloud": {
"ollama/deepseek-v3.1:671b-cloud" : {
"input_cost_per_token": 0.0,
"litellm_provider": "ollama",
"max_input_tokens": 163840,
@ -19531,7 +19531,7 @@
"output_cost_per_token": 0.0,
"supports_function_calling": true
},
"ollama/gpt-oss:120b-cloud": {
"ollama/gpt-oss:120b-cloud" : {
"input_cost_per_token": 0.0,
"litellm_provider": "ollama",
"max_input_tokens": 131072,
@ -19541,7 +19541,7 @@
"output_cost_per_token": 0.0,
"supports_function_calling": true
},
"ollama/gpt-oss:20b-cloud": {
"ollama/gpt-oss:20b-cloud" : {
"input_cost_per_token": 0.0,
"litellm_provider": "ollama",
"max_input_tokens": 131072,
@ -22037,6 +22037,7 @@
"supports_reasoning": true,
"source": "https://cloud.sambanova.ai/plans/pricing"
},
"snowflake/claude-3-5-sonnet": {
"litellm_provider": "snowflake",
"max_input_tokens": 18000,
@ -23231,7 +23232,7 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"global.anthropic.claude-opus-4-5-20251101-v1:0": {
"us.anthropic.claude-opus-4-5-20251101-v1:0": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,