mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
Merge branch 'main' into litellm_tooling_x
This commit is contained in:
Alexsander Hamir
commit
06d2cbaaf2
21 changed files with 2102 additions and 87 deletions
192
docs/my-website/docs/tutorials/claude_code_websearch.md
Normal file
192
docs/my-website/docs/tutorials/claude_code_websearch.md
Normal file
|
|
@ -0,0 +1,192 @@
|
|||
# Claude Code - WebSearch Across All Providers
|
||||
|
||||
Enable Claude Code's web search tool to work with any provider (Bedrock, Azure, Vertex, etc.). LiteLLM automatically intercepts web search requests and executes them server-side.
|
||||
|
||||
## Proxy Configuration
|
||||
|
||||
Add WebSearch interception to your `litellm_config.yaml`:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: bedrock-sonnet
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
||||
aws_region_name: us-east-1
|
||||
|
||||
# Enable WebSearch interception for providers
|
||||
litellm_settings:
|
||||
callbacks:
|
||||
- websearch_interception:
|
||||
enabled_providers:
|
||||
- bedrock
|
||||
- azure
|
||||
- vertex_ai
|
||||
search_tool_name: perplexity-search # Optional: specific search tool
|
||||
|
||||
# Configure search provider
|
||||
search_tools:
|
||||
- search_tool_name: perplexity-search
|
||||
litellm_params:
|
||||
search_provider: perplexity
|
||||
api_key: os.environ/PERPLEXITY_API_KEY
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Configure LiteLLM Proxy
|
||||
|
||||
Create `config.yaml`:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: bedrock-sonnet
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
||||
aws_region_name: us-east-1
|
||||
|
||||
litellm_settings:
|
||||
callbacks:
|
||||
- websearch_interception:
|
||||
enabled_providers: [bedrock]
|
||||
|
||||
search_tools:
|
||||
- search_tool_name: perplexity-search
|
||||
litellm_params:
|
||||
search_provider: perplexity
|
||||
api_key: os.environ/PERPLEXITY_API_KEY
|
||||
```
|
||||
|
||||
### 2. Start Proxy
|
||||
|
||||
```bash
|
||||
export PERPLEXITY_API_KEY=your-key
|
||||
litellm --config config.yaml
|
||||
```
|
||||
|
||||
### 3. Use with Claude Code
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://localhost:4000
|
||||
export ANTHROPIC_API_KEY=sk-1234
|
||||
claude
|
||||
```
|
||||
|
||||
Now use web search in Claude Code - it works with any provider!
|
||||
|
||||
## How It Works
|
||||
|
||||
When Claude Code sends a web search request, LiteLLM:
|
||||
1. Intercepts the native `web_search` tool
|
||||
2. Converts it to LiteLLM's standard format
|
||||
3. Executes the search via Perplexity/Tavily
|
||||
4. Returns the final answer to Claude Code
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CC as Claude Code
|
||||
participant LP as LiteLLM Proxy
|
||||
participant B as Bedrock/Azure/etc
|
||||
participant P as Perplexity/Tavily
|
||||
|
||||
CC->>LP: Request with web_search tool
|
||||
Note over LP: Convert native tool<br/>to LiteLLM format
|
||||
LP->>B: Request with converted tool
|
||||
B-->>LP: Response: tool_use
|
||||
Note over LP: Detect web search<br/>tool_use
|
||||
LP->>P: Execute search
|
||||
P-->>LP: Search results
|
||||
LP->>B: Follow-up with results
|
||||
B-->>LP: Final answer
|
||||
LP-->>CC: Final answer with search results
|
||||
```
|
||||
|
||||
**Result**: One API call from Claude Code → Complete answer with search results
|
||||
|
||||
## Supported Providers
|
||||
|
||||
| Provider | Native Web Search | With LiteLLM |
|
||||
|----------|-------------------|--------------|
|
||||
| **Anthropic** | ✅ Yes | ✅ Yes |
|
||||
| **Bedrock** | ❌ No | ✅ Yes |
|
||||
| **Azure** | ❌ No | ✅ Yes |
|
||||
| **Vertex AI** | ❌ No | ✅ Yes |
|
||||
| **Other Providers** | ❌ No | ✅ Yes |
|
||||
|
||||
## Search Providers
|
||||
|
||||
Configure which search provider to use. LiteLLM supports multiple search providers:
|
||||
|
||||
| Provider | Configuration |
|
||||
|----------|---------------|
|
||||
| **Perplexity** | `search_provider: perplexity` |
|
||||
| **Tavily** | `search_provider: tavily` |
|
||||
|
||||
See [all supported search providers](../search/index.md) for the complete list.
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### WebSearch Interception Parameters
|
||||
|
||||
| Parameter | Type | Required | Description | Example |
|
||||
|-----------|------|----------|-------------|---------|
|
||||
| `enabled_providers` | List[String] | Yes | List of providers to enable web search interception for | `[bedrock, azure, vertex_ai]` |
|
||||
| `search_tool_name` | String | No | Specific search tool from `search_tools` config. If not set, uses first available search tool. | `perplexity-search` |
|
||||
|
||||
### Supported Provider Values
|
||||
|
||||
Use these values in `enabled_providers`:
|
||||
|
||||
| Provider | Value | Description |
|
||||
|----------|-------|-------------|
|
||||
| AWS Bedrock | `bedrock` | Amazon Bedrock Claude models |
|
||||
| Azure OpenAI | `azure` | Azure-hosted models |
|
||||
| Google Vertex AI | `vertex_ai` | Google Cloud Vertex AI |
|
||||
| Any Other | Provider name | Any LiteLLM-supported provider |
|
||||
|
||||
### Complete Configuration Example
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: bedrock-sonnet
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
||||
aws_region_name: us-east-1
|
||||
|
||||
- model_name: azure-gpt4
|
||||
litellm_params:
|
||||
model: azure/gpt-4
|
||||
api_base: https://my-azure.openai.azure.com
|
||||
api_key: os.environ/AZURE_API_KEY
|
||||
|
||||
litellm_settings:
|
||||
callbacks:
|
||||
- websearch_interception:
|
||||
enabled_providers:
|
||||
- bedrock # Enable for AWS Bedrock
|
||||
- azure # Enable for Azure OpenAI
|
||||
- vertex_ai # Enable for Google Vertex
|
||||
search_tool_name: perplexity-search # Optional: use specific search tool
|
||||
|
||||
# Configure search tools
|
||||
search_tools:
|
||||
- search_tool_name: perplexity-search
|
||||
litellm_params:
|
||||
search_provider: perplexity
|
||||
api_key: os.environ/PERPLEXITY_API_KEY
|
||||
|
||||
- search_tool_name: tavily-search
|
||||
litellm_params:
|
||||
search_provider: tavily
|
||||
api_key: os.environ/TAVILY_API_KEY
|
||||
```
|
||||
|
||||
**How search tool selection works:**
|
||||
- If `search_tool_name` is specified → Uses that specific search tool
|
||||
- If `search_tool_name` is not specified → Uses first search tool in `search_tools` list
|
||||
- In example above: Without `search_tool_name`, would use `perplexity-search` (first in list)
|
||||
|
||||
## Related
|
||||
|
||||
- [Claude Code Quickstart](./claude_responses_api.md)
|
||||
- [Claude Code Cost Tracking](./claude_code_customer_tracking.md)
|
||||
- [Using Non-Anthropic Models](./claude_non_anthropic_models.md)
|
||||
487
docs/my-website/release_notes/v1.81.0/index.md
Normal file
487
docs/my-website/release_notes/v1.81.0/index.md
Normal file
|
|
@ -0,0 +1,487 @@
|
|||
---
|
||||
title: "v1.81.0 - Claude Code - Web Search with all LiteLLM Providers"
|
||||
slug: "v1-81-0"
|
||||
date: 2026-01-18T10:00:00
|
||||
authors:
|
||||
- name: Krrish Dholakia
|
||||
title: CEO, LiteLLM
|
||||
url: https://www.linkedin.com/in/krish-d/
|
||||
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
|
||||
- name: Ishaan Jaff
|
||||
title: CTO, LiteLLM
|
||||
url: https://www.linkedin.com/in/reffajnaahsi/
|
||||
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
|
||||
hide_table_of_contents: false
|
||||
---
|
||||
|
||||
import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
## Deploy this version
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="docker" label="Docker">
|
||||
|
||||
``` showLineNumbers title="docker run litellm"
|
||||
docker run \
|
||||
-e STORE_MODEL_IN_DB=True \
|
||||
-p 4000:4000 \
|
||||
docker.litellm.ai/berriai/litellm:v1.81.0
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="pip" label="Pip">
|
||||
|
||||
``` showLineNumbers title="pip install litellm"
|
||||
pip install litellm==1.81.0
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
## Key Highlights
|
||||
|
||||
- **Claude Code** - Support for using web search across Bedrock, Vertex AI, and all LiteLLM providers
|
||||
- **Major Change** - [50MB limit on image URL downloads](#major-change---chatcompletions-image-url-download-size-limit) to improve reliability
|
||||
|
||||
---
|
||||
|
||||
## Major Change - /chat/completions Image URL Download Size Limit
|
||||
|
||||
To improve reliability and prevent memory issues, LiteLLM now includes a configurable **50MB limit** on image URL downloads by default. Previously, there was no limit on image downloads, which could occasionally cause memory issues with very large images.
|
||||
|
||||
### How It Works
|
||||
|
||||
Requests with image URLs exceeding 50MB will receive a helpful error message:
|
||||
|
||||
```bash
|
||||
curl -X POST 'https://your-litellm-proxy.com/chat/completions' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-d '{
|
||||
"model": "gpt-4o",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "What is in this image?"
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/very-large-image.jpg"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
**Error Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"message": "Error: Image size (75.50MB) exceeds maximum allowed size (50.0MB). url=https://example.com/very-large-image.jpg",
|
||||
"type": "ImageFetchError"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuring the Limit
|
||||
|
||||
The default 50MB limit works well for most use cases, but you can easily adjust it if needed:
|
||||
|
||||
**Increase the limit (e.g., to 100MB):**
|
||||
|
||||
```bash
|
||||
export MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=100
|
||||
```
|
||||
|
||||
**Disable image URL downloads (for security):**
|
||||
|
||||
```bash
|
||||
export MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=0
|
||||
```
|
||||
|
||||
**Docker Configuration:**
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
-e MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=100 \
|
||||
-p 4000:4000 \
|
||||
docker.litellm.ai/berriai/litellm:v1.81.0
|
||||
```
|
||||
|
||||
**Proxy Config (config.yaml):**
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
|
||||
# Set via environment variable
|
||||
environment_variables:
|
||||
MAX_IMAGE_URL_DOWNLOAD_SIZE_MB: "100"
|
||||
```
|
||||
|
||||
### Why Add This?
|
||||
|
||||
This feature improves reliability by:
|
||||
- Preventing memory issues from very large images
|
||||
- Aligning with OpenAI's 50MB payload limit
|
||||
- Validating image sizes early (when Content-Length header is available)
|
||||
|
||||
---
|
||||
|
||||
## New Models / Updated Models
|
||||
|
||||
#### New Model Support
|
||||
|
||||
| Provider | Model | Features |
|
||||
| -------- | ----- | -------- |
|
||||
| OpenAI | `gpt-5.2-codex` | Code generation |
|
||||
| Azure | `azure/gpt-5.2-codex` | Code generation |
|
||||
| Cerebras | `cerebras/zai-glm-4.7` | Reasoning, function calling |
|
||||
| Replicate | All chat models | Full support for all Replicate chat models |
|
||||
|
||||
#### Features
|
||||
|
||||
- **[Anthropic](../../docs/providers/anthropic)**
|
||||
- Add missing anthropic tool results in response - [PR #18945](https://github.com/BerriAI/litellm/pull/18945)
|
||||
- Preserve web_fetch_tool_result in multi-turn conversations - [PR #18142](https://github.com/BerriAI/litellm/pull/18142)
|
||||
|
||||
- **[Gemini](../../docs/providers/gemini)**
|
||||
- Add presence_penalty support for Google AI Studio - [PR #18154](https://github.com/BerriAI/litellm/pull/18154)
|
||||
- Forward extra_headers in generateContent adapter - [PR #18935](https://github.com/BerriAI/litellm/pull/18935)
|
||||
- Add medium value support for detail param - [PR #19187](https://github.com/BerriAI/litellm/pull/19187)
|
||||
|
||||
- **[Vertex AI](../../docs/providers/vertex)**
|
||||
- Improve passthrough endpoint URL parsing and construction - [PR #17526](https://github.com/BerriAI/litellm/pull/17526)
|
||||
- Add type object to tool schemas missing type field - [PR #19103](https://github.com/BerriAI/litellm/pull/19103)
|
||||
- Keep type field in Gemini schema when properties is empty - [PR #18979](https://github.com/BerriAI/litellm/pull/18979)
|
||||
|
||||
- **[Bedrock](../../docs/providers/bedrock)**
|
||||
- Add OpenAI-compatible service_tier parameter translation - [PR #18091](https://github.com/BerriAI/litellm/pull/18091)
|
||||
- Add user auth in standard logging object for Bedrock passthrough - [PR #19140](https://github.com/BerriAI/litellm/pull/19140)
|
||||
- Strip throughput tier suffixes from model names - [PR #19147](https://github.com/BerriAI/litellm/pull/19147)
|
||||
|
||||
- **[OCI](../../docs/providers/oci)**
|
||||
- Handle OpenAI-style image_url object in multimodal messages - [PR #18272](https://github.com/BerriAI/litellm/pull/18272)
|
||||
|
||||
- **[Ollama](../../docs/providers/ollama)**
|
||||
- Set finish_reason to tool_calls and remove broken capability check - [PR #18924](https://github.com/BerriAI/litellm/pull/18924)
|
||||
|
||||
- **[Watsonx](../../docs/providers/watsonx/index)**
|
||||
- Allow passing scope ID for Watsonx inferencing - [PR #18959](https://github.com/BerriAI/litellm/pull/18959)
|
||||
|
||||
- **[Replicate](../../docs/providers/replicate)**
|
||||
- Add all chat Replicate models support - [PR #18954](https://github.com/BerriAI/litellm/pull/18954)
|
||||
|
||||
- **[OpenRouter](../../docs/providers/openrouter)**
|
||||
- Add OpenRouter support for image/generation endpoints - [PR #19059](https://github.com/BerriAI/litellm/pull/19059)
|
||||
|
||||
- **[Volcengine](../../docs/providers/volcano)**
|
||||
- Add max_tokens settings for Volcengine models (deepseek-v3-2, glm-4-7, kimi-k2-thinking) - [PR #19076](https://github.com/BerriAI/litellm/pull/19076)
|
||||
|
||||
- **Azure Model Router**
|
||||
- New Model - Azure Model Router on LiteLLM AI Gateway - [PR #19054](https://github.com/BerriAI/litellm/pull/19054)
|
||||
|
||||
- **GPT-5 Models**
|
||||
- Correct context window sizes for GPT-5 model variants - [PR #18928](https://github.com/BerriAI/litellm/pull/18928)
|
||||
- Correct max_input_tokens for GPT-5 models - [PR #19056](https://github.com/BerriAI/litellm/pull/19056)
|
||||
|
||||
- **Text Completion**
|
||||
- Support token IDs (list of integers) as prompt - [PR #18011](https://github.com/BerriAI/litellm/pull/18011)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **[Anthropic](../../docs/providers/anthropic)**
|
||||
- Prevent dropping thinking when any message has thinking_blocks - [PR #18929](https://github.com/BerriAI/litellm/pull/18929)
|
||||
- Fix anthropic token counter with thinking - [PR #19067](https://github.com/BerriAI/litellm/pull/19067)
|
||||
- Add better error handling for Anthropic - [PR #18955](https://github.com/BerriAI/litellm/pull/18955)
|
||||
- Fix Anthropic during call error - [PR #19060](https://github.com/BerriAI/litellm/pull/19060)
|
||||
|
||||
- **[Gemini](../../docs/providers/gemini)**
|
||||
- Fix missing `completion_tokens_details` in Gemini 3 Flash when reasoning_effort is not used - [PR #18898](https://github.com/BerriAI/litellm/pull/18898)
|
||||
- Fix Gemini Image Generation imageConfig parameters - [PR #18948](https://github.com/BerriAI/litellm/pull/18948)
|
||||
|
||||
- **[Vertex AI](../../docs/providers/vertex)**
|
||||
- Fix Vertex AI 400 Error with CachedContent model mismatch - [PR #19193](https://github.com/BerriAI/litellm/pull/19193)
|
||||
- Fix Vertex AI doesn't support structured output - [PR #19201](https://github.com/BerriAI/litellm/pull/19201)
|
||||
|
||||
- **[Bedrock](../../docs/providers/bedrock)**
|
||||
- Fix Claude Code (`/messages`) Bedrock Invoke usage and request signing - [PR #19111](https://github.com/BerriAI/litellm/pull/19111)
|
||||
- Fix model ID encoding for Bedrock passthrough - [PR #18944](https://github.com/BerriAI/litellm/pull/18944)
|
||||
- Respect max_completion_tokens in thinking feature - [PR #18946](https://github.com/BerriAI/litellm/pull/18946)
|
||||
- Fix header forwarding in Bedrock passthrough - [PR #19007](https://github.com/BerriAI/litellm/pull/19007)
|
||||
- Fix Bedrock stability model usage issues - [PR #19199](https://github.com/BerriAI/litellm/pull/19199)
|
||||
|
||||
---
|
||||
|
||||
## LLM API Endpoints
|
||||
|
||||
#### Features
|
||||
|
||||
- **[/messages (Claude Code)](../../docs/providers/anthropic)**
|
||||
- Add support for Tool Search on `/messages` API across Azure, Bedrock, and Anthropic API - [PR #19165](https://github.com/BerriAI/litellm/pull/19165)
|
||||
- Track end-users with Claude Code (`/messages`) for better analytics and monitoring - [PR #19171](https://github.com/BerriAI/litellm/pull/19171)
|
||||
- Add web search support using LiteLLM `/search` endpoint with Claude Code (`/messages`) - [PR #19263](https://github.com/BerriAI/litellm/pull/19263), [PR #19294](https://github.com/BerriAI/litellm/pull/19294)
|
||||
|
||||
- **[/messages (Claude Code) - Bedrock](../../docs/providers/bedrock)**
|
||||
- Add support for Prompt Caching with Bedrock Converse on `/messages` - [PR #19123](https://github.com/BerriAI/litellm/pull/19123)
|
||||
- Ensure budget tokens are passed to Bedrock Converse API correctly on `/messages` - [PR #19107](https://github.com/BerriAI/litellm/pull/19107)
|
||||
|
||||
- **[Responses API](../../docs/response_api)**
|
||||
- Add support for caching for responses API - [PR #19068](https://github.com/BerriAI/litellm/pull/19068)
|
||||
- Add retry policy support to responses API - [PR #19074](https://github.com/BerriAI/litellm/pull/19074)
|
||||
|
||||
- **Realtime API**
|
||||
- Use non-streaming method for endpoint v1/a2a/message/send - [PR #19025](https://github.com/BerriAI/litellm/pull/19025)
|
||||
|
||||
- **Batch API**
|
||||
- Fix batch deletion and retrieve - [PR #18340](https://github.com/BerriAI/litellm/pull/18340)
|
||||
|
||||
#### Bugs
|
||||
|
||||
- **General**
|
||||
- Fix responses content can't be none - [PR #19064](https://github.com/BerriAI/litellm/pull/19064)
|
||||
- Fix model name from query param in realtime request - [PR #19135](https://github.com/BerriAI/litellm/pull/19135)
|
||||
- Fix video status/content credential injection for wildcard models - [PR #18854](https://github.com/BerriAI/litellm/pull/18854)
|
||||
|
||||
---
|
||||
|
||||
## Management Endpoints / UI
|
||||
|
||||
#### Features
|
||||
|
||||
**Virtual Keys**
|
||||
- View deleted keys for audit purposes - [PR #18228](https://github.com/BerriAI/litellm/pull/18228), [PR #19268](https://github.com/BerriAI/litellm/pull/19268)
|
||||
- Add status query parameter for keys list - [PR #19260](https://github.com/BerriAI/litellm/pull/19260)
|
||||
- Refetch keys after key creation - [PR #18994](https://github.com/BerriAI/litellm/pull/18994)
|
||||
- Refresh keys list on delete - [PR #19262](https://github.com/BerriAI/litellm/pull/19262)
|
||||
- Simplify key generate permission error - [PR #18997](https://github.com/BerriAI/litellm/pull/18997)
|
||||
- Add search to key edit team dropdown - [PR #19119](https://github.com/BerriAI/litellm/pull/19119)
|
||||
|
||||
**Teams & Organizations**
|
||||
- View deleted teams for audit purposes - [PR #18228](https://github.com/BerriAI/litellm/pull/18228), [PR #19268](https://github.com/BerriAI/litellm/pull/19268)
|
||||
- Add filters to organization table - [PR #18916](https://github.com/BerriAI/litellm/pull/18916)
|
||||
- Add query parameters to `/organization/list` - [PR #18910](https://github.com/BerriAI/litellm/pull/18910)
|
||||
- Add status query parameter for teams list - [PR #19260](https://github.com/BerriAI/litellm/pull/19260)
|
||||
- Show internal users their spend only - [PR #19227](https://github.com/BerriAI/litellm/pull/19227)
|
||||
- Allow preventing team admins from deleting members from teams - [PR #19128](https://github.com/BerriAI/litellm/pull/19128)
|
||||
- Refactor team member icon buttons - [PR #19192](https://github.com/BerriAI/litellm/pull/19192)
|
||||
|
||||
**Models + Endpoints**
|
||||
- Display health information in public model hub - [PR #19256](https://github.com/BerriAI/litellm/pull/19256), [PR #19258](https://github.com/BerriAI/litellm/pull/19258)
|
||||
- Quality of life improvements for Anthropic models - [PR #19058](https://github.com/BerriAI/litellm/pull/19058)
|
||||
- Create reusable model select component - [PR #19164](https://github.com/BerriAI/litellm/pull/19164)
|
||||
- Edit settings model dropdown - [PR #19186](https://github.com/BerriAI/litellm/pull/19186)
|
||||
- Fix model hub client side exception - [PR #19045](https://github.com/BerriAI/litellm/pull/19045)
|
||||
|
||||
**Usage & Analytics**
|
||||
- Allow top virtual keys and models to show more entries - [PR #19050](https://github.com/BerriAI/litellm/pull/19050)
|
||||
- Fix Y axis on model activity chart - [PR #19055](https://github.com/BerriAI/litellm/pull/19055)
|
||||
- Add Team ID and Team Name in export report - [PR #19047](https://github.com/BerriAI/litellm/pull/19047)
|
||||
- Add user metrics for Prometheus - [PR #18785](https://github.com/BerriAI/litellm/pull/18785)
|
||||
|
||||
**SSO & Auth**
|
||||
- Allow setting custom MSFT Base URLs - [PR #18977](https://github.com/BerriAI/litellm/pull/18977)
|
||||
- Allow overriding env var attribute names - [PR #18998](https://github.com/BerriAI/litellm/pull/18998)
|
||||
- Fix SCIM GET /Users error and enforce SCIM 2.0 compliance - [PR #17420](https://github.com/BerriAI/litellm/pull/17420)
|
||||
- Feature flag for SCIM compliance fix - [PR #18878](https://github.com/BerriAI/litellm/pull/18878)
|
||||
|
||||
**General UI**
|
||||
- Add allowClear to dropdown components for better UX - [PR #18778](https://github.com/BerriAI/litellm/pull/18778)
|
||||
- Add community engagement buttons - [PR #19114](https://github.com/BerriAI/litellm/pull/19114)
|
||||
- UI Feedback Form - why LiteLLM - [PR #18999](https://github.com/BerriAI/litellm/pull/18999)
|
||||
- Refactor user and team table filters to reusable component - [PR #19010](https://github.com/BerriAI/litellm/pull/19010)
|
||||
- Adjusting new badges - [PR #19278](https://github.com/BerriAI/litellm/pull/19278)
|
||||
|
||||
#### Bugs
|
||||
|
||||
- Container API routes return 401 for non-admin users - routes missing from openai_routes - [PR #19115](https://github.com/BerriAI/litellm/pull/19115)
|
||||
- Allow routing to regional endpoints for Containers API - [PR #19118](https://github.com/BerriAI/litellm/pull/19118)
|
||||
- Fix Azure Storage circular reference error - [PR #19120](https://github.com/BerriAI/litellm/pull/19120)
|
||||
- Fix prompt deletion fails with Prisma FieldNotFoundError - [PR #18966](https://github.com/BerriAI/litellm/pull/18966)
|
||||
|
||||
---
|
||||
|
||||
## AI Integrations
|
||||
|
||||
### Logging
|
||||
|
||||
- **[OpenTelemetry](../../docs/proxy/logging#opentelemetry)**
|
||||
- Update semantic conventions to 1.38 (gen_ai attributes) - [PR #18793](https://github.com/BerriAI/litellm/pull/18793)
|
||||
|
||||
- **[LangSmith](../../docs/proxy/logging#langsmith)**
|
||||
- Hoist thread grouping metadata (session_id, thread) - [PR #18982](https://github.com/BerriAI/litellm/pull/18982)
|
||||
|
||||
- **[Langfuse](../../docs/proxy/logging#langfuse)**
|
||||
- Include Langfuse logger in JSON logging when Langfuse callback is used - [PR #19162](https://github.com/BerriAI/litellm/pull/19162)
|
||||
|
||||
- **[Logfire](../../docs/observability/logfire)**
|
||||
- Add ability to customize Logfire base URL through env var - [PR #19148](https://github.com/BerriAI/litellm/pull/19148)
|
||||
|
||||
- **General Logging**
|
||||
- Enable JSON logging via configuration and add regression test - [PR #19037](https://github.com/BerriAI/litellm/pull/19037)
|
||||
- Fix header forwarding for embeddings endpoint - [PR #18960](https://github.com/BerriAI/litellm/pull/18960)
|
||||
- Preserve llm_provider-* headers in error responses - [PR #19020](https://github.com/BerriAI/litellm/pull/19020)
|
||||
- Fix turn_off_message_logging not redacting request messages in proxy_server_request field - [PR #18897](https://github.com/BerriAI/litellm/pull/18897)
|
||||
|
||||
### Guardrails
|
||||
|
||||
- **[Grayswan](../../docs/proxy/guardrails/grayswan)**
|
||||
- Implement fail-open option (default: True) - [PR #18266](https://github.com/BerriAI/litellm/pull/18266)
|
||||
|
||||
- **[Pangea](../../docs/proxy/guardrails/pangea)**
|
||||
- Respect `default_on` during initialization - [PR #18912](https://github.com/BerriAI/litellm/pull/18912)
|
||||
|
||||
- **[Panw Prisma AIRS](../../docs/proxy/guardrails/panw_prisma_airs)**
|
||||
- Add custom violation message support - [PR #19272](https://github.com/BerriAI/litellm/pull/19272)
|
||||
|
||||
- **General Guardrails**
|
||||
- Fix SerializationIterator error and pass tools to guardrail - [PR #18932](https://github.com/BerriAI/litellm/pull/18932)
|
||||
- Properly handle custom guardrails parameters - [PR #18978](https://github.com/BerriAI/litellm/pull/18978)
|
||||
- Use clean error messages for blocked requests - [PR #19023](https://github.com/BerriAI/litellm/pull/19023)
|
||||
- Guardrail moderation support with responses API - [PR #18957](https://github.com/BerriAI/litellm/pull/18957)
|
||||
- Fix model-level guardrails not taking effect - [PR #18895](https://github.com/BerriAI/litellm/pull/18895)
|
||||
|
||||
---
|
||||
|
||||
## Spend Tracking, Budgets and Rate Limiting
|
||||
|
||||
- **Cost Calculation Fixes**
|
||||
- Include IMAGE token count in cost calculation for Gemini models - [PR #18876](https://github.com/BerriAI/litellm/pull/18876)
|
||||
- Fix negative text_tokens when using cache with images - [PR #18768](https://github.com/BerriAI/litellm/pull/18768)
|
||||
- Fix image tokens spend logging for `/images/generations` - [PR #19009](https://github.com/BerriAI/litellm/pull/19009)
|
||||
- Fix incorrect `prompt_tokens_details` in Gemini Image Generation - [PR #19070](https://github.com/BerriAI/litellm/pull/19070)
|
||||
- Fix case-insensitive model cost map lookup - [PR #18208](https://github.com/BerriAI/litellm/pull/18208)
|
||||
|
||||
- **Pricing Updates**
|
||||
- Correct pricing for `openrouter/openai/gpt-oss-20b` - [PR #18899](https://github.com/BerriAI/litellm/pull/18899)
|
||||
- Add pricing for `azure_ai/claude-opus-4-5` - [PR #19003](https://github.com/BerriAI/litellm/pull/19003)
|
||||
- Update Novita models prices - [PR #19005](https://github.com/BerriAI/litellm/pull/19005)
|
||||
- Fix Azure Grok prices - [PR #19102](https://github.com/BerriAI/litellm/pull/19102)
|
||||
- Fix GCP GLM-4.7 pricing - [PR #19172](https://github.com/BerriAI/litellm/pull/19172)
|
||||
- Sync DeepSeek chat/reasoner to V3.2 pricing - [PR #18884](https://github.com/BerriAI/litellm/pull/18884)
|
||||
- Correct cache_read pricing for gemini-2.5-pro models - [PR #18157](https://github.com/BerriAI/litellm/pull/18157)
|
||||
|
||||
- **Budget & Rate Limiting**
|
||||
- Correct budget limit validation operator (>=) for team members - [PR #19207](https://github.com/BerriAI/litellm/pull/19207)
|
||||
- Fix TPM 25% limiting by ensuring priority queue logic - [PR #19092](https://github.com/BerriAI/litellm/pull/19092)
|
||||
- Cleanup spend logs cron verification, fix, and docs - [PR #19085](https://github.com/BerriAI/litellm/pull/19085)
|
||||
|
||||
---
|
||||
|
||||
## MCP Gateway
|
||||
|
||||
- Prevent duplicate MCP reload scheduler registration - [PR #18934](https://github.com/BerriAI/litellm/pull/18934)
|
||||
- Forward MCP extra headers case-insensitively - [PR #18940](https://github.com/BerriAI/litellm/pull/18940)
|
||||
- Fix MCP REST auth checks - [PR #19051](https://github.com/BerriAI/litellm/pull/19051)
|
||||
- Fix generating two telemetry events in responses - [PR #18938](https://github.com/BerriAI/litellm/pull/18938)
|
||||
- Fix MCP chat completions - [PR #19129](https://github.com/BerriAI/litellm/pull/19129)
|
||||
|
||||
---
|
||||
|
||||
## Performance / Loadbalancing / Reliability improvements
|
||||
|
||||
- **Performance Improvements**
|
||||
- Remove bottleneck causing high CPU usage & overhead under heavy load - [PR #19049](https://github.com/BerriAI/litellm/pull/19049)
|
||||
- Add CI enforcement for O(1) operations in `_get_model_cost_key` to prevent performance regressions - [PR #19052](https://github.com/BerriAI/litellm/pull/19052)
|
||||
- Fix Azure embeddings JSON parsing to prevent connection leaks and ensure proper router cooldown - [PR #19167](https://github.com/BerriAI/litellm/pull/19167)
|
||||
- Do not fallback to token counter if `disable_token_counter` is enabled - [PR #19041](https://github.com/BerriAI/litellm/pull/19041)
|
||||
|
||||
- **Reliability**
|
||||
- Add fallback endpoints support - [PR #19185](https://github.com/BerriAI/litellm/pull/19185)
|
||||
- Fix stream_timeout parameter functionality - [PR #19191](https://github.com/BerriAI/litellm/pull/19191)
|
||||
- Fix model matching priority in configuration - [PR #19012](https://github.com/BerriAI/litellm/pull/19012)
|
||||
- Fix num_retries in litellm_params as per config - [PR #18975](https://github.com/BerriAI/litellm/pull/18975)
|
||||
- Handle exceptions without response parameter - [PR #18919](https://github.com/BerriAI/litellm/pull/18919)
|
||||
|
||||
- **Infrastructure**
|
||||
- Add Custom CA certificates to boto3 clients - [PR #18942](https://github.com/BerriAI/litellm/pull/18942)
|
||||
- Update boto3 to 1.40.15 and aioboto3 to 15.5.0 - [PR #19090](https://github.com/BerriAI/litellm/pull/19090)
|
||||
- Make keepalive_timeout parameter work for Gunicorn - [PR #19087](https://github.com/BerriAI/litellm/pull/19087)
|
||||
|
||||
- **Helm Chart**
|
||||
- Fix mount config.yaml as single file in Helm chart - [PR #19146](https://github.com/BerriAI/litellm/pull/19146)
|
||||
- Sync Helm chart versioning with production standards and Docker versions - [PR #18868](https://github.com/BerriAI/litellm/pull/18868)
|
||||
|
||||
---
|
||||
|
||||
## Database Changes
|
||||
|
||||
### Schema Updates
|
||||
|
||||
| Table | Change Type | Description | PR |
|
||||
| ----- | ----------- | ----------- | -- |
|
||||
| `LiteLLM_ProxyModelTable` | New Columns | Added `created_at` and `updated_at` timestamp fields | [PR #18937](https://github.com/BerriAI/litellm/pull/18937) |
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- Add LiteLLM architecture md doc - [PR #19057](https://github.com/BerriAI/litellm/pull/19057), [PR #19252](https://github.com/BerriAI/litellm/pull/19252)
|
||||
- Add troubleshooting guide - [PR #19096](https://github.com/BerriAI/litellm/pull/19096), [PR #19097](https://github.com/BerriAI/litellm/pull/19097), [PR #19099](https://github.com/BerriAI/litellm/pull/19099)
|
||||
- Add structured issue reporting guides for CPU and memory issues - [PR #19117](https://github.com/BerriAI/litellm/pull/19117)
|
||||
- Add Redis requirement warning for high-traffic deployments - [PR #18892](https://github.com/BerriAI/litellm/pull/18892)
|
||||
- Update load balancing and routing with enable_pre_call_checks - [PR #18888](https://github.com/BerriAI/litellm/pull/18888)
|
||||
- Updated pass_through with guided param - [PR #18886](https://github.com/BerriAI/litellm/pull/18886)
|
||||
- Update message content types link and add content types table - [PR #18209](https://github.com/BerriAI/litellm/pull/18209)
|
||||
- Add Redis initialization with kwargs - [PR #19183](https://github.com/BerriAI/litellm/pull/19183)
|
||||
- Improve documentation for routing LLM calls via SAP Gen AI Hub - [PR #19166](https://github.com/BerriAI/litellm/pull/19166)
|
||||
- Deleted Keys and Teams docs - [PR #19291](https://github.com/BerriAI/litellm/pull/19291)
|
||||
- Claude Code end user tracking guide - [PR #19176](https://github.com/BerriAI/litellm/pull/19176)
|
||||
- Add MCP troubleshooting guide - [PR #19122](https://github.com/BerriAI/litellm/pull/19122)
|
||||
- Add auth message UI documentation - [PR #19063](https://github.com/BerriAI/litellm/pull/19063)
|
||||
- Add guide for mounting custom callbacks in Helm/K8s - [PR #19136](https://github.com/BerriAI/litellm/pull/19136)
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
- Fix Swagger UI path execute error with server_root_path in OpenAPI schema - [PR #18947](https://github.com/BerriAI/litellm/pull/18947)
|
||||
- Normalize OpenAI SDK BaseModel choices/messages to avoid Pydantic serializer warnings - [PR #18972](https://github.com/BerriAI/litellm/pull/18972)
|
||||
- Add contextual gap checks and word-form digits - [PR #18301](https://github.com/BerriAI/litellm/pull/18301)
|
||||
- Clean up orphaned files from repository root - [PR #19150](https://github.com/BerriAI/litellm/pull/19150)
|
||||
- Include proxy/prisma_migration.py in non-root - [PR #18971](https://github.com/BerriAI/litellm/pull/18971)
|
||||
- Update prisma_migration.py - [PR #19083](https://github.com/BerriAI/litellm/pull/19083)
|
||||
|
||||
---
|
||||
|
||||
## New Contributors
|
||||
|
||||
* @yogeshwaran10 made their first contribution in [PR #18898](https://github.com/BerriAI/litellm/pull/18898)
|
||||
* @theonlypal made their first contribution in [PR #18937](https://github.com/BerriAI/litellm/pull/18937)
|
||||
* @jonmagic made their first contribution in [PR #18935](https://github.com/BerriAI/litellm/pull/18935)
|
||||
* @houdataali made their first contribution in [PR #19025](https://github.com/BerriAI/litellm/pull/19025)
|
||||
* @hummat made their first contribution in [PR #18972](https://github.com/BerriAI/litellm/pull/18972)
|
||||
* @berkeyalciin made their first contribution in [PR #18966](https://github.com/BerriAI/litellm/pull/18966)
|
||||
* @MateuszOssGit made their first contribution in [PR #18959](https://github.com/BerriAI/litellm/pull/18959)
|
||||
* @xfan001 made their first contribution in [PR #18947](https://github.com/BerriAI/litellm/pull/18947)
|
||||
* @nulone made their first contribution in [PR #18884](https://github.com/BerriAI/litellm/pull/18884)
|
||||
* @debnil-mercor made their first contribution in [PR #18919](https://github.com/BerriAI/litellm/pull/18919)
|
||||
* @hakhundov made their first contribution in [PR #17420](https://github.com/BerriAI/litellm/pull/17420)
|
||||
* @rohanwinsor made their first contribution in [PR #19078](https://github.com/BerriAI/litellm/pull/19078)
|
||||
* @pgolm made their first contribution in [PR #19020](https://github.com/BerriAI/litellm/pull/19020)
|
||||
* @vikigenius made their first contribution in [PR #19148](https://github.com/BerriAI/litellm/pull/19148)
|
||||
* @burnerburnerburnerman made their first contribution in [PR #19090](https://github.com/BerriAI/litellm/pull/19090)
|
||||
* @yfge made their first contribution in [PR #19076](https://github.com/BerriAI/litellm/pull/19076)
|
||||
* @danielnyari-seon made their first contribution in [PR #19083](https://github.com/BerriAI/litellm/pull/19083)
|
||||
* @guilherme-segantini made their first contribution in [PR #19166](https://github.com/BerriAI/litellm/pull/19166)
|
||||
* @jgreek made their first contribution in [PR #19147](https://github.com/BerriAI/litellm/pull/19147)
|
||||
* @anand-kamble made their first contribution in [PR #19193](https://github.com/BerriAI/litellm/pull/19193)
|
||||
* @neubig made their first contribution in [PR #19162](https://github.com/BerriAI/litellm/pull/19162)
|
||||
|
||||
---
|
||||
|
||||
## Full Changelog
|
||||
|
||||
**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.80.15.rc.1...v1.81.0.rc.1)**
|
||||
|
|
@ -122,6 +122,7 @@ const sidebars = {
|
|||
items: [
|
||||
"tutorials/claude_responses_api",
|
||||
"tutorials/claude_code_customer_tracking",
|
||||
"tutorials/claude_code_websearch",
|
||||
"tutorials/claude_mcp",
|
||||
"tutorials/claude_non_anthropic_models",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -329,6 +329,11 @@ ANTHROPIC_WEB_SEARCH_TOOL_MAX_USES = {
|
|||
"medium": 5,
|
||||
"high": 10,
|
||||
}
|
||||
|
||||
# LiteLLM standard web search tool name
|
||||
# Used for web search interception across providers
|
||||
LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"
|
||||
|
||||
DEFAULT_IMAGE_ENDPOINT_MODEL = "dall-e-2"
|
||||
DEFAULT_VIDEO_ENDPOINT_MODEL = "sora-2"
|
||||
|
||||
|
|
|
|||
|
|
@ -143,6 +143,34 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
|
|||
async def async_log_pre_api_call(self, model, messages, kwargs):
|
||||
pass
|
||||
|
||||
async def async_pre_request_hook(
|
||||
self, model: str, messages: List, kwargs: Dict
|
||||
) -> Optional[Dict]:
|
||||
"""
|
||||
Hook called before making the API request to allow modifying request parameters.
|
||||
|
||||
This is specifically designed for modifying the request before it's sent to the provider.
|
||||
Unlike async_log_pre_api_call (which is for logging), this hook is meant for transformations.
|
||||
|
||||
Args:
|
||||
model: The model name
|
||||
messages: The messages list
|
||||
kwargs: The request parameters (tools, stream, temperature, etc.)
|
||||
|
||||
Returns:
|
||||
Optional[Dict]: Modified kwargs to use for the request, or None if no modifications
|
||||
|
||||
Example:
|
||||
```python
|
||||
async def async_pre_request_hook(self, model, messages, kwargs):
|
||||
# Convert native tools to standard format
|
||||
if kwargs.get("tools"):
|
||||
kwargs["tools"] = convert_tools(kwargs["tools"])
|
||||
return kwargs
|
||||
```
|
||||
"""
|
||||
pass
|
||||
|
||||
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
|
||||
pass
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,12 @@ from typing import (
|
|||
import litellm
|
||||
from litellm._logging import print_verbose, verbose_logger
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.proxy._types import LiteLLM_TeamTable, LiteLLM_UserTable, UserAPIKeyAuth
|
||||
from litellm.proxy._types import (
|
||||
LiteLLM_DeletedVerificationToken,
|
||||
LiteLLM_TeamTable,
|
||||
LiteLLM_UserTable,
|
||||
UserAPIKeyAuth,
|
||||
)
|
||||
from litellm.types.integrations.prometheus import *
|
||||
from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name
|
||||
from litellm.types.utils import StandardLoggingPayload
|
||||
|
|
@ -2153,7 +2158,7 @@ class PrometheusLogger(CustomLogger):
|
|||
self,
|
||||
data_fetch_function: Callable[..., Awaitable[Tuple[List[Any], Optional[int]]]],
|
||||
set_metrics_function: Callable[[List[Any]], Awaitable[None]],
|
||||
data_type: Literal["teams", "keys"],
|
||||
data_type: Literal["teams", "keys", "users"],
|
||||
):
|
||||
"""
|
||||
Generic method to initialize budget metrics for teams or API keys.
|
||||
|
|
@ -2245,7 +2250,7 @@ class PrometheusLogger(CustomLogger):
|
|||
|
||||
async def fetch_keys(
|
||||
page_size: int, page: int
|
||||
) -> Tuple[List[Union[str, UserAPIKeyAuth]], Optional[int]]:
|
||||
) -> Tuple[List[Union[str, UserAPIKeyAuth, LiteLLM_DeletedVerificationToken]], Optional[int]]:
|
||||
key_list_response = await _list_key_helper(
|
||||
prisma_client=prisma_client,
|
||||
page=page,
|
||||
|
|
|
|||
|
|
@ -7,6 +7,98 @@ Server-side WebSearch tool execution for models that don't natively support it (
|
|||
User makes **ONE** `litellm.messages.acreate()` call → Gets final answer with search results.
|
||||
The agentic loop happens transparently on the server.
|
||||
|
||||
## LiteLLM Standard Web Search Tool
|
||||
|
||||
LiteLLM defines a standard web search tool format (`litellm_web_search`) that all native provider tools are converted to. This enables consistent interception across providers.
|
||||
|
||||
**Standard Tool Definition** (defined in `tools.py`):
|
||||
```python
|
||||
{
|
||||
"name": "litellm_web_search",
|
||||
"description": "Search the web for information...",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "The search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Tool Name Constant**: `LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"` (defined in `litellm/constants.py`)
|
||||
|
||||
### Supported Tool Formats
|
||||
|
||||
The interception system automatically detects and handles:
|
||||
|
||||
| Tool Format | Example | Provider | Detection Method | Future-Proof |
|
||||
|-------------|---------|----------|------------------|-------------|
|
||||
| **LiteLLM Standard** | `name="litellm_web_search"` | Any | Direct name match | N/A |
|
||||
| **Anthropic Native** | `type="web_search_20250305"` | Bedrock, Claude API | Type prefix: `startswith("web_search_")` | ✅ Yes (web_search_2026, etc.) |
|
||||
| **Claude Code CLI** | `name="web_search"`, `type="web_search_20250305"` | Claude Code | Name + type check | ✅ Yes (version-agnostic) |
|
||||
| **Legacy** | `name="WebSearch"` | Custom | Name match | N/A (backwards compat) |
|
||||
|
||||
**Future Compatibility**: The `startswith("web_search_")` check in `tools.py` automatically supports future Anthropic web search versions.
|
||||
|
||||
### Claude Code CLI Integration
|
||||
|
||||
Claude Code (Anthropic's official CLI) sends web search requests using Anthropic's native tool format:
|
||||
|
||||
```python
|
||||
{
|
||||
"type": "web_search_20250305",
|
||||
"name": "web_search",
|
||||
"max_uses": 8
|
||||
}
|
||||
```
|
||||
|
||||
**What Happens:**
|
||||
1. Claude Code sends native `web_search_20250305` tool to LiteLLM proxy
|
||||
2. LiteLLM intercepts and converts to `litellm_web_search` standard format
|
||||
3. Bedrock receives converted tool (NOT native format)
|
||||
4. Model returns `tool_use` block for `litellm_web_search` (not `server_tool_use`)
|
||||
5. LiteLLM's agentic loop intercepts the `tool_use`
|
||||
6. Executes `litellm.asearch()` using configured provider (Perplexity, Tavily, etc.)
|
||||
7. Returns final answer to Claude Code user
|
||||
|
||||
**Without Interception**: Bedrock would receive native tool → try to execute natively → return `web_search_tool_result_error` with `invalid_tool_input`
|
||||
|
||||
**With Interception**: LiteLLM converts → Bedrock returns tool_use → LiteLLM executes search → Returns final answer ✅
|
||||
|
||||
### Native Tool Conversion
|
||||
|
||||
Native tools are converted to LiteLLM standard format **before** sending to the provider:
|
||||
|
||||
1. **Conversion Point** (`litellm/llms/anthropic/experimental_pass_through/messages/handler.py`):
|
||||
- In `anthropic_messages()` function (lines 60-127)
|
||||
- Runs BEFORE the API request is made
|
||||
- Detects native web search tools using `is_web_search_tool()`
|
||||
- Converts to `litellm_web_search` format using `get_litellm_web_search_tool()`
|
||||
- Prevents provider from executing search natively (avoids `web_search_tool_result_error`)
|
||||
|
||||
2. **Response Detection** (`transformation.py`):
|
||||
- Detects `tool_use` blocks with any web search tool name
|
||||
- Handles: `litellm_web_search`, `WebSearch`, `web_search`
|
||||
- Extracts search queries for execution
|
||||
|
||||
**Example Conversion**:
|
||||
```python
|
||||
# Input (Claude Code's native tool)
|
||||
{
|
||||
"type": "web_search_20250305",
|
||||
"name": "web_search",
|
||||
"max_uses": 8
|
||||
}
|
||||
|
||||
# Output (LiteLLM standard)
|
||||
{
|
||||
"name": "litellm_web_search",
|
||||
"description": "Search the web for information...",
|
||||
"input_schema": {...}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Request Flow
|
||||
|
|
@ -63,6 +155,9 @@ sequenceDiagram
|
|||
| Component | File | Purpose |
|
||||
|-----------|------|---------|
|
||||
| **WebSearchInterceptionLogger** | `handler.py` | CustomLogger that implements agentic loop hooks |
|
||||
| **Tool Standardization** | `tools.py` | Standard tool definition, detection, and utilities |
|
||||
| **Tool Name Constant** | `constants.py` | `LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"` |
|
||||
| **Tool Conversion** | `anthropic/.../ handler.py` | Converts native tools to LiteLLM standard before API call |
|
||||
| **Transformation Logic** | `transformation.py` | Detect tool_use, build tool_result messages, format search responses |
|
||||
| **Agentic Loop Hooks** | `integrations/custom_logger.py` | Base hooks: `async_should_run_agentic_loop()`, `async_run_agentic_loop()` |
|
||||
| **Hook Orchestration** | `llms/custom_httpx/llm_http_handler.py` | `_call_agentic_completion_hooks()` - calls hooks after response |
|
||||
|
|
@ -74,7 +169,10 @@ sequenceDiagram
|
|||
## Configuration
|
||||
|
||||
```python
|
||||
from litellm.integrations.websearch_interception import WebSearchInterceptionLogger
|
||||
from litellm.integrations.websearch_interception import (
|
||||
WebSearchInterceptionLogger,
|
||||
get_litellm_web_search_tool,
|
||||
)
|
||||
from litellm.types.utils import LlmProviders
|
||||
|
||||
# Enable for Bedrock with specific search tool
|
||||
|
|
@ -85,13 +183,25 @@ litellm.callbacks = [
|
|||
)
|
||||
]
|
||||
|
||||
# Make request (streaming or non-streaming both work)
|
||||
# Make request with LiteLLM standard tool (recommended)
|
||||
response = await litellm.messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0",
|
||||
model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
messages=[{"role": "user", "content": "What is LiteLLM?"}],
|
||||
tools=[{"name": "WebSearch", ...}],
|
||||
tools=[get_litellm_web_search_tool()], # LiteLLM standard
|
||||
max_tokens=1024,
|
||||
stream=True # Auto-converted to non-streaming
|
||||
)
|
||||
|
||||
# OR send native tools - they're auto-converted to LiteLLM standard
|
||||
response = await litellm.messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
messages=[{"role": "user", "content": "What is LiteLLM?"}],
|
||||
tools=[{
|
||||
"type": "web_search_20250305", # Native Anthropic format
|
||||
"name": "web_search",
|
||||
"max_uses": 8
|
||||
}],
|
||||
max_tokens=1024,
|
||||
stream=True # Streaming is automatically converted to non-streaming for WebSearch
|
||||
)
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -8,5 +8,13 @@ support server-side tool calling (e.g., Bedrock/Claude).
|
|||
from litellm.integrations.websearch_interception.handler import (
|
||||
WebSearchInterceptionLogger,
|
||||
)
|
||||
from litellm.integrations.websearch_interception.tools import (
|
||||
get_litellm_web_search_tool,
|
||||
is_web_search_tool,
|
||||
)
|
||||
|
||||
__all__ = ["WebSearchInterceptionLogger"]
|
||||
__all__ = [
|
||||
"WebSearchInterceptionLogger",
|
||||
"get_litellm_web_search_tool",
|
||||
"is_web_search_tool",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -12,7 +12,12 @@ from typing import Any, Dict, List, Optional, Tuple, Union, cast
|
|||
import litellm
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.anthropic_interface import messages as anthropic_messages
|
||||
from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.integrations.websearch_interception.tools import (
|
||||
get_litellm_web_search_tool,
|
||||
is_web_search_tool,
|
||||
)
|
||||
from litellm.integrations.websearch_interception.transformation import (
|
||||
WebSearchTransformation,
|
||||
)
|
||||
|
|
@ -57,6 +62,55 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
for p in enabled_providers
|
||||
]
|
||||
self.search_tool_name = search_tool_name
|
||||
self._request_has_websearch = False # Track if current request has web search
|
||||
|
||||
async def async_pre_call_deployment_hook(
|
||||
self, kwargs: Dict[str, Any], call_type: Optional[Any]
|
||||
) -> Optional[dict]:
|
||||
"""
|
||||
Pre-call hook to convert native Anthropic web_search tools to regular tools.
|
||||
|
||||
This prevents Bedrock from trying to execute web search server-side (which fails).
|
||||
Instead, we convert it to a regular tool so the model returns tool_use blocks
|
||||
that we can intercept and execute ourselves.
|
||||
"""
|
||||
# Check if this is for an enabled provider
|
||||
custom_llm_provider = kwargs.get("litellm_params", {}).get("custom_llm_provider", "")
|
||||
if custom_llm_provider not in self.enabled_providers:
|
||||
return None
|
||||
|
||||
# Check if request has tools with native web_search
|
||||
tools = kwargs.get("tools")
|
||||
if not tools:
|
||||
return None
|
||||
|
||||
# Check if any tool is a web search tool (native or already LiteLLM standard)
|
||||
has_websearch = any(is_web_search_tool(t) for t in tools)
|
||||
|
||||
if not has_websearch:
|
||||
return None
|
||||
|
||||
verbose_logger.debug(
|
||||
"WebSearchInterception: Converting native web_search tools to LiteLLM standard"
|
||||
)
|
||||
|
||||
# Convert native/custom web_search tools to LiteLLM standard
|
||||
converted_tools = []
|
||||
for tool in tools:
|
||||
if is_web_search_tool(tool):
|
||||
# Convert to LiteLLM standard web search tool
|
||||
converted_tool = get_litellm_web_search_tool()
|
||||
converted_tools.append(converted_tool)
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Converted {tool.get('name', 'unknown')} "
|
||||
f"(type={tool.get('type', 'none')}) to {LITELLM_WEB_SEARCH_TOOL_NAME}"
|
||||
)
|
||||
else:
|
||||
# Keep other tools as-is
|
||||
converted_tools.append(tool)
|
||||
|
||||
# Return modified kwargs with converted tools
|
||||
return {"tools": converted_tools}
|
||||
|
||||
@classmethod
|
||||
def from_config_yaml(
|
||||
|
|
@ -104,6 +158,83 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
search_tool_name=search_tool_name,
|
||||
)
|
||||
|
||||
async def async_pre_request_hook(
|
||||
self, model: str, messages: List[Dict], kwargs: Dict
|
||||
) -> Optional[Dict]:
|
||||
"""
|
||||
Pre-request hook to convert native web search tools to LiteLLM standard.
|
||||
|
||||
This hook is called before the API request is made, allowing us to:
|
||||
1. Detect native web search tools (web_search_20250305, etc.)
|
||||
2. Convert them to LiteLLM standard format (litellm_web_search)
|
||||
3. Convert stream=True to stream=False for interception
|
||||
|
||||
This prevents providers like Bedrock from trying to execute web search
|
||||
natively (which fails), and ensures our agentic loop can intercept tool_use.
|
||||
|
||||
Returns:
|
||||
Modified kwargs dict with converted tools, or None if no modifications needed
|
||||
"""
|
||||
# Check if this request is for an enabled provider
|
||||
custom_llm_provider = kwargs.get("litellm_params", {}).get(
|
||||
"custom_llm_provider", ""
|
||||
)
|
||||
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Pre-request hook called"
|
||||
f" - custom_llm_provider={custom_llm_provider}"
|
||||
f" - enabled_providers={self.enabled_providers}"
|
||||
)
|
||||
|
||||
if custom_llm_provider not in self.enabled_providers:
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Skipping - provider {custom_llm_provider} not in {self.enabled_providers}"
|
||||
)
|
||||
return None
|
||||
|
||||
# Check if request has tools
|
||||
tools = kwargs.get("tools")
|
||||
if not tools:
|
||||
return None
|
||||
|
||||
# Check if any tool is a web search tool
|
||||
has_websearch = any(is_web_search_tool(t) for t in tools)
|
||||
if not has_websearch:
|
||||
return None
|
||||
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Pre-request hook triggered for provider={custom_llm_provider}"
|
||||
)
|
||||
|
||||
# Convert native web search tools to LiteLLM standard
|
||||
converted_tools = []
|
||||
for tool in tools:
|
||||
if is_web_search_tool(tool):
|
||||
standard_tool = get_litellm_web_search_tool()
|
||||
converted_tools.append(standard_tool)
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Converted {tool.get('name', 'unknown')} "
|
||||
f"(type={tool.get('type', 'none')}) to {LITELLM_WEB_SEARCH_TOOL_NAME}"
|
||||
)
|
||||
else:
|
||||
converted_tools.append(tool)
|
||||
|
||||
# Update kwargs with converted tools
|
||||
kwargs["tools"] = converted_tools
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Tools after conversion: {[t.get('name') for t in converted_tools]}"
|
||||
)
|
||||
|
||||
# Convert stream=True to stream=False for WebSearch interception
|
||||
if kwargs.get("stream"):
|
||||
verbose_logger.debug(
|
||||
"WebSearchInterception: Converting stream=True to stream=False"
|
||||
)
|
||||
kwargs["stream"] = False
|
||||
kwargs["_websearch_interception_converted_stream"] = True
|
||||
|
||||
return kwargs
|
||||
|
||||
async def async_should_run_agentic_loop(
|
||||
self,
|
||||
response: Any,
|
||||
|
|
@ -128,11 +259,11 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
)
|
||||
return False, {}
|
||||
|
||||
# Check if tools include WebSearch
|
||||
has_websearch_tool = any(t.get("name") == "WebSearch" for t in (tools or []))
|
||||
# Check if tools include any web search tool (LiteLLM standard or native)
|
||||
has_websearch_tool = any(is_web_search_tool(t) for t in (tools or []))
|
||||
if not has_websearch_tool:
|
||||
verbose_logger.debug(
|
||||
"WebSearchInterception: No WebSearch tool in request"
|
||||
"WebSearchInterception: No web search tool in request"
|
||||
)
|
||||
return False, {}
|
||||
|
||||
|
|
|
|||
95
litellm/integrations/websearch_interception/tools.py
Normal file
95
litellm/integrations/websearch_interception/tools.py
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
"""
|
||||
LiteLLM Web Search Tool Definition
|
||||
|
||||
This module defines the standard web search tool used across LiteLLM.
|
||||
Native provider tools (like Anthropic's web_search_20250305) are converted
|
||||
to this format for consistent interception and execution.
|
||||
"""
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
|
||||
|
||||
|
||||
def get_litellm_web_search_tool() -> Dict[str, Any]:
|
||||
"""
|
||||
Get the standard LiteLLM web search tool definition.
|
||||
|
||||
This is the canonical tool definition that all native web search tools
|
||||
(like Anthropic's web_search_20250305, Claude Code's web_search, etc.)
|
||||
are converted to for interception.
|
||||
|
||||
Returns:
|
||||
Dict containing the Anthropic-style tool definition with:
|
||||
- name: Tool name
|
||||
- description: What the tool does
|
||||
- input_schema: JSON schema for tool parameters
|
||||
|
||||
Example:
|
||||
>>> tool = get_litellm_web_search_tool()
|
||||
>>> tool['name']
|
||||
'litellm_web_search'
|
||||
"""
|
||||
return {
|
||||
"name": LITELLM_WEB_SEARCH_TOOL_NAME,
|
||||
"description": (
|
||||
"Search the web for information. Use this when you need current "
|
||||
"information or answers to questions that require up-to-date data."
|
||||
),
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The search query to execute"
|
||||
}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def is_web_search_tool(tool: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Check if a tool is a web search tool (native or LiteLLM standard).
|
||||
|
||||
Detects:
|
||||
- LiteLLM standard: name == "litellm_web_search"
|
||||
- Anthropic native: type starts with "web_search_" (e.g., "web_search_20250305")
|
||||
- Claude Code: name == "web_search" with a type field
|
||||
- Custom: name == "WebSearch" (legacy format)
|
||||
|
||||
Args:
|
||||
tool: Tool dictionary to check
|
||||
|
||||
Returns:
|
||||
True if tool is a web search tool
|
||||
|
||||
Example:
|
||||
>>> is_web_search_tool({"name": "litellm_web_search"})
|
||||
True
|
||||
>>> is_web_search_tool({"type": "web_search_20250305", "name": "web_search"})
|
||||
True
|
||||
>>> is_web_search_tool({"name": "calculator"})
|
||||
False
|
||||
"""
|
||||
tool_name = tool.get("name", "")
|
||||
tool_type = tool.get("type", "")
|
||||
|
||||
# Check for LiteLLM standard tool
|
||||
if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME:
|
||||
return True
|
||||
|
||||
# Check for native Anthropic web_search_* types
|
||||
if tool_type.startswith("web_search_"):
|
||||
return True
|
||||
|
||||
# Check for Claude Code's web_search with a type field
|
||||
if tool_name == "web_search" and tool_type:
|
||||
return True
|
||||
|
||||
# Check for legacy WebSearch format
|
||||
if tool_name == "WebSearch":
|
||||
return True
|
||||
|
||||
return False
|
||||
|
|
@ -7,6 +7,7 @@ Transforms between Anthropic tool_use format and LiteLLM search format.
|
|||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
|
||||
from litellm.llms.base_llm.search.transformation import SearchResponse
|
||||
|
||||
|
||||
|
|
@ -94,17 +95,21 @@ class WebSearchTransformation:
|
|||
block_id = getattr(block, "id", None)
|
||||
block_input = getattr(block, "input", {})
|
||||
|
||||
if block_type == "tool_use" and block_name == "WebSearch":
|
||||
# Check for LiteLLM standard or legacy web search tools
|
||||
# Handles: litellm_web_search, WebSearch, web_search
|
||||
if block_type == "tool_use" and block_name in (
|
||||
LITELLM_WEB_SEARCH_TOOL_NAME, "WebSearch", "web_search"
|
||||
):
|
||||
# Convert to dict for easier handling
|
||||
tool_call = {
|
||||
"id": block_id,
|
||||
"type": "tool_use",
|
||||
"name": "WebSearch",
|
||||
"name": block_name, # Preserve original name
|
||||
"input": block_input,
|
||||
}
|
||||
tool_calls.append(tool_call)
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Found WebSearch tool_use with id={tool_call['id']}"
|
||||
f"WebSearchInterception: Found {block_name} tool_use with id={tool_call['id']}"
|
||||
)
|
||||
|
||||
return len(tool_calls) > 0, tool_calls
|
||||
|
|
|
|||
|
|
@ -0,0 +1,246 @@
|
|||
"""
|
||||
Fake Streaming Iterator for Anthropic Messages
|
||||
|
||||
This module provides a fake streaming iterator that converts non-streaming
|
||||
Anthropic Messages responses into proper streaming format.
|
||||
|
||||
Used when WebSearch interception converts stream=True to stream=False but
|
||||
the LLM doesn't make a tool call, and we need to return a stream to the user.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Any, Dict, List, cast
|
||||
|
||||
from litellm.types.llms.anthropic_messages.anthropic_response import (
|
||||
AnthropicMessagesResponse,
|
||||
)
|
||||
|
||||
|
||||
class FakeAnthropicMessagesStreamIterator:
|
||||
"""
|
||||
Fake streaming iterator for Anthropic Messages responses.
|
||||
|
||||
Used when we need to convert a non-streaming response to a streaming format,
|
||||
such as when WebSearch interception converts stream=True to stream=False but
|
||||
the LLM doesn't make a tool call.
|
||||
|
||||
This creates a proper Anthropic-style streaming response with multiple events:
|
||||
- message_start
|
||||
- content_block_start (for each content block)
|
||||
- content_block_delta (for text content, chunked)
|
||||
- content_block_stop
|
||||
- message_delta (for usage)
|
||||
- message_stop
|
||||
"""
|
||||
|
||||
def __init__(self, response: AnthropicMessagesResponse):
|
||||
self.response = response
|
||||
self.chunks = self._create_streaming_chunks()
|
||||
self.current_index = 0
|
||||
|
||||
def _create_streaming_chunks(self) -> List[bytes]:
|
||||
"""Convert the non-streaming response to streaming chunks"""
|
||||
chunks = []
|
||||
|
||||
# Cast response to dict for easier access
|
||||
response_dict = cast(Dict[str, Any], self.response)
|
||||
|
||||
# 1. message_start event
|
||||
usage = response_dict.get("usage", {})
|
||||
message_start = {
|
||||
"type": "message_start",
|
||||
"message": {
|
||||
"id": response_dict.get("id"),
|
||||
"type": "message",
|
||||
"role": response_dict.get("role", "assistant"),
|
||||
"model": response_dict.get("model"),
|
||||
"content": [],
|
||||
"stop_reason": None,
|
||||
"stop_sequence": None,
|
||||
"usage": {
|
||||
"input_tokens": usage.get("input_tokens", 0) if usage else 0,
|
||||
"output_tokens": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: message_start\ndata: {json.dumps(message_start)}\n\n".encode())
|
||||
|
||||
# 2-4. For each content block, send start/delta/stop events
|
||||
content_blocks = response_dict.get("content", [])
|
||||
if content_blocks:
|
||||
for index, block in enumerate(content_blocks):
|
||||
# Cast block to dict for easier access
|
||||
block_dict = cast(Dict[str, Any], block)
|
||||
block_type = block_dict.get("type")
|
||||
|
||||
if block_type == "text":
|
||||
# content_block_start
|
||||
content_block_start = {
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": {
|
||||
"type": "text",
|
||||
"text": ""
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
|
||||
|
||||
# content_block_delta (send full text as one delta for simplicity)
|
||||
text = block_dict.get("text", "")
|
||||
content_block_delta = {
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": {
|
||||
"type": "text_delta",
|
||||
"text": text
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode())
|
||||
|
||||
# content_block_stop
|
||||
content_block_stop = {
|
||||
"type": "content_block_stop",
|
||||
"index": index
|
||||
}
|
||||
chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
|
||||
|
||||
elif block_type == "thinking":
|
||||
# content_block_start for thinking
|
||||
content_block_start = {
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": {
|
||||
"type": "thinking",
|
||||
"thinking": "",
|
||||
"signature": ""
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
|
||||
|
||||
# content_block_delta for thinking text
|
||||
thinking_text = block_dict.get("thinking", "")
|
||||
if thinking_text:
|
||||
content_block_delta = {
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": {
|
||||
"type": "thinking_delta",
|
||||
"thinking": thinking_text
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode())
|
||||
|
||||
# content_block_delta for signature (if present)
|
||||
signature = block_dict.get("signature", "")
|
||||
if signature:
|
||||
signature_delta = {
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": {
|
||||
"type": "signature_delta",
|
||||
"signature": signature
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_delta\ndata: {json.dumps(signature_delta)}\n\n".encode())
|
||||
|
||||
# content_block_stop
|
||||
content_block_stop = {
|
||||
"type": "content_block_stop",
|
||||
"index": index
|
||||
}
|
||||
chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
|
||||
|
||||
elif block_type == "redacted_thinking":
|
||||
# content_block_start for redacted_thinking
|
||||
content_block_start = {
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": {
|
||||
"type": "redacted_thinking"
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
|
||||
|
||||
# content_block_stop (no delta for redacted thinking)
|
||||
content_block_stop = {
|
||||
"type": "content_block_stop",
|
||||
"index": index
|
||||
}
|
||||
chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
|
||||
|
||||
elif block_type == "tool_use":
|
||||
# content_block_start
|
||||
content_block_start = {
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": {
|
||||
"type": "tool_use",
|
||||
"id": block_dict.get("id"),
|
||||
"name": block_dict.get("name"),
|
||||
"input": {}
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode())
|
||||
|
||||
# content_block_delta (send input as JSON delta)
|
||||
input_data = block_dict.get("input", {})
|
||||
content_block_delta = {
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": {
|
||||
"type": "input_json_delta",
|
||||
"partial_json": json.dumps(input_data)
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode())
|
||||
|
||||
# content_block_stop
|
||||
content_block_stop = {
|
||||
"type": "content_block_stop",
|
||||
"index": index
|
||||
}
|
||||
chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode())
|
||||
|
||||
# 5. message_delta event (with final usage and stop_reason)
|
||||
message_delta = {
|
||||
"type": "message_delta",
|
||||
"delta": {
|
||||
"stop_reason": response_dict.get("stop_reason"),
|
||||
"stop_sequence": response_dict.get("stop_sequence")
|
||||
},
|
||||
"usage": {
|
||||
"output_tokens": usage.get("output_tokens", 0) if usage else 0
|
||||
}
|
||||
}
|
||||
chunks.append(f"event: message_delta\ndata: {json.dumps(message_delta)}\n\n".encode())
|
||||
|
||||
# 6. message_stop event
|
||||
message_stop = {
|
||||
"type": "message_stop",
|
||||
"usage": usage if usage else {}
|
||||
}
|
||||
chunks.append(f"event: message_stop\ndata: {json.dumps(message_stop)}\n\n".encode())
|
||||
|
||||
return chunks
|
||||
|
||||
def __aiter__(self):
|
||||
return self
|
||||
|
||||
async def __anext__(self):
|
||||
if self.current_index >= len(self.chunks):
|
||||
raise StopAsyncIteration
|
||||
|
||||
chunk = self.chunks[self.current_index]
|
||||
self.current_index += 1
|
||||
return chunk
|
||||
|
||||
def __iter__(self):
|
||||
return self
|
||||
|
||||
def __next__(self):
|
||||
if self.current_index >= len(self.chunks):
|
||||
raise StopIteration
|
||||
|
||||
chunk = self.chunks[self.current_index]
|
||||
self.current_index += 1
|
||||
return chunk
|
||||
|
|
@ -33,6 +33,70 @@ base_llm_http_handler = BaseLLMHTTPHandler()
|
|||
#################################################
|
||||
|
||||
|
||||
async def _execute_pre_request_hooks(
|
||||
model: str,
|
||||
messages: List[Dict],
|
||||
tools: Optional[List[Dict]],
|
||||
stream: Optional[bool],
|
||||
custom_llm_provider: Optional[str],
|
||||
**kwargs,
|
||||
) -> Dict:
|
||||
"""
|
||||
Execute pre-request hooks from CustomLogger callbacks.
|
||||
|
||||
Allows CustomLoggers to modify request parameters before the API call.
|
||||
Used for WebSearch tool conversion, stream modification, etc.
|
||||
|
||||
Args:
|
||||
model: Model name
|
||||
messages: List of messages
|
||||
tools: Optional tools list
|
||||
stream: Optional stream flag
|
||||
custom_llm_provider: Provider name (if not set, will be extracted from model)
|
||||
**kwargs: Additional request parameters
|
||||
|
||||
Returns:
|
||||
Dict containing all (potentially modified) request parameters including tools, stream
|
||||
"""
|
||||
# If custom_llm_provider not provided, extract from model
|
||||
if not custom_llm_provider:
|
||||
try:
|
||||
_, custom_llm_provider, _, _ = litellm.get_llm_provider(model=model)
|
||||
except Exception:
|
||||
# If extraction fails, continue without provider
|
||||
pass
|
||||
|
||||
# Build complete request kwargs dict
|
||||
request_kwargs = {
|
||||
"tools": tools,
|
||||
"stream": stream,
|
||||
"litellm_params": {
|
||||
"custom_llm_provider": custom_llm_provider,
|
||||
},
|
||||
**kwargs,
|
||||
}
|
||||
|
||||
if not litellm.callbacks:
|
||||
return request_kwargs
|
||||
|
||||
from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
|
||||
|
||||
for callback in litellm.callbacks:
|
||||
if not isinstance(callback, _CustomLogger):
|
||||
continue
|
||||
|
||||
# Call the pre-request hook
|
||||
modified_kwargs = await callback.async_pre_request_hook(
|
||||
model, messages, request_kwargs
|
||||
)
|
||||
|
||||
# If hook returned modified kwargs, use them
|
||||
if modified_kwargs is not None:
|
||||
request_kwargs = modified_kwargs
|
||||
|
||||
return request_kwargs
|
||||
|
||||
|
||||
@client
|
||||
async def anthropic_messages(
|
||||
max_tokens: int,
|
||||
|
|
@ -57,39 +121,24 @@ async def anthropic_messages(
|
|||
"""
|
||||
Async: Make llm api request in Anthropic /messages API spec
|
||||
"""
|
||||
# WebSearch Interception: Convert stream=True to stream=False if WebSearch interception is enabled
|
||||
# This allows transparent server-side agentic loop execution for streaming requests
|
||||
if stream and tools and any(t.get("name") == "WebSearch" for t in tools):
|
||||
# Extract provider using litellm's helper function
|
||||
try:
|
||||
_, provider, _, _ = litellm.get_llm_provider(
|
||||
model=model,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
)
|
||||
except Exception:
|
||||
# Fallback to simple split if helper fails
|
||||
provider = model.split("/")[0] if "/" in model else ""
|
||||
# Execute pre-request hooks to allow CustomLoggers to modify request
|
||||
request_kwargs = await _execute_pre_request_hooks(
|
||||
model=model,
|
||||
messages=messages,
|
||||
tools=tools,
|
||||
stream=stream,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
# Check if WebSearch interception is enabled in callbacks
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.integrations.websearch_interception import (
|
||||
WebSearchInterceptionLogger,
|
||||
)
|
||||
if litellm.callbacks:
|
||||
for callback in litellm.callbacks:
|
||||
if isinstance(callback, WebSearchInterceptionLogger):
|
||||
# Check if provider is enabled for interception
|
||||
if provider in callback.enabled_providers:
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Converting stream=True to stream=False for WebSearch interception "
|
||||
f"(provider={provider})"
|
||||
)
|
||||
stream = False
|
||||
break
|
||||
# Extract modified parameters
|
||||
tools = request_kwargs.pop("tools", tools)
|
||||
stream = request_kwargs.pop("stream", stream)
|
||||
# Remove litellm_params from kwargs (only needed for hooks)
|
||||
request_kwargs.pop("litellm_params", None)
|
||||
# Merge back any other modifications
|
||||
kwargs.update(request_kwargs)
|
||||
|
||||
local_vars = locals()
|
||||
loop = asyncio.get_event_loop()
|
||||
kwargs["is_async"] = True
|
||||
|
||||
|
|
@ -206,6 +255,11 @@ def anthropic_messages_handler(
|
|||
"model": original_model,
|
||||
"custom_llm_provider": custom_llm_provider,
|
||||
}
|
||||
|
||||
# Check if stream was converted for WebSearch interception
|
||||
# This is set in the async wrapper above when stream=True is converted to stream=False
|
||||
if kwargs.get("_websearch_interception_converted_stream", False):
|
||||
litellm_logging_obj.model_call_details["websearch_interception_converted_stream"] = True
|
||||
|
||||
if litellm_params.mock_response and isinstance(litellm_params.mock_response, str):
|
||||
|
||||
|
|
|
|||
|
|
@ -4418,6 +4418,41 @@ class BaseLLMHTTPHandler:
|
|||
f"LiteLLM.AgenticHookError: Exception in agentic completion hooks: {str(e)}"
|
||||
)
|
||||
|
||||
# Check if we need to convert response to fake stream
|
||||
# This happens when:
|
||||
# 1. Stream was originally True but converted to False for WebSearch interception
|
||||
# 2. No agentic loop ran (LLM didn't use the tool)
|
||||
# 3. We have a non-streaming response that needs to be converted to streaming
|
||||
websearch_converted_stream = (
|
||||
logging_obj.model_call_details.get("websearch_interception_converted_stream", False)
|
||||
if logging_obj is not None
|
||||
else False
|
||||
)
|
||||
|
||||
if websearch_converted_stream:
|
||||
from typing import cast
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.fake_stream_iterator import (
|
||||
FakeAnthropicMessagesStreamIterator,
|
||||
)
|
||||
from litellm.types.llms.anthropic_messages.anthropic_response import (
|
||||
AnthropicMessagesResponse,
|
||||
)
|
||||
|
||||
verbose_logger.debug(
|
||||
"WebSearchInterception: No tool call made, converting non-streaming response to fake stream"
|
||||
)
|
||||
|
||||
# Convert the non-streaming response to a fake stream
|
||||
# The response should be an AnthropicMessagesResponse (dict)
|
||||
if isinstance(response, dict):
|
||||
# Create a fake streaming iterator
|
||||
fake_stream = FakeAnthropicMessagesStreamIterator(
|
||||
response=cast(AnthropicMessagesResponse, response)
|
||||
)
|
||||
return fake_stream
|
||||
|
||||
return None
|
||||
|
||||
def _handle_error(
|
||||
|
|
|
|||
|
|
@ -46,7 +46,21 @@ model_list:
|
|||
api_base: https://krish-mh44t553-eastus2.services.ai.azure.com
|
||||
api_key: os.environ/AZURE_ANTHROPIC_API_KEY
|
||||
|
||||
# Search Tools Configuration - Define search providers for WebSearch interception
|
||||
# search_tools:
|
||||
# - search_tool_name: "my-perplexity-search"
|
||||
# litellm_params:
|
||||
# search_provider: "perplexity" # Can be: perplexity, brave, etc.
|
||||
|
||||
litellm_settings:
|
||||
callbacks: ["websearch_interception"]
|
||||
# WebSearch Interception - Automatically intercepts and executes WebSearch tool calls
|
||||
# for models that don't natively support web search (e.g., Bedrock/Claude)
|
||||
websearch_interception_params:
|
||||
enabled_providers: ["bedrock"] # List of providers to enable interception for
|
||||
search_tool_name: "my-perplexity-search" # Optional: Name of search tool from search_tools config
|
||||
|
||||
general_settings:
|
||||
store_prompts_in_spend_logs: true
|
||||
forward_client_headers_to_llm_api: true
|
||||
forward_client_headers_to_llm_api: true
|
||||
|
||||
|
|
|
|||
8
poetry.lock
generated
8
poetry.lock
generated
|
|
@ -3081,15 +3081,15 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "litellm-proxy-extras"
|
||||
version = "0.4.21"
|
||||
version = "0.4.23"
|
||||
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
|
||||
optional = true
|
||||
python-versions = "!=2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,>=3.8"
|
||||
groups = ["main"]
|
||||
markers = "extra == \"proxy\""
|
||||
files = [
|
||||
{file = "litellm_proxy_extras-0.4.21-py3-none-any.whl", hash = "sha256:83a1734e9773610945230606012e602bbcbfba1c60fde836d51102c1a296f166"},
|
||||
{file = "litellm_proxy_extras-0.4.21.tar.gz", hash = "sha256:fa0e012984aa8e5114f88f4bad53d6abb589e5ca3eab445f74f8ddeceb62d848"},
|
||||
{file = "litellm_proxy_extras-0.4.23-py3-none-any.whl", hash = "sha256:dfda21203dde9fd97cf364396a9b5be0cfdf00fa9846439ee33ce11b7a52f9ce"},
|
||||
{file = "litellm_proxy_extras-0.4.23.tar.gz", hash = "sha256:8e3f95576dc2a296e7f73d8c87e73628bd899b4644c45863960fe3c3762d8f64"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -7981,4 +7981,4 @@ utils = ["numpydoc"]
|
|||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = ">=3.9,<4.0"
|
||||
content-hash = "ea62b77c662ab9fc486e421c576f0868bcde16d62a24703ee1f4916a0465ffb2"
|
||||
content-hash = "2d6b3d8d44919c29315b5e645befbf745a276714a2454c563d460a6a001b90af"
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[tool.poetry]
|
||||
name = "litellm"
|
||||
version = "1.80.17"
|
||||
version = "1.81.0"
|
||||
description = "Library to easily interface with LLM API providers"
|
||||
authors = ["BerriAI"]
|
||||
license = "MIT"
|
||||
|
|
@ -167,7 +167,7 @@ requires = ["poetry-core", "wheel"]
|
|||
build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.commitizen]
|
||||
version = "1.80.17"
|
||||
version = "1.81.0"
|
||||
version_files = [
|
||||
"pyproject.toml:^version"
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,33 +0,0 @@
|
|||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
import pytest
|
||||
|
||||
sys.path.insert(
|
||||
0, os.path.abspath("../../")
|
||||
) # Adds the parent directory to the system-path
|
||||
|
||||
import litellm
|
||||
|
||||
|
||||
@pytest.mark.asyncio()
|
||||
async def test_aiohttp_openai():
|
||||
litellm.set_verbose = True
|
||||
response = await litellm.acompletion(
|
||||
model="aiohttp_openai/fake-model",
|
||||
messages=[{"role": "user", "content": "Hello, world!"}],
|
||||
api_base="https://exampleopenaiendpoint-production.up.railway.app/v1/chat/completions",
|
||||
api_key="fake-key",
|
||||
)
|
||||
print(response)
|
||||
|
||||
|
||||
@pytest.mark.asyncio()
|
||||
async def test_aiohttp_openai_gpt_4o():
|
||||
litellm.set_verbose = True
|
||||
response = await litellm.acompletion(
|
||||
model="aiohttp_openai/gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello, world!"}],
|
||||
)
|
||||
print(response)
|
||||
|
|
@ -30,6 +30,7 @@ def test_deepseek_mock_completion(stream):
|
|||
messages=[{"role": "user", "content": "Hello, world!"}],
|
||||
api_base="https://exampleopenaiendpoint-production.up.railway.app/v1/chat/completions",
|
||||
stream=stream,
|
||||
mock_response="Hello! How can I help you today?",
|
||||
)
|
||||
print(f"response: {response}")
|
||||
if stream:
|
||||
|
|
|
|||
|
|
@ -1358,9 +1358,10 @@ def test_router_fallbacks_with_custom_model_costs():
|
|||
"model_name": "claude-sonnet-4-5-20250929",
|
||||
"litellm_params": {
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"api_key": os.environ["ANTHROPIC_API_KEY"],
|
||||
"api_key": os.environ.get("ANTHROPIC_API_KEY", "fake-key"),
|
||||
"input_cost_per_token": 30,
|
||||
"output_cost_per_token": 60,
|
||||
"mock_response": "Hello! How can I help you today?",
|
||||
},
|
||||
},
|
||||
{
|
||||
|
|
@ -1371,6 +1372,7 @@ def test_router_fallbacks_with_custom_model_costs():
|
|||
"output_cost_per_token": 0.000015, # 15$/M
|
||||
"api_base": "https://exampleopenaiendpoint-production.up.railway.app",
|
||||
"api_key": "my-fake-key",
|
||||
"mock_response": "Hello! How can I help you today?",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
|
|
|||
|
|
@ -323,3 +323,632 @@ async def test_websearch_interception_streaming():
|
|||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
async def test_websearch_interception_no_tool_call_streaming():
|
||||
"""
|
||||
Test WebSearch interception when LLM doesn't make a tool call with streaming.
|
||||
|
||||
This tests the scenario where:
|
||||
1. User requests stream=True
|
||||
2. WebSearch tool is provided
|
||||
3. LLM decides NOT to use the tool (just responds with text)
|
||||
4. System should return a fake stream
|
||||
"""
|
||||
print("\n" + "="*80)
|
||||
print("E2E TEST 3: WebSearch Interception (No Tool Call, Streaming)")
|
||||
print("="*80)
|
||||
|
||||
# Router already initialized from test 1
|
||||
print("\n✅ Using existing router configuration")
|
||||
print("✅ WebSearch interception already enabled for Bedrock")
|
||||
|
||||
try:
|
||||
# Make request with WebSearch tool AND stream=True
|
||||
# Use a query that the LLM will answer directly without using the tool
|
||||
print("\n📞 Making litellm.messages.acreate() call with stream=True...")
|
||||
print(f" Model: bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0")
|
||||
print(f" Query: 'What is 2+2?'")
|
||||
print(f" Tools: WebSearch")
|
||||
print(f" Stream: True")
|
||||
|
||||
response = await messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0",
|
||||
messages=[{"role": "user", "content": "What is 2+2? Just give me the answer, no need to search."}],
|
||||
tools=[
|
||||
{
|
||||
"name": "WebSearch",
|
||||
"description": "Search the web for information",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The search query",
|
||||
}
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
stream=True, # REQUEST STREAMING
|
||||
)
|
||||
|
||||
print("\n✅ Received response!")
|
||||
|
||||
# Check if response is actually a stream (async generator or async iterator)
|
||||
import inspect
|
||||
is_async_gen = inspect.isasyncgen(response)
|
||||
is_async_iter = hasattr(response, '__aiter__') and hasattr(response, '__anext__')
|
||||
is_stream = is_async_gen or is_async_iter
|
||||
|
||||
if not is_stream:
|
||||
print("\n❌ TEST 3 FAILED: Response is NOT a stream")
|
||||
print(f"❌ Expected a fake stream when LLM doesn't use the tool")
|
||||
print(f"❌ Response type: {type(response)}")
|
||||
return False
|
||||
|
||||
print(f"✅ Response is a stream (async_gen={is_async_gen}, async_iter={is_async_iter})")
|
||||
print("\n📦 Consuming stream chunks:")
|
||||
|
||||
chunks = []
|
||||
chunk_count = 0
|
||||
async for chunk in response:
|
||||
chunk_count += 1
|
||||
print(f"\n--- Chunk {chunk_count} ---")
|
||||
print(f" Type: {type(chunk)}")
|
||||
print(f" Content: {chunk[:200] if isinstance(chunk, bytes) else str(chunk)[:200]}...")
|
||||
chunks.append(chunk)
|
||||
|
||||
print(f"\n✅ Received {len(chunks)} stream chunk(s)")
|
||||
|
||||
if len(chunks) > 0:
|
||||
print("\n" + "="*80)
|
||||
print("✅ TEST 3 PASSED!")
|
||||
print("="*80)
|
||||
print("✅ User made ONE litellm.messages.acreate() call with stream=True")
|
||||
print("✅ LLM didn't use the WebSearch tool")
|
||||
print("✅ Got back a fake stream (not a non-streaming response)")
|
||||
print("✅ WebSearch interception handles no-tool-call case correctly!")
|
||||
print("="*80)
|
||||
return True
|
||||
else:
|
||||
print("\n❌ TEST 3 FAILED: No chunks received")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test 3 failed with error: {str(e)}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
async def test_claude_code_native_websearch():
|
||||
"""
|
||||
Test WebSearch interception with Claude Code's native web_search_20250305 tool.
|
||||
|
||||
This tests the exact request format that Claude Code sends:
|
||||
- tools: [{'type': 'web_search_20250305', 'name': 'web_search', 'max_uses': 8}]
|
||||
- Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
||||
"""
|
||||
print("\n" + "="*80)
|
||||
print("E2E TEST: Claude Code Native WebSearch (web_search_20250305)")
|
||||
print("="*80)
|
||||
|
||||
# Router already initialized from test 1
|
||||
print("\n✅ Using existing router configuration")
|
||||
print("✅ WebSearch interception already enabled for Bedrock")
|
||||
|
||||
try:
|
||||
# Make request with Claude Code's exact native web_search tool format
|
||||
print("\n📞 Making litellm.messages.acreate() call...")
|
||||
print(f" Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0")
|
||||
print(f" Query: 'Perform a web search for the query: litellm what is it'")
|
||||
print(f" Tools: Native web_search_20250305")
|
||||
print(f" Stream: False")
|
||||
|
||||
response = await messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
messages=[{"role": "user", "content": "Perform a web search for the query: litellm what is it"}],
|
||||
tools=[
|
||||
{
|
||||
"type": "web_search_20250305",
|
||||
"name": "web_search",
|
||||
"max_uses": 8
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
stream=False,
|
||||
)
|
||||
|
||||
print("\n✅ Received response!")
|
||||
|
||||
# Handle both dict and object responses
|
||||
if isinstance(response, dict):
|
||||
response_id = response.get("id")
|
||||
response_model = response.get("model")
|
||||
response_stop_reason = response.get("stop_reason")
|
||||
response_content = response.get("content", [])
|
||||
else:
|
||||
response_id = response.id
|
||||
response_model = response.model
|
||||
response_stop_reason = response.stop_reason
|
||||
response_content = response.content
|
||||
|
||||
print(f"\n📄 Response ID: {response_id}")
|
||||
print(f"📄 Model: {response_model}")
|
||||
print(f"📄 Stop Reason: {response_stop_reason}")
|
||||
print(f"📄 Content blocks: {len(response_content)}")
|
||||
|
||||
# Debug: Print all content block types
|
||||
for i, block in enumerate(response_content):
|
||||
block_type = block.get("type") if isinstance(block, dict) else block.type
|
||||
print(f" Block {i}: type={block_type}")
|
||||
if block_type == "tool_use":
|
||||
block_name = block.get("name") if isinstance(block, dict) else block.name
|
||||
print(f" name={block_name}")
|
||||
|
||||
# Validate response
|
||||
assert response is not None, "Response should not be None"
|
||||
assert response_content is not None, "Response should have content"
|
||||
assert len(response_content) > 0, "Response should have at least one content block"
|
||||
|
||||
# Check if response contains tool_use (means interception didn't work)
|
||||
has_tool_use = any(
|
||||
(block.get("type") if isinstance(block, dict) else block.type) == "tool_use"
|
||||
for block in response_content
|
||||
)
|
||||
|
||||
# Check if we got a text response
|
||||
has_text = any(
|
||||
(block.get("type") if isinstance(block, dict) else block.type) == "text"
|
||||
for block in response_content
|
||||
)
|
||||
|
||||
if has_tool_use:
|
||||
print("\n❌ TEST FAILED: Interception did not work")
|
||||
print(f"❌ Stop reason: {response_stop_reason}")
|
||||
print("❌ Response contains tool_use blocks")
|
||||
return False
|
||||
|
||||
elif has_text and response_stop_reason != "tool_use":
|
||||
text_block = next(
|
||||
block for block in response_content
|
||||
if (block.get("type") if isinstance(block, dict) else block.type) == "text"
|
||||
)
|
||||
text_content = text_block.get("text") if isinstance(text_block, dict) else text_block.text
|
||||
|
||||
print(f"\n📝 Response Text:")
|
||||
print(f" {text_content[:200]}...")
|
||||
|
||||
if "litellm" in text_content.lower():
|
||||
print("\n" + "="*80)
|
||||
print("✅ TEST PASSED!")
|
||||
print("="*80)
|
||||
print("✅ Claude Code's native web_search_20250305 tool was intercepted")
|
||||
print("✅ Tool was converted to LiteLLM standard format")
|
||||
print("✅ User made ONE litellm.messages.acreate() call")
|
||||
print("✅ Got back final answer with search results")
|
||||
print("✅ Agentic loop executed transparently")
|
||||
print("✅ WebSearch interception working with Claude Code!")
|
||||
print("="*80)
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ Got text response but doesn't mention LiteLLM")
|
||||
return False
|
||||
else:
|
||||
print("\n❌ Unexpected response format")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test failed with error: {str(e)}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import asyncio
|
||||
|
||||
async def run_all_tests():
|
||||
"""Run all E2E tests"""
|
||||
test_results = []
|
||||
|
||||
# Test 1: Non-streaming
|
||||
result1 = await test_websearch_interception_non_streaming()
|
||||
test_results.append(("Non-Streaming", result1))
|
||||
|
||||
# Test 2: Streaming
|
||||
result2 = await test_websearch_interception_streaming()
|
||||
test_results.append(("Streaming", result2))
|
||||
|
||||
# Test 3: No tool call with streaming
|
||||
result3 = await test_websearch_interception_no_tool_call_streaming()
|
||||
test_results.append(("No Tool Call Streaming", result3))
|
||||
|
||||
# Test 4: Claude Code native web_search
|
||||
result4 = await test_claude_code_native_websearch()
|
||||
test_results.append(("Claude Code Native WebSearch", result4))
|
||||
|
||||
# Print summary
|
||||
print("\n" + "="*80)
|
||||
print("TEST SUMMARY")
|
||||
print("="*80)
|
||||
for test_name, result in test_results:
|
||||
status = "✅ PASSED" if result else "❌ FAILED"
|
||||
print(f"{test_name}: {status}")
|
||||
print("="*80)
|
||||
|
||||
# Return overall result
|
||||
return all(result for _, result in test_results)
|
||||
|
||||
result = asyncio.run(run_all_tests())
|
||||
import sys
|
||||
sys.exit(0 if result else 1)
|
||||
|
||||
|
||||
async def test_litellm_standard_websearch_tool():
|
||||
"""
|
||||
PRIORITY TEST #1: Test with the canonical litellm_web_search tool format.
|
||||
|
||||
This validates that using get_litellm_web_search_tool() directly
|
||||
works end-to-end without any conversion needed.
|
||||
"""
|
||||
print("\n" + "="*80)
|
||||
print("E2E TEST: LiteLLM Standard WebSearch Tool")
|
||||
print("="*80)
|
||||
|
||||
from litellm.integrations.websearch_interception import get_litellm_web_search_tool
|
||||
|
||||
print("\n✅ Using existing router configuration")
|
||||
print("✅ WebSearch interception already enabled for Bedrock")
|
||||
|
||||
try:
|
||||
print("\n📞 Making litellm.messages.acreate() call...")
|
||||
print(f" Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0")
|
||||
print(f" Query: 'What is the latest news about AI?'")
|
||||
print(f" Tool: litellm_web_search (standard format, no conversion needed)")
|
||||
print(f" Stream: False")
|
||||
|
||||
response = await messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
messages=[{"role": "user", "content": "What is the latest news about AI? Give me a brief overview."}],
|
||||
tools=[get_litellm_web_search_tool()],
|
||||
max_tokens=1024,
|
||||
stream=False,
|
||||
)
|
||||
|
||||
print("\n✅ Received response!")
|
||||
|
||||
if isinstance(response, dict):
|
||||
response_id = response.get("id")
|
||||
response_stop_reason = response.get("stop_reason")
|
||||
response_content = response.get("content", [])
|
||||
else:
|
||||
response_id = response.id
|
||||
response_stop_reason = response.stop_reason
|
||||
response_content = response.content
|
||||
|
||||
print(f"\n📄 Response ID: {response_id}")
|
||||
print(f"📄 Stop Reason: {response_stop_reason}")
|
||||
print(f"📄 Content blocks: {len(response_content)}")
|
||||
|
||||
for i, block in enumerate(response_content):
|
||||
block_type = block.get("type") if isinstance(block, dict) else block.type
|
||||
print(f" Block {i}: type={block_type}")
|
||||
|
||||
has_tool_use = any(
|
||||
(block.get("type") if isinstance(block, dict) else block.type) == "tool_use"
|
||||
for block in response_content
|
||||
)
|
||||
|
||||
has_text = any(
|
||||
(block.get("type") if isinstance(block, dict) else block.type) == "text"
|
||||
for block in response_content
|
||||
)
|
||||
|
||||
if has_tool_use:
|
||||
print("\n❌ TEST FAILED: Interception did not work")
|
||||
return False
|
||||
|
||||
elif has_text and response_stop_reason != "tool_use":
|
||||
text_block = next(
|
||||
block for block in response_content
|
||||
if (block.get("type") if isinstance(block, dict) else block.type) == "text"
|
||||
)
|
||||
text_content = text_block.get("text") if isinstance(text_block, dict) else text_block.text
|
||||
|
||||
print(f"\n📝 Response Text: {text_content[:200]}...")
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("✅ TEST PASSED!")
|
||||
print("="*80)
|
||||
print("✅ LiteLLM standard tool format works without conversion")
|
||||
print("✅ Agentic loop executed transparently")
|
||||
print("="*80)
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Unexpected response format")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test failed with error: {str(e)}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
async def test_claude_code_native_websearch_streaming():
|
||||
"""
|
||||
PRIORITY TEST #2: Test Claude Code's native tool WITH stream=True.
|
||||
|
||||
Validates:
|
||||
- Native tool conversion (web_search_20250305 → litellm_web_search)
|
||||
- Stream=True → Stream=False conversion
|
||||
- Agentic loop executes with both conversions
|
||||
"""
|
||||
print("\n" + "="*80)
|
||||
print("E2E TEST: Claude Code Native WebSearch + Streaming")
|
||||
print("="*80)
|
||||
|
||||
print("\n✅ Using existing router configuration")
|
||||
print("✅ WebSearch interception already enabled for Bedrock")
|
||||
|
||||
try:
|
||||
print("\n📞 Making litellm.messages.acreate() call with stream=True...")
|
||||
print(f" Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0")
|
||||
print(f" Tool: Native web_search_20250305")
|
||||
print(f" Stream: True (will be converted to False)")
|
||||
|
||||
response = await messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
messages=[{"role": "user", "content": "Search for the latest AI developments."}],
|
||||
tools=[{"type": "web_search_20250305", "name": "web_search", "max_uses": 8}],
|
||||
max_tokens=1024,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
print("\n✅ Received response!")
|
||||
|
||||
import inspect
|
||||
is_stream = inspect.isasyncgen(response)
|
||||
|
||||
if is_stream:
|
||||
print("\n⚠️ Response is a stream (stream conversion didn't work)")
|
||||
return False
|
||||
|
||||
print("✅ Response is NOT a stream (conversion worked!)")
|
||||
|
||||
if isinstance(response, dict):
|
||||
response_stop_reason = response.get("stop_reason")
|
||||
response_content = response.get("content", [])
|
||||
else:
|
||||
response_stop_reason = response.stop_reason
|
||||
response_content = response.content
|
||||
|
||||
has_tool_use = any(
|
||||
(block.get("type") if isinstance(block, dict) else block.type) == "tool_use"
|
||||
for block in response_content
|
||||
)
|
||||
|
||||
has_text = any(
|
||||
(block.get("type") if isinstance(block, dict) else block.type) == "text"
|
||||
for block in response_content
|
||||
)
|
||||
|
||||
if has_tool_use:
|
||||
print("\n❌ TEST FAILED: Interception did not work")
|
||||
return False
|
||||
|
||||
elif has_text and response_stop_reason != "tool_use":
|
||||
print("\n" + "="*80)
|
||||
print("✅ TEST PASSED!")
|
||||
print("="*80)
|
||||
print("✅ Native tool converted to litellm_web_search")
|
||||
print("✅ Stream=True converted to Stream=False")
|
||||
print("✅ Both conversions working together!")
|
||||
print("="*80)
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Unexpected response format")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test failed with error: {str(e)}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def test_is_web_search_tool_detection():
|
||||
"""
|
||||
PRIORITY TEST #3: Unit test for is_web_search_tool() utility.
|
||||
|
||||
Validates detection of all supported formats including future versions.
|
||||
"""
|
||||
print("\n" + "="*80)
|
||||
print("UNIT TEST: Web Search Tool Detection")
|
||||
print("="*80)
|
||||
|
||||
from litellm.integrations.websearch_interception import is_web_search_tool
|
||||
|
||||
test_cases = [
|
||||
({"name": "litellm_web_search"}, True, "LiteLLM standard tool"),
|
||||
({"type": "web_search_20250305", "name": "web_search", "max_uses": 8}, True, "Current Anthropic native (2025)"),
|
||||
({"type": "web_search_2026", "name": "web_search"}, True, "Future Anthropic native (2026)"),
|
||||
({"type": "web_search_20270615", "name": "web_search"}, True, "Future Anthropic native (2027)"),
|
||||
({"name": "web_search", "type": "web_search_20250305"}, True, "Claude Code format"),
|
||||
({"name": "WebSearch"}, True, "Legacy WebSearch"),
|
||||
({"name": "calculator"}, False, "Non-web-search tool"),
|
||||
({"name": "some_tool", "type": "function"}, False, "Other tool with type"),
|
||||
({"type": "custom_tool"}, False, "Custom tool type"),
|
||||
]
|
||||
|
||||
passed = 0
|
||||
failed = 0
|
||||
|
||||
for tool, expected, description in test_cases:
|
||||
result = is_web_search_tool(tool)
|
||||
if result == expected:
|
||||
print(f" ✅ PASS: {description}")
|
||||
passed += 1
|
||||
else:
|
||||
print(f" ❌ FAIL: {description}")
|
||||
print(f" Tool: {tool}")
|
||||
print(f" Expected: {expected}, Got: {result}")
|
||||
failed += 1
|
||||
|
||||
print(f"\n📊 Results: {passed} passed, {failed} failed")
|
||||
|
||||
if failed == 0:
|
||||
print("\n" + "="*80)
|
||||
print("✅ ALL DETECTION TESTS PASSED!")
|
||||
print("="*80)
|
||||
print("✅ Detects all current formats")
|
||||
print("✅ Future-proof for new web_search_* versions")
|
||||
print("="*80)
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Some detection tests failed")
|
||||
return False
|
||||
|
||||
|
||||
async def test_pre_request_hook_modifies_request_body():
|
||||
"""
|
||||
Unit test to verify async_pre_request_hook correctly modifies request body.
|
||||
|
||||
Tests that:
|
||||
1. WebSearchInterceptionLogger is active
|
||||
2. Native web_search_20250305 tool is converted to litellm_web_search
|
||||
3. Stream is converted from True to False
|
||||
4. Modified parameters reach the API call
|
||||
"""
|
||||
import asyncio
|
||||
from unittest.mock import AsyncMock, patch, MagicMock
|
||||
from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME
|
||||
|
||||
litellm._turn_on_debug()
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("UNIT TEST: Pre-Request Hook Modifies Request Body")
|
||||
print("="*80)
|
||||
|
||||
# Initialize WebSearchInterceptionLogger
|
||||
litellm.callbacks = [
|
||||
WebSearchInterceptionLogger(
|
||||
enabled_providers=[LlmProviders.BEDROCK],
|
||||
search_tool_name="test-search-tool"
|
||||
)
|
||||
]
|
||||
|
||||
print("✅ WebSearchInterceptionLogger initialized")
|
||||
|
||||
# Track what actually gets sent to the API
|
||||
captured_request = {}
|
||||
|
||||
def mock_anthropic_messages_handler(
|
||||
max_tokens,
|
||||
messages,
|
||||
model,
|
||||
metadata=None,
|
||||
stop_sequences=None,
|
||||
stream=None,
|
||||
system=None,
|
||||
temperature=None,
|
||||
thinking=None,
|
||||
tool_choice=None,
|
||||
tools=None,
|
||||
top_k=None,
|
||||
top_p=None,
|
||||
container=None,
|
||||
api_key=None,
|
||||
api_base=None,
|
||||
client=None,
|
||||
custom_llm_provider=None,
|
||||
**kwargs
|
||||
):
|
||||
"""Mock handler that captures the actual request parameters"""
|
||||
# Capture what gets sent to the handler (after hook modifications)
|
||||
captured_request['tools'] = tools
|
||||
captured_request['stream'] = stream
|
||||
captured_request['max_tokens'] = max_tokens
|
||||
captured_request['model'] = model
|
||||
|
||||
# Return a mock response (non-streaming)
|
||||
from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicMessagesResponse
|
||||
return AnthropicMessagesResponse(
|
||||
id="msg_test",
|
||||
type="message",
|
||||
role="assistant",
|
||||
content=[{
|
||||
"type": "text",
|
||||
"text": "Test response"
|
||||
}],
|
||||
model="claude-sonnet-4-5",
|
||||
stop_reason="end_turn",
|
||||
usage={
|
||||
"input_tokens": 10,
|
||||
"output_tokens": 20
|
||||
}
|
||||
)
|
||||
|
||||
# Patch the anthropic_messages_handler function (called after hooks)
|
||||
with patch('litellm.llms.anthropic.experimental_pass_through.messages.handler.anthropic_messages_handler',
|
||||
side_effect=mock_anthropic_messages_handler):
|
||||
|
||||
print("\n📝 Making request with native web_search_20250305 tool (stream=True)...")
|
||||
|
||||
# Make the request with native tool format
|
||||
response = await messages.acreate(
|
||||
model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
messages=[{"role": "user", "content": "Test query"}],
|
||||
tools=[{
|
||||
"type": "web_search_20250305",
|
||||
"name": "web_search",
|
||||
"max_uses": 8
|
||||
}],
|
||||
max_tokens=100,
|
||||
stream=True # Should be converted to False
|
||||
)
|
||||
|
||||
print("\n🔍 Verifying request modifications...")
|
||||
|
||||
# Verify tool was converted
|
||||
tools = captured_request.get('tools')
|
||||
print(f"\n Captured tools: {tools}")
|
||||
|
||||
if tools and len(tools) > 0:
|
||||
tool = tools[0]
|
||||
tool_name = tool.get('name')
|
||||
|
||||
if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME:
|
||||
print(f" ✅ Tool converted: web_search_20250305 → {LITELLM_WEB_SEARCH_TOOL_NAME}")
|
||||
else:
|
||||
print(f" ❌ Tool NOT converted: expected {LITELLM_WEB_SEARCH_TOOL_NAME}, got {tool_name}")
|
||||
return False
|
||||
else:
|
||||
print(" ❌ No tools captured in request")
|
||||
return False
|
||||
|
||||
# Verify stream was converted
|
||||
stream = captured_request.get('stream')
|
||||
print(f" Captured stream: {stream}")
|
||||
|
||||
if stream is False:
|
||||
print(" ✅ Stream converted: True → False")
|
||||
else:
|
||||
print(f" ❌ Stream NOT converted: expected False, got {stream}")
|
||||
return False
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("✅ PRE-REQUEST HOOK TEST PASSED!")
|
||||
print("="*80)
|
||||
print("✅ CustomLogger is active")
|
||||
print("✅ async_pre_request_hook modifies request body")
|
||||
print("✅ Tool conversion works correctly")
|
||||
print("✅ Stream conversion works correctly")
|
||||
print("="*80)
|
||||
|
||||
return True
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue