diff --git a/docs/my-website/docs/tutorials/claude_code_websearch.md b/docs/my-website/docs/tutorials/claude_code_websearch.md new file mode 100644 index 00000000000..cc2f79666da --- /dev/null +++ b/docs/my-website/docs/tutorials/claude_code_websearch.md @@ -0,0 +1,192 @@ +# Claude Code - WebSearch Across All Providers + +Enable Claude Code's web search tool to work with any provider (Bedrock, Azure, Vertex, etc.). LiteLLM automatically intercepts web search requests and executes them server-side. + +## Proxy Configuration + +Add WebSearch interception to your `litellm_config.yaml`: + +```yaml +model_list: + - model_name: bedrock-sonnet + litellm_params: + model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0 + aws_region_name: us-east-1 + +# Enable WebSearch interception for providers +litellm_settings: + callbacks: + - websearch_interception: + enabled_providers: + - bedrock + - azure + - vertex_ai + search_tool_name: perplexity-search # Optional: specific search tool + +# Configure search provider +search_tools: + - search_tool_name: perplexity-search + litellm_params: + search_provider: perplexity + api_key: os.environ/PERPLEXITY_API_KEY +``` + +## Quick Start + +### 1. Configure LiteLLM Proxy + +Create `config.yaml`: + +```yaml +model_list: + - model_name: bedrock-sonnet + litellm_params: + model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0 + aws_region_name: us-east-1 + +litellm_settings: + callbacks: + - websearch_interception: + enabled_providers: [bedrock] + +search_tools: + - search_tool_name: perplexity-search + litellm_params: + search_provider: perplexity + api_key: os.environ/PERPLEXITY_API_KEY +``` + +### 2. Start Proxy + +```bash +export PERPLEXITY_API_KEY=your-key +litellm --config config.yaml +``` + +### 3. Use with Claude Code + +```bash +export ANTHROPIC_BASE_URL=http://localhost:4000 +export ANTHROPIC_API_KEY=sk-1234 +claude +``` + +Now use web search in Claude Code - it works with any provider! + +## How It Works + +When Claude Code sends a web search request, LiteLLM: +1. Intercepts the native `web_search` tool +2. Converts it to LiteLLM's standard format +3. Executes the search via Perplexity/Tavily +4. Returns the final answer to Claude Code + +```mermaid +sequenceDiagram + participant CC as Claude Code + participant LP as LiteLLM Proxy + participant B as Bedrock/Azure/etc + participant P as Perplexity/Tavily + + CC->>LP: Request with web_search tool + Note over LP: Convert native tool
to LiteLLM format + LP->>B: Request with converted tool + B-->>LP: Response: tool_use + Note over LP: Detect web search
tool_use + LP->>P: Execute search + P-->>LP: Search results + LP->>B: Follow-up with results + B-->>LP: Final answer + LP-->>CC: Final answer with search results +``` + +**Result**: One API call from Claude Code → Complete answer with search results + +## Supported Providers + +| Provider | Native Web Search | With LiteLLM | +|----------|-------------------|--------------| +| **Anthropic** | ✅ Yes | ✅ Yes | +| **Bedrock** | ❌ No | ✅ Yes | +| **Azure** | ❌ No | ✅ Yes | +| **Vertex AI** | ❌ No | ✅ Yes | +| **Other Providers** | ❌ No | ✅ Yes | + +## Search Providers + +Configure which search provider to use. LiteLLM supports multiple search providers: + +| Provider | Configuration | +|----------|---------------| +| **Perplexity** | `search_provider: perplexity` | +| **Tavily** | `search_provider: tavily` | + +See [all supported search providers](../search/index.md) for the complete list. + +## Configuration Options + +### WebSearch Interception Parameters + +| Parameter | Type | Required | Description | Example | +|-----------|------|----------|-------------|---------| +| `enabled_providers` | List[String] | Yes | List of providers to enable web search interception for | `[bedrock, azure, vertex_ai]` | +| `search_tool_name` | String | No | Specific search tool from `search_tools` config. If not set, uses first available search tool. | `perplexity-search` | + +### Supported Provider Values + +Use these values in `enabled_providers`: + +| Provider | Value | Description | +|----------|-------|-------------| +| AWS Bedrock | `bedrock` | Amazon Bedrock Claude models | +| Azure OpenAI | `azure` | Azure-hosted models | +| Google Vertex AI | `vertex_ai` | Google Cloud Vertex AI | +| Any Other | Provider name | Any LiteLLM-supported provider | + +### Complete Configuration Example + +```yaml +model_list: + - model_name: bedrock-sonnet + litellm_params: + model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0 + aws_region_name: us-east-1 + + - model_name: azure-gpt4 + litellm_params: + model: azure/gpt-4 + api_base: https://my-azure.openai.azure.com + api_key: os.environ/AZURE_API_KEY + +litellm_settings: + callbacks: + - websearch_interception: + enabled_providers: + - bedrock # Enable for AWS Bedrock + - azure # Enable for Azure OpenAI + - vertex_ai # Enable for Google Vertex + search_tool_name: perplexity-search # Optional: use specific search tool + +# Configure search tools +search_tools: + - search_tool_name: perplexity-search + litellm_params: + search_provider: perplexity + api_key: os.environ/PERPLEXITY_API_KEY + + - search_tool_name: tavily-search + litellm_params: + search_provider: tavily + api_key: os.environ/TAVILY_API_KEY +``` + +**How search tool selection works:** +- If `search_tool_name` is specified → Uses that specific search tool +- If `search_tool_name` is not specified → Uses first search tool in `search_tools` list +- In example above: Without `search_tool_name`, would use `perplexity-search` (first in list) + +## Related + +- [Claude Code Quickstart](./claude_responses_api.md) +- [Claude Code Cost Tracking](./claude_code_customer_tracking.md) +- [Using Non-Anthropic Models](./claude_non_anthropic_models.md) diff --git a/docs/my-website/release_notes/v1.81.0/index.md b/docs/my-website/release_notes/v1.81.0/index.md new file mode 100644 index 00000000000..071422f96be --- /dev/null +++ b/docs/my-website/release_notes/v1.81.0/index.md @@ -0,0 +1,487 @@ +--- +title: "v1.81.0 - Claude Code - Web Search with all LiteLLM Providers" +slug: "v1-81-0" +date: 2026-01-18T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg + - name: Ishaan Jaff + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Deploy this version + + + + +``` showLineNumbers title="docker run litellm" +docker run \ +-e STORE_MODEL_IN_DB=True \ +-p 4000:4000 \ +docker.litellm.ai/berriai/litellm:v1.81.0 +``` + + + + + +``` showLineNumbers title="pip install litellm" +pip install litellm==1.81.0 +``` + + + + +--- + +## Key Highlights + +- **Claude Code** - Support for using web search across Bedrock, Vertex AI, and all LiteLLM providers +- **Major Change** - [50MB limit on image URL downloads](#major-change---chatcompletions-image-url-download-size-limit) to improve reliability + +--- + +## Major Change - /chat/completions Image URL Download Size Limit + +To improve reliability and prevent memory issues, LiteLLM now includes a configurable **50MB limit** on image URL downloads by default. Previously, there was no limit on image downloads, which could occasionally cause memory issues with very large images. + +### How It Works + +Requests with image URLs exceeding 50MB will receive a helpful error message: + +```bash +curl -X POST 'https://your-litellm-proxy.com/chat/completions' \ + -H 'Content-Type: application/json' \ + -H 'Authorization: Bearer sk-1234' \ + -d '{ + "model": "gpt-4o", + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "What is in this image?" + }, + { + "type": "image_url", + "image_url": { + "url": "https://example.com/very-large-image.jpg" + } + } + ] + } + ] + }' +``` + +**Error Response:** + +```json +{ + "error": { + "message": "Error: Image size (75.50MB) exceeds maximum allowed size (50.0MB). url=https://example.com/very-large-image.jpg", + "type": "ImageFetchError" + } +} +``` + +### Configuring the Limit + +The default 50MB limit works well for most use cases, but you can easily adjust it if needed: + +**Increase the limit (e.g., to 100MB):** + +```bash +export MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=100 +``` + +**Disable image URL downloads (for security):** + +```bash +export MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=0 +``` + +**Docker Configuration:** + +```bash +docker run \ + -e MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=100 \ + -p 4000:4000 \ + docker.litellm.ai/berriai/litellm:v1.81.0 +``` + +**Proxy Config (config.yaml):** + +```yaml +general_settings: + master_key: sk-1234 + +# Set via environment variable +environment_variables: + MAX_IMAGE_URL_DOWNLOAD_SIZE_MB: "100" +``` + +### Why Add This? + +This feature improves reliability by: +- Preventing memory issues from very large images +- Aligning with OpenAI's 50MB payload limit +- Validating image sizes early (when Content-Length header is available) + +--- + +## New Models / Updated Models + +#### New Model Support + +| Provider | Model | Features | +| -------- | ----- | -------- | +| OpenAI | `gpt-5.2-codex` | Code generation | +| Azure | `azure/gpt-5.2-codex` | Code generation | +| Cerebras | `cerebras/zai-glm-4.7` | Reasoning, function calling | +| Replicate | All chat models | Full support for all Replicate chat models | + +#### Features + +- **[Anthropic](../../docs/providers/anthropic)** + - Add missing anthropic tool results in response - [PR #18945](https://github.com/BerriAI/litellm/pull/18945) + - Preserve web_fetch_tool_result in multi-turn conversations - [PR #18142](https://github.com/BerriAI/litellm/pull/18142) + +- **[Gemini](../../docs/providers/gemini)** + - Add presence_penalty support for Google AI Studio - [PR #18154](https://github.com/BerriAI/litellm/pull/18154) + - Forward extra_headers in generateContent adapter - [PR #18935](https://github.com/BerriAI/litellm/pull/18935) + - Add medium value support for detail param - [PR #19187](https://github.com/BerriAI/litellm/pull/19187) + +- **[Vertex AI](../../docs/providers/vertex)** + - Improve passthrough endpoint URL parsing and construction - [PR #17526](https://github.com/BerriAI/litellm/pull/17526) + - Add type object to tool schemas missing type field - [PR #19103](https://github.com/BerriAI/litellm/pull/19103) + - Keep type field in Gemini schema when properties is empty - [PR #18979](https://github.com/BerriAI/litellm/pull/18979) + +- **[Bedrock](../../docs/providers/bedrock)** + - Add OpenAI-compatible service_tier parameter translation - [PR #18091](https://github.com/BerriAI/litellm/pull/18091) + - Add user auth in standard logging object for Bedrock passthrough - [PR #19140](https://github.com/BerriAI/litellm/pull/19140) + - Strip throughput tier suffixes from model names - [PR #19147](https://github.com/BerriAI/litellm/pull/19147) + +- **[OCI](../../docs/providers/oci)** + - Handle OpenAI-style image_url object in multimodal messages - [PR #18272](https://github.com/BerriAI/litellm/pull/18272) + +- **[Ollama](../../docs/providers/ollama)** + - Set finish_reason to tool_calls and remove broken capability check - [PR #18924](https://github.com/BerriAI/litellm/pull/18924) + +- **[Watsonx](../../docs/providers/watsonx/index)** + - Allow passing scope ID for Watsonx inferencing - [PR #18959](https://github.com/BerriAI/litellm/pull/18959) + +- **[Replicate](../../docs/providers/replicate)** + - Add all chat Replicate models support - [PR #18954](https://github.com/BerriAI/litellm/pull/18954) + +- **[OpenRouter](../../docs/providers/openrouter)** + - Add OpenRouter support for image/generation endpoints - [PR #19059](https://github.com/BerriAI/litellm/pull/19059) + +- **[Volcengine](../../docs/providers/volcano)** + - Add max_tokens settings for Volcengine models (deepseek-v3-2, glm-4-7, kimi-k2-thinking) - [PR #19076](https://github.com/BerriAI/litellm/pull/19076) + +- **Azure Model Router** + - New Model - Azure Model Router on LiteLLM AI Gateway - [PR #19054](https://github.com/BerriAI/litellm/pull/19054) + +- **GPT-5 Models** + - Correct context window sizes for GPT-5 model variants - [PR #18928](https://github.com/BerriAI/litellm/pull/18928) + - Correct max_input_tokens for GPT-5 models - [PR #19056](https://github.com/BerriAI/litellm/pull/19056) + +- **Text Completion** + - Support token IDs (list of integers) as prompt - [PR #18011](https://github.com/BerriAI/litellm/pull/18011) + +### Bug Fixes + +- **[Anthropic](../../docs/providers/anthropic)** + - Prevent dropping thinking when any message has thinking_blocks - [PR #18929](https://github.com/BerriAI/litellm/pull/18929) + - Fix anthropic token counter with thinking - [PR #19067](https://github.com/BerriAI/litellm/pull/19067) + - Add better error handling for Anthropic - [PR #18955](https://github.com/BerriAI/litellm/pull/18955) + - Fix Anthropic during call error - [PR #19060](https://github.com/BerriAI/litellm/pull/19060) + +- **[Gemini](../../docs/providers/gemini)** + - Fix missing `completion_tokens_details` in Gemini 3 Flash when reasoning_effort is not used - [PR #18898](https://github.com/BerriAI/litellm/pull/18898) + - Fix Gemini Image Generation imageConfig parameters - [PR #18948](https://github.com/BerriAI/litellm/pull/18948) + +- **[Vertex AI](../../docs/providers/vertex)** + - Fix Vertex AI 400 Error with CachedContent model mismatch - [PR #19193](https://github.com/BerriAI/litellm/pull/19193) + - Fix Vertex AI doesn't support structured output - [PR #19201](https://github.com/BerriAI/litellm/pull/19201) + +- **[Bedrock](../../docs/providers/bedrock)** + - Fix Claude Code (`/messages`) Bedrock Invoke usage and request signing - [PR #19111](https://github.com/BerriAI/litellm/pull/19111) + - Fix model ID encoding for Bedrock passthrough - [PR #18944](https://github.com/BerriAI/litellm/pull/18944) + - Respect max_completion_tokens in thinking feature - [PR #18946](https://github.com/BerriAI/litellm/pull/18946) + - Fix header forwarding in Bedrock passthrough - [PR #19007](https://github.com/BerriAI/litellm/pull/19007) + - Fix Bedrock stability model usage issues - [PR #19199](https://github.com/BerriAI/litellm/pull/19199) + +--- + +## LLM API Endpoints + +#### Features + +- **[/messages (Claude Code)](../../docs/providers/anthropic)** + - Add support for Tool Search on `/messages` API across Azure, Bedrock, and Anthropic API - [PR #19165](https://github.com/BerriAI/litellm/pull/19165) + - Track end-users with Claude Code (`/messages`) for better analytics and monitoring - [PR #19171](https://github.com/BerriAI/litellm/pull/19171) + - Add web search support using LiteLLM `/search` endpoint with Claude Code (`/messages`) - [PR #19263](https://github.com/BerriAI/litellm/pull/19263), [PR #19294](https://github.com/BerriAI/litellm/pull/19294) + +- **[/messages (Claude Code) - Bedrock](../../docs/providers/bedrock)** + - Add support for Prompt Caching with Bedrock Converse on `/messages` - [PR #19123](https://github.com/BerriAI/litellm/pull/19123) + - Ensure budget tokens are passed to Bedrock Converse API correctly on `/messages` - [PR #19107](https://github.com/BerriAI/litellm/pull/19107) + +- **[Responses API](../../docs/response_api)** + - Add support for caching for responses API - [PR #19068](https://github.com/BerriAI/litellm/pull/19068) + - Add retry policy support to responses API - [PR #19074](https://github.com/BerriAI/litellm/pull/19074) + +- **Realtime API** + - Use non-streaming method for endpoint v1/a2a/message/send - [PR #19025](https://github.com/BerriAI/litellm/pull/19025) + +- **Batch API** + - Fix batch deletion and retrieve - [PR #18340](https://github.com/BerriAI/litellm/pull/18340) + +#### Bugs + +- **General** + - Fix responses content can't be none - [PR #19064](https://github.com/BerriAI/litellm/pull/19064) + - Fix model name from query param in realtime request - [PR #19135](https://github.com/BerriAI/litellm/pull/19135) + - Fix video status/content credential injection for wildcard models - [PR #18854](https://github.com/BerriAI/litellm/pull/18854) + +--- + +## Management Endpoints / UI + +#### Features + +**Virtual Keys** +- View deleted keys for audit purposes - [PR #18228](https://github.com/BerriAI/litellm/pull/18228), [PR #19268](https://github.com/BerriAI/litellm/pull/19268) +- Add status query parameter for keys list - [PR #19260](https://github.com/BerriAI/litellm/pull/19260) +- Refetch keys after key creation - [PR #18994](https://github.com/BerriAI/litellm/pull/18994) +- Refresh keys list on delete - [PR #19262](https://github.com/BerriAI/litellm/pull/19262) +- Simplify key generate permission error - [PR #18997](https://github.com/BerriAI/litellm/pull/18997) +- Add search to key edit team dropdown - [PR #19119](https://github.com/BerriAI/litellm/pull/19119) + +**Teams & Organizations** +- View deleted teams for audit purposes - [PR #18228](https://github.com/BerriAI/litellm/pull/18228), [PR #19268](https://github.com/BerriAI/litellm/pull/19268) +- Add filters to organization table - [PR #18916](https://github.com/BerriAI/litellm/pull/18916) +- Add query parameters to `/organization/list` - [PR #18910](https://github.com/BerriAI/litellm/pull/18910) +- Add status query parameter for teams list - [PR #19260](https://github.com/BerriAI/litellm/pull/19260) +- Show internal users their spend only - [PR #19227](https://github.com/BerriAI/litellm/pull/19227) +- Allow preventing team admins from deleting members from teams - [PR #19128](https://github.com/BerriAI/litellm/pull/19128) +- Refactor team member icon buttons - [PR #19192](https://github.com/BerriAI/litellm/pull/19192) + +**Models + Endpoints** +- Display health information in public model hub - [PR #19256](https://github.com/BerriAI/litellm/pull/19256), [PR #19258](https://github.com/BerriAI/litellm/pull/19258) +- Quality of life improvements for Anthropic models - [PR #19058](https://github.com/BerriAI/litellm/pull/19058) +- Create reusable model select component - [PR #19164](https://github.com/BerriAI/litellm/pull/19164) +- Edit settings model dropdown - [PR #19186](https://github.com/BerriAI/litellm/pull/19186) +- Fix model hub client side exception - [PR #19045](https://github.com/BerriAI/litellm/pull/19045) + +**Usage & Analytics** +- Allow top virtual keys and models to show more entries - [PR #19050](https://github.com/BerriAI/litellm/pull/19050) +- Fix Y axis on model activity chart - [PR #19055](https://github.com/BerriAI/litellm/pull/19055) +- Add Team ID and Team Name in export report - [PR #19047](https://github.com/BerriAI/litellm/pull/19047) +- Add user metrics for Prometheus - [PR #18785](https://github.com/BerriAI/litellm/pull/18785) + +**SSO & Auth** +- Allow setting custom MSFT Base URLs - [PR #18977](https://github.com/BerriAI/litellm/pull/18977) +- Allow overriding env var attribute names - [PR #18998](https://github.com/BerriAI/litellm/pull/18998) +- Fix SCIM GET /Users error and enforce SCIM 2.0 compliance - [PR #17420](https://github.com/BerriAI/litellm/pull/17420) +- Feature flag for SCIM compliance fix - [PR #18878](https://github.com/BerriAI/litellm/pull/18878) + +**General UI** +- Add allowClear to dropdown components for better UX - [PR #18778](https://github.com/BerriAI/litellm/pull/18778) +- Add community engagement buttons - [PR #19114](https://github.com/BerriAI/litellm/pull/19114) +- UI Feedback Form - why LiteLLM - [PR #18999](https://github.com/BerriAI/litellm/pull/18999) +- Refactor user and team table filters to reusable component - [PR #19010](https://github.com/BerriAI/litellm/pull/19010) +- Adjusting new badges - [PR #19278](https://github.com/BerriAI/litellm/pull/19278) + +#### Bugs + +- Container API routes return 401 for non-admin users - routes missing from openai_routes - [PR #19115](https://github.com/BerriAI/litellm/pull/19115) +- Allow routing to regional endpoints for Containers API - [PR #19118](https://github.com/BerriAI/litellm/pull/19118) +- Fix Azure Storage circular reference error - [PR #19120](https://github.com/BerriAI/litellm/pull/19120) +- Fix prompt deletion fails with Prisma FieldNotFoundError - [PR #18966](https://github.com/BerriAI/litellm/pull/18966) + +--- + +## AI Integrations + +### Logging + +- **[OpenTelemetry](../../docs/proxy/logging#opentelemetry)** + - Update semantic conventions to 1.38 (gen_ai attributes) - [PR #18793](https://github.com/BerriAI/litellm/pull/18793) + +- **[LangSmith](../../docs/proxy/logging#langsmith)** + - Hoist thread grouping metadata (session_id, thread) - [PR #18982](https://github.com/BerriAI/litellm/pull/18982) + +- **[Langfuse](../../docs/proxy/logging#langfuse)** + - Include Langfuse logger in JSON logging when Langfuse callback is used - [PR #19162](https://github.com/BerriAI/litellm/pull/19162) + +- **[Logfire](../../docs/observability/logfire)** + - Add ability to customize Logfire base URL through env var - [PR #19148](https://github.com/BerriAI/litellm/pull/19148) + +- **General Logging** + - Enable JSON logging via configuration and add regression test - [PR #19037](https://github.com/BerriAI/litellm/pull/19037) + - Fix header forwarding for embeddings endpoint - [PR #18960](https://github.com/BerriAI/litellm/pull/18960) + - Preserve llm_provider-* headers in error responses - [PR #19020](https://github.com/BerriAI/litellm/pull/19020) + - Fix turn_off_message_logging not redacting request messages in proxy_server_request field - [PR #18897](https://github.com/BerriAI/litellm/pull/18897) + +### Guardrails + +- **[Grayswan](../../docs/proxy/guardrails/grayswan)** + - Implement fail-open option (default: True) - [PR #18266](https://github.com/BerriAI/litellm/pull/18266) + +- **[Pangea](../../docs/proxy/guardrails/pangea)** + - Respect `default_on` during initialization - [PR #18912](https://github.com/BerriAI/litellm/pull/18912) + +- **[Panw Prisma AIRS](../../docs/proxy/guardrails/panw_prisma_airs)** + - Add custom violation message support - [PR #19272](https://github.com/BerriAI/litellm/pull/19272) + +- **General Guardrails** + - Fix SerializationIterator error and pass tools to guardrail - [PR #18932](https://github.com/BerriAI/litellm/pull/18932) + - Properly handle custom guardrails parameters - [PR #18978](https://github.com/BerriAI/litellm/pull/18978) + - Use clean error messages for blocked requests - [PR #19023](https://github.com/BerriAI/litellm/pull/19023) + - Guardrail moderation support with responses API - [PR #18957](https://github.com/BerriAI/litellm/pull/18957) + - Fix model-level guardrails not taking effect - [PR #18895](https://github.com/BerriAI/litellm/pull/18895) + +--- + +## Spend Tracking, Budgets and Rate Limiting + +- **Cost Calculation Fixes** + - Include IMAGE token count in cost calculation for Gemini models - [PR #18876](https://github.com/BerriAI/litellm/pull/18876) + - Fix negative text_tokens when using cache with images - [PR #18768](https://github.com/BerriAI/litellm/pull/18768) + - Fix image tokens spend logging for `/images/generations` - [PR #19009](https://github.com/BerriAI/litellm/pull/19009) + - Fix incorrect `prompt_tokens_details` in Gemini Image Generation - [PR #19070](https://github.com/BerriAI/litellm/pull/19070) + - Fix case-insensitive model cost map lookup - [PR #18208](https://github.com/BerriAI/litellm/pull/18208) + +- **Pricing Updates** + - Correct pricing for `openrouter/openai/gpt-oss-20b` - [PR #18899](https://github.com/BerriAI/litellm/pull/18899) + - Add pricing for `azure_ai/claude-opus-4-5` - [PR #19003](https://github.com/BerriAI/litellm/pull/19003) + - Update Novita models prices - [PR #19005](https://github.com/BerriAI/litellm/pull/19005) + - Fix Azure Grok prices - [PR #19102](https://github.com/BerriAI/litellm/pull/19102) + - Fix GCP GLM-4.7 pricing - [PR #19172](https://github.com/BerriAI/litellm/pull/19172) + - Sync DeepSeek chat/reasoner to V3.2 pricing - [PR #18884](https://github.com/BerriAI/litellm/pull/18884) + - Correct cache_read pricing for gemini-2.5-pro models - [PR #18157](https://github.com/BerriAI/litellm/pull/18157) + +- **Budget & Rate Limiting** + - Correct budget limit validation operator (>=) for team members - [PR #19207](https://github.com/BerriAI/litellm/pull/19207) + - Fix TPM 25% limiting by ensuring priority queue logic - [PR #19092](https://github.com/BerriAI/litellm/pull/19092) + - Cleanup spend logs cron verification, fix, and docs - [PR #19085](https://github.com/BerriAI/litellm/pull/19085) + +--- + +## MCP Gateway + +- Prevent duplicate MCP reload scheduler registration - [PR #18934](https://github.com/BerriAI/litellm/pull/18934) +- Forward MCP extra headers case-insensitively - [PR #18940](https://github.com/BerriAI/litellm/pull/18940) +- Fix MCP REST auth checks - [PR #19051](https://github.com/BerriAI/litellm/pull/19051) +- Fix generating two telemetry events in responses - [PR #18938](https://github.com/BerriAI/litellm/pull/18938) +- Fix MCP chat completions - [PR #19129](https://github.com/BerriAI/litellm/pull/19129) + +--- + +## Performance / Loadbalancing / Reliability improvements + +- **Performance Improvements** + - Remove bottleneck causing high CPU usage & overhead under heavy load - [PR #19049](https://github.com/BerriAI/litellm/pull/19049) + - Add CI enforcement for O(1) operations in `_get_model_cost_key` to prevent performance regressions - [PR #19052](https://github.com/BerriAI/litellm/pull/19052) + - Fix Azure embeddings JSON parsing to prevent connection leaks and ensure proper router cooldown - [PR #19167](https://github.com/BerriAI/litellm/pull/19167) + - Do not fallback to token counter if `disable_token_counter` is enabled - [PR #19041](https://github.com/BerriAI/litellm/pull/19041) + +- **Reliability** + - Add fallback endpoints support - [PR #19185](https://github.com/BerriAI/litellm/pull/19185) + - Fix stream_timeout parameter functionality - [PR #19191](https://github.com/BerriAI/litellm/pull/19191) + - Fix model matching priority in configuration - [PR #19012](https://github.com/BerriAI/litellm/pull/19012) + - Fix num_retries in litellm_params as per config - [PR #18975](https://github.com/BerriAI/litellm/pull/18975) + - Handle exceptions without response parameter - [PR #18919](https://github.com/BerriAI/litellm/pull/18919) + +- **Infrastructure** + - Add Custom CA certificates to boto3 clients - [PR #18942](https://github.com/BerriAI/litellm/pull/18942) + - Update boto3 to 1.40.15 and aioboto3 to 15.5.0 - [PR #19090](https://github.com/BerriAI/litellm/pull/19090) + - Make keepalive_timeout parameter work for Gunicorn - [PR #19087](https://github.com/BerriAI/litellm/pull/19087) + +- **Helm Chart** + - Fix mount config.yaml as single file in Helm chart - [PR #19146](https://github.com/BerriAI/litellm/pull/19146) + - Sync Helm chart versioning with production standards and Docker versions - [PR #18868](https://github.com/BerriAI/litellm/pull/18868) + +--- + +## Database Changes + +### Schema Updates + +| Table | Change Type | Description | PR | +| ----- | ----------- | ----------- | -- | +| `LiteLLM_ProxyModelTable` | New Columns | Added `created_at` and `updated_at` timestamp fields | [PR #18937](https://github.com/BerriAI/litellm/pull/18937) | + +--- + +## Documentation Updates + +- Add LiteLLM architecture md doc - [PR #19057](https://github.com/BerriAI/litellm/pull/19057), [PR #19252](https://github.com/BerriAI/litellm/pull/19252) +- Add troubleshooting guide - [PR #19096](https://github.com/BerriAI/litellm/pull/19096), [PR #19097](https://github.com/BerriAI/litellm/pull/19097), [PR #19099](https://github.com/BerriAI/litellm/pull/19099) +- Add structured issue reporting guides for CPU and memory issues - [PR #19117](https://github.com/BerriAI/litellm/pull/19117) +- Add Redis requirement warning for high-traffic deployments - [PR #18892](https://github.com/BerriAI/litellm/pull/18892) +- Update load balancing and routing with enable_pre_call_checks - [PR #18888](https://github.com/BerriAI/litellm/pull/18888) +- Updated pass_through with guided param - [PR #18886](https://github.com/BerriAI/litellm/pull/18886) +- Update message content types link and add content types table - [PR #18209](https://github.com/BerriAI/litellm/pull/18209) +- Add Redis initialization with kwargs - [PR #19183](https://github.com/BerriAI/litellm/pull/19183) +- Improve documentation for routing LLM calls via SAP Gen AI Hub - [PR #19166](https://github.com/BerriAI/litellm/pull/19166) +- Deleted Keys and Teams docs - [PR #19291](https://github.com/BerriAI/litellm/pull/19291) +- Claude Code end user tracking guide - [PR #19176](https://github.com/BerriAI/litellm/pull/19176) +- Add MCP troubleshooting guide - [PR #19122](https://github.com/BerriAI/litellm/pull/19122) +- Add auth message UI documentation - [PR #19063](https://github.com/BerriAI/litellm/pull/19063) +- Add guide for mounting custom callbacks in Helm/K8s - [PR #19136](https://github.com/BerriAI/litellm/pull/19136) + +--- + +## Bug Fixes + +- Fix Swagger UI path execute error with server_root_path in OpenAPI schema - [PR #18947](https://github.com/BerriAI/litellm/pull/18947) +- Normalize OpenAI SDK BaseModel choices/messages to avoid Pydantic serializer warnings - [PR #18972](https://github.com/BerriAI/litellm/pull/18972) +- Add contextual gap checks and word-form digits - [PR #18301](https://github.com/BerriAI/litellm/pull/18301) +- Clean up orphaned files from repository root - [PR #19150](https://github.com/BerriAI/litellm/pull/19150) +- Include proxy/prisma_migration.py in non-root - [PR #18971](https://github.com/BerriAI/litellm/pull/18971) +- Update prisma_migration.py - [PR #19083](https://github.com/BerriAI/litellm/pull/19083) + +--- + +## New Contributors + +* @yogeshwaran10 made their first contribution in [PR #18898](https://github.com/BerriAI/litellm/pull/18898) +* @theonlypal made their first contribution in [PR #18937](https://github.com/BerriAI/litellm/pull/18937) +* @jonmagic made their first contribution in [PR #18935](https://github.com/BerriAI/litellm/pull/18935) +* @houdataali made their first contribution in [PR #19025](https://github.com/BerriAI/litellm/pull/19025) +* @hummat made their first contribution in [PR #18972](https://github.com/BerriAI/litellm/pull/18972) +* @berkeyalciin made their first contribution in [PR #18966](https://github.com/BerriAI/litellm/pull/18966) +* @MateuszOssGit made their first contribution in [PR #18959](https://github.com/BerriAI/litellm/pull/18959) +* @xfan001 made their first contribution in [PR #18947](https://github.com/BerriAI/litellm/pull/18947) +* @nulone made their first contribution in [PR #18884](https://github.com/BerriAI/litellm/pull/18884) +* @debnil-mercor made their first contribution in [PR #18919](https://github.com/BerriAI/litellm/pull/18919) +* @hakhundov made their first contribution in [PR #17420](https://github.com/BerriAI/litellm/pull/17420) +* @rohanwinsor made their first contribution in [PR #19078](https://github.com/BerriAI/litellm/pull/19078) +* @pgolm made their first contribution in [PR #19020](https://github.com/BerriAI/litellm/pull/19020) +* @vikigenius made their first contribution in [PR #19148](https://github.com/BerriAI/litellm/pull/19148) +* @burnerburnerburnerman made their first contribution in [PR #19090](https://github.com/BerriAI/litellm/pull/19090) +* @yfge made their first contribution in [PR #19076](https://github.com/BerriAI/litellm/pull/19076) +* @danielnyari-seon made their first contribution in [PR #19083](https://github.com/BerriAI/litellm/pull/19083) +* @guilherme-segantini made their first contribution in [PR #19166](https://github.com/BerriAI/litellm/pull/19166) +* @jgreek made their first contribution in [PR #19147](https://github.com/BerriAI/litellm/pull/19147) +* @anand-kamble made their first contribution in [PR #19193](https://github.com/BerriAI/litellm/pull/19193) +* @neubig made their first contribution in [PR #19162](https://github.com/BerriAI/litellm/pull/19162) + +--- + +## Full Changelog + +**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.80.15.rc.1...v1.81.0.rc.1)** diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index fd651c3adc5..102e3dfe1c5 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -122,6 +122,7 @@ const sidebars = { items: [ "tutorials/claude_responses_api", "tutorials/claude_code_customer_tracking", + "tutorials/claude_code_websearch", "tutorials/claude_mcp", "tutorials/claude_non_anthropic_models", ] diff --git a/litellm/constants.py b/litellm/constants.py index dba79b2f186..3bdd943481e 100644 --- a/litellm/constants.py +++ b/litellm/constants.py @@ -329,6 +329,11 @@ ANTHROPIC_WEB_SEARCH_TOOL_MAX_USES = { "medium": 5, "high": 10, } + +# LiteLLM standard web search tool name +# Used for web search interception across providers +LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search" + DEFAULT_IMAGE_ENDPOINT_MODEL = "dall-e-2" DEFAULT_VIDEO_ENDPOINT_MODEL = "sora-2" diff --git a/litellm/integrations/custom_logger.py b/litellm/integrations/custom_logger.py index 317613420a5..12243a19184 100644 --- a/litellm/integrations/custom_logger.py +++ b/litellm/integrations/custom_logger.py @@ -143,6 +143,34 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac async def async_log_pre_api_call(self, model, messages, kwargs): pass + async def async_pre_request_hook( + self, model: str, messages: List, kwargs: Dict + ) -> Optional[Dict]: + """ + Hook called before making the API request to allow modifying request parameters. + + This is specifically designed for modifying the request before it's sent to the provider. + Unlike async_log_pre_api_call (which is for logging), this hook is meant for transformations. + + Args: + model: The model name + messages: The messages list + kwargs: The request parameters (tools, stream, temperature, etc.) + + Returns: + Optional[Dict]: Modified kwargs to use for the request, or None if no modifications + + Example: + ```python + async def async_pre_request_hook(self, model, messages, kwargs): + # Convert native tools to standard format + if kwargs.get("tools"): + kwargs["tools"] = convert_tools(kwargs["tools"]) + return kwargs + ``` + """ + pass + async def async_log_success_event(self, kwargs, response_obj, start_time, end_time): pass diff --git a/litellm/integrations/prometheus.py b/litellm/integrations/prometheus.py index 1e1da803e48..b490c21174f 100644 --- a/litellm/integrations/prometheus.py +++ b/litellm/integrations/prometheus.py @@ -21,7 +21,12 @@ from typing import ( import litellm from litellm._logging import print_verbose, verbose_logger from litellm.integrations.custom_logger import CustomLogger -from litellm.proxy._types import LiteLLM_TeamTable, LiteLLM_UserTable, UserAPIKeyAuth +from litellm.proxy._types import ( + LiteLLM_DeletedVerificationToken, + LiteLLM_TeamTable, + LiteLLM_UserTable, + UserAPIKeyAuth, +) from litellm.types.integrations.prometheus import * from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name from litellm.types.utils import StandardLoggingPayload @@ -2153,7 +2158,7 @@ class PrometheusLogger(CustomLogger): self, data_fetch_function: Callable[..., Awaitable[Tuple[List[Any], Optional[int]]]], set_metrics_function: Callable[[List[Any]], Awaitable[None]], - data_type: Literal["teams", "keys"], + data_type: Literal["teams", "keys", "users"], ): """ Generic method to initialize budget metrics for teams or API keys. @@ -2245,7 +2250,7 @@ class PrometheusLogger(CustomLogger): async def fetch_keys( page_size: int, page: int - ) -> Tuple[List[Union[str, UserAPIKeyAuth]], Optional[int]]: + ) -> Tuple[List[Union[str, UserAPIKeyAuth, LiteLLM_DeletedVerificationToken]], Optional[int]]: key_list_response = await _list_key_helper( prisma_client=prisma_client, page=page, diff --git a/litellm/integrations/websearch_interception/ARCHITECTURE.md b/litellm/integrations/websearch_interception/ARCHITECTURE.md index 345741c3c03..3aa0a1558d7 100644 --- a/litellm/integrations/websearch_interception/ARCHITECTURE.md +++ b/litellm/integrations/websearch_interception/ARCHITECTURE.md @@ -7,6 +7,98 @@ Server-side WebSearch tool execution for models that don't natively support it ( User makes **ONE** `litellm.messages.acreate()` call → Gets final answer with search results. The agentic loop happens transparently on the server. +## LiteLLM Standard Web Search Tool + +LiteLLM defines a standard web search tool format (`litellm_web_search`) that all native provider tools are converted to. This enables consistent interception across providers. + +**Standard Tool Definition** (defined in `tools.py`): +```python +{ + "name": "litellm_web_search", + "description": "Search the web for information...", + "input_schema": { + "type": "object", + "properties": { + "query": {"type": "string", "description": "The search query"} + }, + "required": ["query"] + } +} +``` + +**Tool Name Constant**: `LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"` (defined in `litellm/constants.py`) + +### Supported Tool Formats + +The interception system automatically detects and handles: + +| Tool Format | Example | Provider | Detection Method | Future-Proof | +|-------------|---------|----------|------------------|-------------| +| **LiteLLM Standard** | `name="litellm_web_search"` | Any | Direct name match | N/A | +| **Anthropic Native** | `type="web_search_20250305"` | Bedrock, Claude API | Type prefix: `startswith("web_search_")` | ✅ Yes (web_search_2026, etc.) | +| **Claude Code CLI** | `name="web_search"`, `type="web_search_20250305"` | Claude Code | Name + type check | ✅ Yes (version-agnostic) | +| **Legacy** | `name="WebSearch"` | Custom | Name match | N/A (backwards compat) | + +**Future Compatibility**: The `startswith("web_search_")` check in `tools.py` automatically supports future Anthropic web search versions. + +### Claude Code CLI Integration + +Claude Code (Anthropic's official CLI) sends web search requests using Anthropic's native tool format: + +```python +{ + "type": "web_search_20250305", + "name": "web_search", + "max_uses": 8 +} +``` + +**What Happens:** +1. Claude Code sends native `web_search_20250305` tool to LiteLLM proxy +2. LiteLLM intercepts and converts to `litellm_web_search` standard format +3. Bedrock receives converted tool (NOT native format) +4. Model returns `tool_use` block for `litellm_web_search` (not `server_tool_use`) +5. LiteLLM's agentic loop intercepts the `tool_use` +6. Executes `litellm.asearch()` using configured provider (Perplexity, Tavily, etc.) +7. Returns final answer to Claude Code user + +**Without Interception**: Bedrock would receive native tool → try to execute natively → return `web_search_tool_result_error` with `invalid_tool_input` + +**With Interception**: LiteLLM converts → Bedrock returns tool_use → LiteLLM executes search → Returns final answer ✅ + +### Native Tool Conversion + +Native tools are converted to LiteLLM standard format **before** sending to the provider: + +1. **Conversion Point** (`litellm/llms/anthropic/experimental_pass_through/messages/handler.py`): + - In `anthropic_messages()` function (lines 60-127) + - Runs BEFORE the API request is made + - Detects native web search tools using `is_web_search_tool()` + - Converts to `litellm_web_search` format using `get_litellm_web_search_tool()` + - Prevents provider from executing search natively (avoids `web_search_tool_result_error`) + +2. **Response Detection** (`transformation.py`): + - Detects `tool_use` blocks with any web search tool name + - Handles: `litellm_web_search`, `WebSearch`, `web_search` + - Extracts search queries for execution + +**Example Conversion**: +```python +# Input (Claude Code's native tool) +{ + "type": "web_search_20250305", + "name": "web_search", + "max_uses": 8 +} + +# Output (LiteLLM standard) +{ + "name": "litellm_web_search", + "description": "Search the web for information...", + "input_schema": {...} +} +``` + --- ## Request Flow @@ -63,6 +155,9 @@ sequenceDiagram | Component | File | Purpose | |-----------|------|---------| | **WebSearchInterceptionLogger** | `handler.py` | CustomLogger that implements agentic loop hooks | +| **Tool Standardization** | `tools.py` | Standard tool definition, detection, and utilities | +| **Tool Name Constant** | `constants.py` | `LITELLM_WEB_SEARCH_TOOL_NAME = "litellm_web_search"` | +| **Tool Conversion** | `anthropic/.../ handler.py` | Converts native tools to LiteLLM standard before API call | | **Transformation Logic** | `transformation.py` | Detect tool_use, build tool_result messages, format search responses | | **Agentic Loop Hooks** | `integrations/custom_logger.py` | Base hooks: `async_should_run_agentic_loop()`, `async_run_agentic_loop()` | | **Hook Orchestration** | `llms/custom_httpx/llm_http_handler.py` | `_call_agentic_completion_hooks()` - calls hooks after response | @@ -74,7 +169,10 @@ sequenceDiagram ## Configuration ```python -from litellm.integrations.websearch_interception import WebSearchInterceptionLogger +from litellm.integrations.websearch_interception import ( + WebSearchInterceptionLogger, + get_litellm_web_search_tool, +) from litellm.types.utils import LlmProviders # Enable for Bedrock with specific search tool @@ -85,13 +183,25 @@ litellm.callbacks = [ ) ] -# Make request (streaming or non-streaming both work) +# Make request with LiteLLM standard tool (recommended) response = await litellm.messages.acreate( - model="bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", + model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", messages=[{"role": "user", "content": "What is LiteLLM?"}], - tools=[{"name": "WebSearch", ...}], + tools=[get_litellm_web_search_tool()], # LiteLLM standard + max_tokens=1024, + stream=True # Auto-converted to non-streaming +) + +# OR send native tools - they're auto-converted to LiteLLM standard +response = await litellm.messages.acreate( + model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + messages=[{"role": "user", "content": "What is LiteLLM?"}], + tools=[{ + "type": "web_search_20250305", # Native Anthropic format + "name": "web_search", + "max_uses": 8 + }], max_tokens=1024, - stream=True # Streaming is automatically converted to non-streaming for WebSearch ) ``` diff --git a/litellm/integrations/websearch_interception/__init__.py b/litellm/integrations/websearch_interception/__init__.py index c0feb5235e2..f5b1963c1cf 100644 --- a/litellm/integrations/websearch_interception/__init__.py +++ b/litellm/integrations/websearch_interception/__init__.py @@ -8,5 +8,13 @@ support server-side tool calling (e.g., Bedrock/Claude). from litellm.integrations.websearch_interception.handler import ( WebSearchInterceptionLogger, ) +from litellm.integrations.websearch_interception.tools import ( + get_litellm_web_search_tool, + is_web_search_tool, +) -__all__ = ["WebSearchInterceptionLogger"] +__all__ = [ + "WebSearchInterceptionLogger", + "get_litellm_web_search_tool", + "is_web_search_tool", +] diff --git a/litellm/integrations/websearch_interception/handler.py b/litellm/integrations/websearch_interception/handler.py index 0b08bc2312a..943a2bb4f36 100644 --- a/litellm/integrations/websearch_interception/handler.py +++ b/litellm/integrations/websearch_interception/handler.py @@ -12,7 +12,12 @@ from typing import Any, Dict, List, Optional, Tuple, Union, cast import litellm from litellm._logging import verbose_logger from litellm.anthropic_interface import messages as anthropic_messages +from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME from litellm.integrations.custom_logger import CustomLogger +from litellm.integrations.websearch_interception.tools import ( + get_litellm_web_search_tool, + is_web_search_tool, +) from litellm.integrations.websearch_interception.transformation import ( WebSearchTransformation, ) @@ -57,6 +62,55 @@ class WebSearchInterceptionLogger(CustomLogger): for p in enabled_providers ] self.search_tool_name = search_tool_name + self._request_has_websearch = False # Track if current request has web search + + async def async_pre_call_deployment_hook( + self, kwargs: Dict[str, Any], call_type: Optional[Any] + ) -> Optional[dict]: + """ + Pre-call hook to convert native Anthropic web_search tools to regular tools. + + This prevents Bedrock from trying to execute web search server-side (which fails). + Instead, we convert it to a regular tool so the model returns tool_use blocks + that we can intercept and execute ourselves. + """ + # Check if this is for an enabled provider + custom_llm_provider = kwargs.get("litellm_params", {}).get("custom_llm_provider", "") + if custom_llm_provider not in self.enabled_providers: + return None + + # Check if request has tools with native web_search + tools = kwargs.get("tools") + if not tools: + return None + + # Check if any tool is a web search tool (native or already LiteLLM standard) + has_websearch = any(is_web_search_tool(t) for t in tools) + + if not has_websearch: + return None + + verbose_logger.debug( + "WebSearchInterception: Converting native web_search tools to LiteLLM standard" + ) + + # Convert native/custom web_search tools to LiteLLM standard + converted_tools = [] + for tool in tools: + if is_web_search_tool(tool): + # Convert to LiteLLM standard web search tool + converted_tool = get_litellm_web_search_tool() + converted_tools.append(converted_tool) + verbose_logger.debug( + f"WebSearchInterception: Converted {tool.get('name', 'unknown')} " + f"(type={tool.get('type', 'none')}) to {LITELLM_WEB_SEARCH_TOOL_NAME}" + ) + else: + # Keep other tools as-is + converted_tools.append(tool) + + # Return modified kwargs with converted tools + return {"tools": converted_tools} @classmethod def from_config_yaml( @@ -104,6 +158,83 @@ class WebSearchInterceptionLogger(CustomLogger): search_tool_name=search_tool_name, ) + async def async_pre_request_hook( + self, model: str, messages: List[Dict], kwargs: Dict + ) -> Optional[Dict]: + """ + Pre-request hook to convert native web search tools to LiteLLM standard. + + This hook is called before the API request is made, allowing us to: + 1. Detect native web search tools (web_search_20250305, etc.) + 2. Convert them to LiteLLM standard format (litellm_web_search) + 3. Convert stream=True to stream=False for interception + + This prevents providers like Bedrock from trying to execute web search + natively (which fails), and ensures our agentic loop can intercept tool_use. + + Returns: + Modified kwargs dict with converted tools, or None if no modifications needed + """ + # Check if this request is for an enabled provider + custom_llm_provider = kwargs.get("litellm_params", {}).get( + "custom_llm_provider", "" + ) + + verbose_logger.debug( + f"WebSearchInterception: Pre-request hook called" + f" - custom_llm_provider={custom_llm_provider}" + f" - enabled_providers={self.enabled_providers}" + ) + + if custom_llm_provider not in self.enabled_providers: + verbose_logger.debug( + f"WebSearchInterception: Skipping - provider {custom_llm_provider} not in {self.enabled_providers}" + ) + return None + + # Check if request has tools + tools = kwargs.get("tools") + if not tools: + return None + + # Check if any tool is a web search tool + has_websearch = any(is_web_search_tool(t) for t in tools) + if not has_websearch: + return None + + verbose_logger.debug( + f"WebSearchInterception: Pre-request hook triggered for provider={custom_llm_provider}" + ) + + # Convert native web search tools to LiteLLM standard + converted_tools = [] + for tool in tools: + if is_web_search_tool(tool): + standard_tool = get_litellm_web_search_tool() + converted_tools.append(standard_tool) + verbose_logger.debug( + f"WebSearchInterception: Converted {tool.get('name', 'unknown')} " + f"(type={tool.get('type', 'none')}) to {LITELLM_WEB_SEARCH_TOOL_NAME}" + ) + else: + converted_tools.append(tool) + + # Update kwargs with converted tools + kwargs["tools"] = converted_tools + verbose_logger.debug( + f"WebSearchInterception: Tools after conversion: {[t.get('name') for t in converted_tools]}" + ) + + # Convert stream=True to stream=False for WebSearch interception + if kwargs.get("stream"): + verbose_logger.debug( + "WebSearchInterception: Converting stream=True to stream=False" + ) + kwargs["stream"] = False + kwargs["_websearch_interception_converted_stream"] = True + + return kwargs + async def async_should_run_agentic_loop( self, response: Any, @@ -128,11 +259,11 @@ class WebSearchInterceptionLogger(CustomLogger): ) return False, {} - # Check if tools include WebSearch - has_websearch_tool = any(t.get("name") == "WebSearch" for t in (tools or [])) + # Check if tools include any web search tool (LiteLLM standard or native) + has_websearch_tool = any(is_web_search_tool(t) for t in (tools or [])) if not has_websearch_tool: verbose_logger.debug( - "WebSearchInterception: No WebSearch tool in request" + "WebSearchInterception: No web search tool in request" ) return False, {} diff --git a/litellm/integrations/websearch_interception/tools.py b/litellm/integrations/websearch_interception/tools.py new file mode 100644 index 00000000000..4f8b7372fe3 --- /dev/null +++ b/litellm/integrations/websearch_interception/tools.py @@ -0,0 +1,95 @@ +""" +LiteLLM Web Search Tool Definition + +This module defines the standard web search tool used across LiteLLM. +Native provider tools (like Anthropic's web_search_20250305) are converted +to this format for consistent interception and execution. +""" + +from typing import Any, Dict + +from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME + + +def get_litellm_web_search_tool() -> Dict[str, Any]: + """ + Get the standard LiteLLM web search tool definition. + + This is the canonical tool definition that all native web search tools + (like Anthropic's web_search_20250305, Claude Code's web_search, etc.) + are converted to for interception. + + Returns: + Dict containing the Anthropic-style tool definition with: + - name: Tool name + - description: What the tool does + - input_schema: JSON schema for tool parameters + + Example: + >>> tool = get_litellm_web_search_tool() + >>> tool['name'] + 'litellm_web_search' + """ + return { + "name": LITELLM_WEB_SEARCH_TOOL_NAME, + "description": ( + "Search the web for information. Use this when you need current " + "information or answers to questions that require up-to-date data." + ), + "input_schema": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": "The search query to execute" + } + }, + "required": ["query"] + } + } + + +def is_web_search_tool(tool: Dict[str, Any]) -> bool: + """ + Check if a tool is a web search tool (native or LiteLLM standard). + + Detects: + - LiteLLM standard: name == "litellm_web_search" + - Anthropic native: type starts with "web_search_" (e.g., "web_search_20250305") + - Claude Code: name == "web_search" with a type field + - Custom: name == "WebSearch" (legacy format) + + Args: + tool: Tool dictionary to check + + Returns: + True if tool is a web search tool + + Example: + >>> is_web_search_tool({"name": "litellm_web_search"}) + True + >>> is_web_search_tool({"type": "web_search_20250305", "name": "web_search"}) + True + >>> is_web_search_tool({"name": "calculator"}) + False + """ + tool_name = tool.get("name", "") + tool_type = tool.get("type", "") + + # Check for LiteLLM standard tool + if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME: + return True + + # Check for native Anthropic web_search_* types + if tool_type.startswith("web_search_"): + return True + + # Check for Claude Code's web_search with a type field + if tool_name == "web_search" and tool_type: + return True + + # Check for legacy WebSearch format + if tool_name == "WebSearch": + return True + + return False diff --git a/litellm/integrations/websearch_interception/transformation.py b/litellm/integrations/websearch_interception/transformation.py index e8211311281..313358822a5 100644 --- a/litellm/integrations/websearch_interception/transformation.py +++ b/litellm/integrations/websearch_interception/transformation.py @@ -7,6 +7,7 @@ Transforms between Anthropic tool_use format and LiteLLM search format. from typing import Any, Dict, List, Tuple from litellm._logging import verbose_logger +from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME from litellm.llms.base_llm.search.transformation import SearchResponse @@ -94,17 +95,21 @@ class WebSearchTransformation: block_id = getattr(block, "id", None) block_input = getattr(block, "input", {}) - if block_type == "tool_use" and block_name == "WebSearch": + # Check for LiteLLM standard or legacy web search tools + # Handles: litellm_web_search, WebSearch, web_search + if block_type == "tool_use" and block_name in ( + LITELLM_WEB_SEARCH_TOOL_NAME, "WebSearch", "web_search" + ): # Convert to dict for easier handling tool_call = { "id": block_id, "type": "tool_use", - "name": "WebSearch", + "name": block_name, # Preserve original name "input": block_input, } tool_calls.append(tool_call) verbose_logger.debug( - f"WebSearchInterception: Found WebSearch tool_use with id={tool_call['id']}" + f"WebSearchInterception: Found {block_name} tool_use with id={tool_call['id']}" ) return len(tool_calls) > 0, tool_calls diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py b/litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py new file mode 100644 index 00000000000..542ae20b602 --- /dev/null +++ b/litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py @@ -0,0 +1,246 @@ +""" +Fake Streaming Iterator for Anthropic Messages + +This module provides a fake streaming iterator that converts non-streaming +Anthropic Messages responses into proper streaming format. + +Used when WebSearch interception converts stream=True to stream=False but +the LLM doesn't make a tool call, and we need to return a stream to the user. +""" + +import json +from typing import Any, Dict, List, cast + +from litellm.types.llms.anthropic_messages.anthropic_response import ( + AnthropicMessagesResponse, +) + + +class FakeAnthropicMessagesStreamIterator: + """ + Fake streaming iterator for Anthropic Messages responses. + + Used when we need to convert a non-streaming response to a streaming format, + such as when WebSearch interception converts stream=True to stream=False but + the LLM doesn't make a tool call. + + This creates a proper Anthropic-style streaming response with multiple events: + - message_start + - content_block_start (for each content block) + - content_block_delta (for text content, chunked) + - content_block_stop + - message_delta (for usage) + - message_stop + """ + + def __init__(self, response: AnthropicMessagesResponse): + self.response = response + self.chunks = self._create_streaming_chunks() + self.current_index = 0 + + def _create_streaming_chunks(self) -> List[bytes]: + """Convert the non-streaming response to streaming chunks""" + chunks = [] + + # Cast response to dict for easier access + response_dict = cast(Dict[str, Any], self.response) + + # 1. message_start event + usage = response_dict.get("usage", {}) + message_start = { + "type": "message_start", + "message": { + "id": response_dict.get("id"), + "type": "message", + "role": response_dict.get("role", "assistant"), + "model": response_dict.get("model"), + "content": [], + "stop_reason": None, + "stop_sequence": None, + "usage": { + "input_tokens": usage.get("input_tokens", 0) if usage else 0, + "output_tokens": 0 + } + } + } + chunks.append(f"event: message_start\ndata: {json.dumps(message_start)}\n\n".encode()) + + # 2-4. For each content block, send start/delta/stop events + content_blocks = response_dict.get("content", []) + if content_blocks: + for index, block in enumerate(content_blocks): + # Cast block to dict for easier access + block_dict = cast(Dict[str, Any], block) + block_type = block_dict.get("type") + + if block_type == "text": + # content_block_start + content_block_start = { + "type": "content_block_start", + "index": index, + "content_block": { + "type": "text", + "text": "" + } + } + chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()) + + # content_block_delta (send full text as one delta for simplicity) + text = block_dict.get("text", "") + content_block_delta = { + "type": "content_block_delta", + "index": index, + "delta": { + "type": "text_delta", + "text": text + } + } + chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()) + + # content_block_stop + content_block_stop = { + "type": "content_block_stop", + "index": index + } + chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()) + + elif block_type == "thinking": + # content_block_start for thinking + content_block_start = { + "type": "content_block_start", + "index": index, + "content_block": { + "type": "thinking", + "thinking": "", + "signature": "" + } + } + chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()) + + # content_block_delta for thinking text + thinking_text = block_dict.get("thinking", "") + if thinking_text: + content_block_delta = { + "type": "content_block_delta", + "index": index, + "delta": { + "type": "thinking_delta", + "thinking": thinking_text + } + } + chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()) + + # content_block_delta for signature (if present) + signature = block_dict.get("signature", "") + if signature: + signature_delta = { + "type": "content_block_delta", + "index": index, + "delta": { + "type": "signature_delta", + "signature": signature + } + } + chunks.append(f"event: content_block_delta\ndata: {json.dumps(signature_delta)}\n\n".encode()) + + # content_block_stop + content_block_stop = { + "type": "content_block_stop", + "index": index + } + chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()) + + elif block_type == "redacted_thinking": + # content_block_start for redacted_thinking + content_block_start = { + "type": "content_block_start", + "index": index, + "content_block": { + "type": "redacted_thinking" + } + } + chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()) + + # content_block_stop (no delta for redacted thinking) + content_block_stop = { + "type": "content_block_stop", + "index": index + } + chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()) + + elif block_type == "tool_use": + # content_block_start + content_block_start = { + "type": "content_block_start", + "index": index, + "content_block": { + "type": "tool_use", + "id": block_dict.get("id"), + "name": block_dict.get("name"), + "input": {} + } + } + chunks.append(f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()) + + # content_block_delta (send input as JSON delta) + input_data = block_dict.get("input", {}) + content_block_delta = { + "type": "content_block_delta", + "index": index, + "delta": { + "type": "input_json_delta", + "partial_json": json.dumps(input_data) + } + } + chunks.append(f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()) + + # content_block_stop + content_block_stop = { + "type": "content_block_stop", + "index": index + } + chunks.append(f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()) + + # 5. message_delta event (with final usage and stop_reason) + message_delta = { + "type": "message_delta", + "delta": { + "stop_reason": response_dict.get("stop_reason"), + "stop_sequence": response_dict.get("stop_sequence") + }, + "usage": { + "output_tokens": usage.get("output_tokens", 0) if usage else 0 + } + } + chunks.append(f"event: message_delta\ndata: {json.dumps(message_delta)}\n\n".encode()) + + # 6. message_stop event + message_stop = { + "type": "message_stop", + "usage": usage if usage else {} + } + chunks.append(f"event: message_stop\ndata: {json.dumps(message_stop)}\n\n".encode()) + + return chunks + + def __aiter__(self): + return self + + async def __anext__(self): + if self.current_index >= len(self.chunks): + raise StopAsyncIteration + + chunk = self.chunks[self.current_index] + self.current_index += 1 + return chunk + + def __iter__(self): + return self + + def __next__(self): + if self.current_index >= len(self.chunks): + raise StopIteration + + chunk = self.chunks[self.current_index] + self.current_index += 1 + return chunk diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/handler.py b/litellm/llms/anthropic/experimental_pass_through/messages/handler.py index 11245b1bdba..7e5a4f22a7f 100644 --- a/litellm/llms/anthropic/experimental_pass_through/messages/handler.py +++ b/litellm/llms/anthropic/experimental_pass_through/messages/handler.py @@ -33,6 +33,70 @@ base_llm_http_handler = BaseLLMHTTPHandler() ################################################# +async def _execute_pre_request_hooks( + model: str, + messages: List[Dict], + tools: Optional[List[Dict]], + stream: Optional[bool], + custom_llm_provider: Optional[str], + **kwargs, +) -> Dict: + """ + Execute pre-request hooks from CustomLogger callbacks. + + Allows CustomLoggers to modify request parameters before the API call. + Used for WebSearch tool conversion, stream modification, etc. + + Args: + model: Model name + messages: List of messages + tools: Optional tools list + stream: Optional stream flag + custom_llm_provider: Provider name (if not set, will be extracted from model) + **kwargs: Additional request parameters + + Returns: + Dict containing all (potentially modified) request parameters including tools, stream + """ + # If custom_llm_provider not provided, extract from model + if not custom_llm_provider: + try: + _, custom_llm_provider, _, _ = litellm.get_llm_provider(model=model) + except Exception: + # If extraction fails, continue without provider + pass + + # Build complete request kwargs dict + request_kwargs = { + "tools": tools, + "stream": stream, + "litellm_params": { + "custom_llm_provider": custom_llm_provider, + }, + **kwargs, + } + + if not litellm.callbacks: + return request_kwargs + + from litellm.integrations.custom_logger import CustomLogger as _CustomLogger + + for callback in litellm.callbacks: + if not isinstance(callback, _CustomLogger): + continue + + # Call the pre-request hook + modified_kwargs = await callback.async_pre_request_hook( + model, messages, request_kwargs + ) + + # If hook returned modified kwargs, use them + if modified_kwargs is not None: + request_kwargs = modified_kwargs + + return request_kwargs + + @client async def anthropic_messages( max_tokens: int, @@ -57,39 +121,24 @@ async def anthropic_messages( """ Async: Make llm api request in Anthropic /messages API spec """ - # WebSearch Interception: Convert stream=True to stream=False if WebSearch interception is enabled - # This allows transparent server-side agentic loop execution for streaming requests - if stream and tools and any(t.get("name") == "WebSearch" for t in tools): - # Extract provider using litellm's helper function - try: - _, provider, _, _ = litellm.get_llm_provider( - model=model, - custom_llm_provider=custom_llm_provider, - api_base=api_base, - api_key=api_key, - ) - except Exception: - # Fallback to simple split if helper fails - provider = model.split("/")[0] if "/" in model else "" + # Execute pre-request hooks to allow CustomLoggers to modify request + request_kwargs = await _execute_pre_request_hooks( + model=model, + messages=messages, + tools=tools, + stream=stream, + custom_llm_provider=custom_llm_provider, + **kwargs, + ) - # Check if WebSearch interception is enabled in callbacks - from litellm._logging import verbose_logger - from litellm.integrations.websearch_interception import ( - WebSearchInterceptionLogger, - ) - if litellm.callbacks: - for callback in litellm.callbacks: - if isinstance(callback, WebSearchInterceptionLogger): - # Check if provider is enabled for interception - if provider in callback.enabled_providers: - verbose_logger.debug( - f"WebSearchInterception: Converting stream=True to stream=False for WebSearch interception " - f"(provider={provider})" - ) - stream = False - break + # Extract modified parameters + tools = request_kwargs.pop("tools", tools) + stream = request_kwargs.pop("stream", stream) + # Remove litellm_params from kwargs (only needed for hooks) + request_kwargs.pop("litellm_params", None) + # Merge back any other modifications + kwargs.update(request_kwargs) - local_vars = locals() loop = asyncio.get_event_loop() kwargs["is_async"] = True @@ -206,6 +255,11 @@ def anthropic_messages_handler( "model": original_model, "custom_llm_provider": custom_llm_provider, } + + # Check if stream was converted for WebSearch interception + # This is set in the async wrapper above when stream=True is converted to stream=False + if kwargs.get("_websearch_interception_converted_stream", False): + litellm_logging_obj.model_call_details["websearch_interception_converted_stream"] = True if litellm_params.mock_response and isinstance(litellm_params.mock_response, str): diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py index 490786155c6..ab1e735fca7 100644 --- a/litellm/llms/custom_httpx/llm_http_handler.py +++ b/litellm/llms/custom_httpx/llm_http_handler.py @@ -4418,6 +4418,41 @@ class BaseLLMHTTPHandler: f"LiteLLM.AgenticHookError: Exception in agentic completion hooks: {str(e)}" ) + # Check if we need to convert response to fake stream + # This happens when: + # 1. Stream was originally True but converted to False for WebSearch interception + # 2. No agentic loop ran (LLM didn't use the tool) + # 3. We have a non-streaming response that needs to be converted to streaming + websearch_converted_stream = ( + logging_obj.model_call_details.get("websearch_interception_converted_stream", False) + if logging_obj is not None + else False + ) + + if websearch_converted_stream: + from typing import cast + + from litellm._logging import verbose_logger + from litellm.llms.anthropic.experimental_pass_through.messages.fake_stream_iterator import ( + FakeAnthropicMessagesStreamIterator, + ) + from litellm.types.llms.anthropic_messages.anthropic_response import ( + AnthropicMessagesResponse, + ) + + verbose_logger.debug( + "WebSearchInterception: No tool call made, converting non-streaming response to fake stream" + ) + + # Convert the non-streaming response to a fake stream + # The response should be an AnthropicMessagesResponse (dict) + if isinstance(response, dict): + # Create a fake streaming iterator + fake_stream = FakeAnthropicMessagesStreamIterator( + response=cast(AnthropicMessagesResponse, response) + ) + return fake_stream + return None def _handle_error( diff --git a/litellm/proxy/proxy_config.yaml b/litellm/proxy/proxy_config.yaml index 87e02a142ee..cf852805f83 100644 --- a/litellm/proxy/proxy_config.yaml +++ b/litellm/proxy/proxy_config.yaml @@ -46,7 +46,21 @@ model_list: api_base: https://krish-mh44t553-eastus2.services.ai.azure.com api_key: os.environ/AZURE_ANTHROPIC_API_KEY +# Search Tools Configuration - Define search providers for WebSearch interception +# search_tools: +# - search_tool_name: "my-perplexity-search" +# litellm_params: +# search_provider: "perplexity" # Can be: perplexity, brave, etc. + +litellm_settings: + callbacks: ["websearch_interception"] + # WebSearch Interception - Automatically intercepts and executes WebSearch tool calls + # for models that don't natively support web search (e.g., Bedrock/Claude) + websearch_interception_params: + enabled_providers: ["bedrock"] # List of providers to enable interception for + search_tool_name: "my-perplexity-search" # Optional: Name of search tool from search_tools config general_settings: store_prompts_in_spend_logs: true - forward_client_headers_to_llm_api: true \ No newline at end of file + forward_client_headers_to_llm_api: true + diff --git a/poetry.lock b/poetry.lock index 3bafdb157ca..b76f9c30f00 100644 --- a/poetry.lock +++ b/poetry.lock @@ -3081,15 +3081,15 @@ files = [ [[package]] name = "litellm-proxy-extras" -version = "0.4.21" +version = "0.4.23" description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package." optional = true python-versions = "!=2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,>=3.8" groups = ["main"] markers = "extra == \"proxy\"" files = [ - {file = "litellm_proxy_extras-0.4.21-py3-none-any.whl", hash = "sha256:83a1734e9773610945230606012e602bbcbfba1c60fde836d51102c1a296f166"}, - {file = "litellm_proxy_extras-0.4.21.tar.gz", hash = "sha256:fa0e012984aa8e5114f88f4bad53d6abb589e5ca3eab445f74f8ddeceb62d848"}, + {file = "litellm_proxy_extras-0.4.23-py3-none-any.whl", hash = "sha256:dfda21203dde9fd97cf364396a9b5be0cfdf00fa9846439ee33ce11b7a52f9ce"}, + {file = "litellm_proxy_extras-0.4.23.tar.gz", hash = "sha256:8e3f95576dc2a296e7f73d8c87e73628bd899b4644c45863960fe3c3762d8f64"}, ] [[package]] @@ -7981,4 +7981,4 @@ utils = ["numpydoc"] [metadata] lock-version = "2.1" python-versions = ">=3.9,<4.0" -content-hash = "ea62b77c662ab9fc486e421c576f0868bcde16d62a24703ee1f4916a0465ffb2" +content-hash = "2d6b3d8d44919c29315b5e645befbf745a276714a2454c563d460a6a001b90af" diff --git a/pyproject.toml b/pyproject.toml index 53f8cbb22f2..d6242b27786 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "litellm" -version = "1.80.17" +version = "1.81.0" description = "Library to easily interface with LLM API providers" authors = ["BerriAI"] license = "MIT" @@ -167,7 +167,7 @@ requires = ["poetry-core", "wheel"] build-backend = "poetry.core.masonry.api" [tool.commitizen] -version = "1.80.17" +version = "1.81.0" version_files = [ "pyproject.toml:^version" ] diff --git a/tests/llm_translation/test_aiohttp_openai.py b/tests/llm_translation/test_aiohttp_openai.py deleted file mode 100644 index 5b92c924ec7..00000000000 --- a/tests/llm_translation/test_aiohttp_openai.py +++ /dev/null @@ -1,33 +0,0 @@ -import json -import os -import sys -from datetime import datetime -import pytest - -sys.path.insert( - 0, os.path.abspath("../../") -) # Adds the parent directory to the system-path - -import litellm - - -@pytest.mark.asyncio() -async def test_aiohttp_openai(): - litellm.set_verbose = True - response = await litellm.acompletion( - model="aiohttp_openai/fake-model", - messages=[{"role": "user", "content": "Hello, world!"}], - api_base="https://exampleopenaiendpoint-production.up.railway.app/v1/chat/completions", - api_key="fake-key", - ) - print(response) - - -@pytest.mark.asyncio() -async def test_aiohttp_openai_gpt_4o(): - litellm.set_verbose = True - response = await litellm.acompletion( - model="aiohttp_openai/gpt-4o", - messages=[{"role": "user", "content": "Hello, world!"}], - ) - print(response) diff --git a/tests/llm_translation/test_deepseek_completion.py b/tests/llm_translation/test_deepseek_completion.py index 6c27ea59d4a..79a18655980 100644 --- a/tests/llm_translation/test_deepseek_completion.py +++ b/tests/llm_translation/test_deepseek_completion.py @@ -30,6 +30,7 @@ def test_deepseek_mock_completion(stream): messages=[{"role": "user", "content": "Hello, world!"}], api_base="https://exampleopenaiendpoint-production.up.railway.app/v1/chat/completions", stream=stream, + mock_response="Hello! How can I help you today?", ) print(f"response: {response}") if stream: diff --git a/tests/local_testing/test_router_fallbacks.py b/tests/local_testing/test_router_fallbacks.py index 4ae2ca19d92..3a634f6aa37 100644 --- a/tests/local_testing/test_router_fallbacks.py +++ b/tests/local_testing/test_router_fallbacks.py @@ -1358,9 +1358,10 @@ def test_router_fallbacks_with_custom_model_costs(): "model_name": "claude-sonnet-4-5-20250929", "litellm_params": { "model": "claude-sonnet-4-5-20250929", - "api_key": os.environ["ANTHROPIC_API_KEY"], + "api_key": os.environ.get("ANTHROPIC_API_KEY", "fake-key"), "input_cost_per_token": 30, "output_cost_per_token": 60, + "mock_response": "Hello! How can I help you today?", }, }, { @@ -1371,6 +1372,7 @@ def test_router_fallbacks_with_custom_model_costs(): "output_cost_per_token": 0.000015, # 15$/M "api_base": "https://exampleopenaiendpoint-production.up.railway.app", "api_key": "my-fake-key", + "mock_response": "Hello! How can I help you today?", }, }, ] diff --git a/tests/pass_through_unit_tests/test_websearch_interception_e2e.py b/tests/pass_through_unit_tests/test_websearch_interception_e2e.py index 2dec9da8b70..bf50c1c9cd2 100644 --- a/tests/pass_through_unit_tests/test_websearch_interception_e2e.py +++ b/tests/pass_through_unit_tests/test_websearch_interception_e2e.py @@ -323,3 +323,632 @@ async def test_websearch_interception_streaming(): import traceback traceback.print_exc() return False + + +async def test_websearch_interception_no_tool_call_streaming(): + """ + Test WebSearch interception when LLM doesn't make a tool call with streaming. + + This tests the scenario where: + 1. User requests stream=True + 2. WebSearch tool is provided + 3. LLM decides NOT to use the tool (just responds with text) + 4. System should return a fake stream + """ + print("\n" + "="*80) + print("E2E TEST 3: WebSearch Interception (No Tool Call, Streaming)") + print("="*80) + + # Router already initialized from test 1 + print("\n✅ Using existing router configuration") + print("✅ WebSearch interception already enabled for Bedrock") + + try: + # Make request with WebSearch tool AND stream=True + # Use a query that the LLM will answer directly without using the tool + print("\n📞 Making litellm.messages.acreate() call with stream=True...") + print(f" Model: bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0") + print(f" Query: 'What is 2+2?'") + print(f" Tools: WebSearch") + print(f" Stream: True") + + response = await messages.acreate( + model="bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", + messages=[{"role": "user", "content": "What is 2+2? Just give me the answer, no need to search."}], + tools=[ + { + "name": "WebSearch", + "description": "Search the web for information", + "input_schema": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": "The search query", + } + }, + "required": ["query"], + }, + } + ], + max_tokens=1024, + stream=True, # REQUEST STREAMING + ) + + print("\n✅ Received response!") + + # Check if response is actually a stream (async generator or async iterator) + import inspect + is_async_gen = inspect.isasyncgen(response) + is_async_iter = hasattr(response, '__aiter__') and hasattr(response, '__anext__') + is_stream = is_async_gen or is_async_iter + + if not is_stream: + print("\n❌ TEST 3 FAILED: Response is NOT a stream") + print(f"❌ Expected a fake stream when LLM doesn't use the tool") + print(f"❌ Response type: {type(response)}") + return False + + print(f"✅ Response is a stream (async_gen={is_async_gen}, async_iter={is_async_iter})") + print("\n📦 Consuming stream chunks:") + + chunks = [] + chunk_count = 0 + async for chunk in response: + chunk_count += 1 + print(f"\n--- Chunk {chunk_count} ---") + print(f" Type: {type(chunk)}") + print(f" Content: {chunk[:200] if isinstance(chunk, bytes) else str(chunk)[:200]}...") + chunks.append(chunk) + + print(f"\n✅ Received {len(chunks)} stream chunk(s)") + + if len(chunks) > 0: + print("\n" + "="*80) + print("✅ TEST 3 PASSED!") + print("="*80) + print("✅ User made ONE litellm.messages.acreate() call with stream=True") + print("✅ LLM didn't use the WebSearch tool") + print("✅ Got back a fake stream (not a non-streaming response)") + print("✅ WebSearch interception handles no-tool-call case correctly!") + print("="*80) + return True + else: + print("\n❌ TEST 3 FAILED: No chunks received") + return False + + except Exception as e: + print(f"\n❌ Test 3 failed with error: {str(e)}") + import traceback + traceback.print_exc() + return False + + +async def test_claude_code_native_websearch(): + """ + Test WebSearch interception with Claude Code's native web_search_20250305 tool. + + This tests the exact request format that Claude Code sends: + - tools: [{'type': 'web_search_20250305', 'name': 'web_search', 'max_uses': 8}] + - Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0 + """ + print("\n" + "="*80) + print("E2E TEST: Claude Code Native WebSearch (web_search_20250305)") + print("="*80) + + # Router already initialized from test 1 + print("\n✅ Using existing router configuration") + print("✅ WebSearch interception already enabled for Bedrock") + + try: + # Make request with Claude Code's exact native web_search tool format + print("\n📞 Making litellm.messages.acreate() call...") + print(f" Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0") + print(f" Query: 'Perform a web search for the query: litellm what is it'") + print(f" Tools: Native web_search_20250305") + print(f" Stream: False") + + response = await messages.acreate( + model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + messages=[{"role": "user", "content": "Perform a web search for the query: litellm what is it"}], + tools=[ + { + "type": "web_search_20250305", + "name": "web_search", + "max_uses": 8 + } + ], + max_tokens=1024, + stream=False, + ) + + print("\n✅ Received response!") + + # Handle both dict and object responses + if isinstance(response, dict): + response_id = response.get("id") + response_model = response.get("model") + response_stop_reason = response.get("stop_reason") + response_content = response.get("content", []) + else: + response_id = response.id + response_model = response.model + response_stop_reason = response.stop_reason + response_content = response.content + + print(f"\n📄 Response ID: {response_id}") + print(f"📄 Model: {response_model}") + print(f"📄 Stop Reason: {response_stop_reason}") + print(f"📄 Content blocks: {len(response_content)}") + + # Debug: Print all content block types + for i, block in enumerate(response_content): + block_type = block.get("type") if isinstance(block, dict) else block.type + print(f" Block {i}: type={block_type}") + if block_type == "tool_use": + block_name = block.get("name") if isinstance(block, dict) else block.name + print(f" name={block_name}") + + # Validate response + assert response is not None, "Response should not be None" + assert response_content is not None, "Response should have content" + assert len(response_content) > 0, "Response should have at least one content block" + + # Check if response contains tool_use (means interception didn't work) + has_tool_use = any( + (block.get("type") if isinstance(block, dict) else block.type) == "tool_use" + for block in response_content + ) + + # Check if we got a text response + has_text = any( + (block.get("type") if isinstance(block, dict) else block.type) == "text" + for block in response_content + ) + + if has_tool_use: + print("\n❌ TEST FAILED: Interception did not work") + print(f"❌ Stop reason: {response_stop_reason}") + print("❌ Response contains tool_use blocks") + return False + + elif has_text and response_stop_reason != "tool_use": + text_block = next( + block for block in response_content + if (block.get("type") if isinstance(block, dict) else block.type) == "text" + ) + text_content = text_block.get("text") if isinstance(text_block, dict) else text_block.text + + print(f"\n📝 Response Text:") + print(f" {text_content[:200]}...") + + if "litellm" in text_content.lower(): + print("\n" + "="*80) + print("✅ TEST PASSED!") + print("="*80) + print("✅ Claude Code's native web_search_20250305 tool was intercepted") + print("✅ Tool was converted to LiteLLM standard format") + print("✅ User made ONE litellm.messages.acreate() call") + print("✅ Got back final answer with search results") + print("✅ Agentic loop executed transparently") + print("✅ WebSearch interception working with Claude Code!") + print("="*80) + return True + else: + print("\n⚠️ Got text response but doesn't mention LiteLLM") + return False + else: + print("\n❌ Unexpected response format") + return False + + except Exception as e: + print(f"\n❌ Test failed with error: {str(e)}") + import traceback + traceback.print_exc() + return False + + +if __name__ == "__main__": + import asyncio + + async def run_all_tests(): + """Run all E2E tests""" + test_results = [] + + # Test 1: Non-streaming + result1 = await test_websearch_interception_non_streaming() + test_results.append(("Non-Streaming", result1)) + + # Test 2: Streaming + result2 = await test_websearch_interception_streaming() + test_results.append(("Streaming", result2)) + + # Test 3: No tool call with streaming + result3 = await test_websearch_interception_no_tool_call_streaming() + test_results.append(("No Tool Call Streaming", result3)) + + # Test 4: Claude Code native web_search + result4 = await test_claude_code_native_websearch() + test_results.append(("Claude Code Native WebSearch", result4)) + + # Print summary + print("\n" + "="*80) + print("TEST SUMMARY") + print("="*80) + for test_name, result in test_results: + status = "✅ PASSED" if result else "❌ FAILED" + print(f"{test_name}: {status}") + print("="*80) + + # Return overall result + return all(result for _, result in test_results) + + result = asyncio.run(run_all_tests()) + import sys + sys.exit(0 if result else 1) + + +async def test_litellm_standard_websearch_tool(): + """ + PRIORITY TEST #1: Test with the canonical litellm_web_search tool format. + + This validates that using get_litellm_web_search_tool() directly + works end-to-end without any conversion needed. + """ + print("\n" + "="*80) + print("E2E TEST: LiteLLM Standard WebSearch Tool") + print("="*80) + + from litellm.integrations.websearch_interception import get_litellm_web_search_tool + + print("\n✅ Using existing router configuration") + print("✅ WebSearch interception already enabled for Bedrock") + + try: + print("\n📞 Making litellm.messages.acreate() call...") + print(f" Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0") + print(f" Query: 'What is the latest news about AI?'") + print(f" Tool: litellm_web_search (standard format, no conversion needed)") + print(f" Stream: False") + + response = await messages.acreate( + model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + messages=[{"role": "user", "content": "What is the latest news about AI? Give me a brief overview."}], + tools=[get_litellm_web_search_tool()], + max_tokens=1024, + stream=False, + ) + + print("\n✅ Received response!") + + if isinstance(response, dict): + response_id = response.get("id") + response_stop_reason = response.get("stop_reason") + response_content = response.get("content", []) + else: + response_id = response.id + response_stop_reason = response.stop_reason + response_content = response.content + + print(f"\n📄 Response ID: {response_id}") + print(f"📄 Stop Reason: {response_stop_reason}") + print(f"📄 Content blocks: {len(response_content)}") + + for i, block in enumerate(response_content): + block_type = block.get("type") if isinstance(block, dict) else block.type + print(f" Block {i}: type={block_type}") + + has_tool_use = any( + (block.get("type") if isinstance(block, dict) else block.type) == "tool_use" + for block in response_content + ) + + has_text = any( + (block.get("type") if isinstance(block, dict) else block.type) == "text" + for block in response_content + ) + + if has_tool_use: + print("\n❌ TEST FAILED: Interception did not work") + return False + + elif has_text and response_stop_reason != "tool_use": + text_block = next( + block for block in response_content + if (block.get("type") if isinstance(block, dict) else block.type) == "text" + ) + text_content = text_block.get("text") if isinstance(text_block, dict) else text_block.text + + print(f"\n📝 Response Text: {text_content[:200]}...") + + print("\n" + "="*80) + print("✅ TEST PASSED!") + print("="*80) + print("✅ LiteLLM standard tool format works without conversion") + print("✅ Agentic loop executed transparently") + print("="*80) + return True + else: + print("\n❌ Unexpected response format") + return False + + except Exception as e: + print(f"\n❌ Test failed with error: {str(e)}") + import traceback + traceback.print_exc() + return False + + +async def test_claude_code_native_websearch_streaming(): + """ + PRIORITY TEST #2: Test Claude Code's native tool WITH stream=True. + + Validates: + - Native tool conversion (web_search_20250305 → litellm_web_search) + - Stream=True → Stream=False conversion + - Agentic loop executes with both conversions + """ + print("\n" + "="*80) + print("E2E TEST: Claude Code Native WebSearch + Streaming") + print("="*80) + + print("\n✅ Using existing router configuration") + print("✅ WebSearch interception already enabled for Bedrock") + + try: + print("\n📞 Making litellm.messages.acreate() call with stream=True...") + print(f" Model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0") + print(f" Tool: Native web_search_20250305") + print(f" Stream: True (will be converted to False)") + + response = await messages.acreate( + model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + messages=[{"role": "user", "content": "Search for the latest AI developments."}], + tools=[{"type": "web_search_20250305", "name": "web_search", "max_uses": 8}], + max_tokens=1024, + stream=True, + ) + + print("\n✅ Received response!") + + import inspect + is_stream = inspect.isasyncgen(response) + + if is_stream: + print("\n⚠️ Response is a stream (stream conversion didn't work)") + return False + + print("✅ Response is NOT a stream (conversion worked!)") + + if isinstance(response, dict): + response_stop_reason = response.get("stop_reason") + response_content = response.get("content", []) + else: + response_stop_reason = response.stop_reason + response_content = response.content + + has_tool_use = any( + (block.get("type") if isinstance(block, dict) else block.type) == "tool_use" + for block in response_content + ) + + has_text = any( + (block.get("type") if isinstance(block, dict) else block.type) == "text" + for block in response_content + ) + + if has_tool_use: + print("\n❌ TEST FAILED: Interception did not work") + return False + + elif has_text and response_stop_reason != "tool_use": + print("\n" + "="*80) + print("✅ TEST PASSED!") + print("="*80) + print("✅ Native tool converted to litellm_web_search") + print("✅ Stream=True converted to Stream=False") + print("✅ Both conversions working together!") + print("="*80) + return True + else: + print("\n❌ Unexpected response format") + return False + + except Exception as e: + print(f"\n❌ Test failed with error: {str(e)}") + import traceback + traceback.print_exc() + return False + + +def test_is_web_search_tool_detection(): + """ + PRIORITY TEST #3: Unit test for is_web_search_tool() utility. + + Validates detection of all supported formats including future versions. + """ + print("\n" + "="*80) + print("UNIT TEST: Web Search Tool Detection") + print("="*80) + + from litellm.integrations.websearch_interception import is_web_search_tool + + test_cases = [ + ({"name": "litellm_web_search"}, True, "LiteLLM standard tool"), + ({"type": "web_search_20250305", "name": "web_search", "max_uses": 8}, True, "Current Anthropic native (2025)"), + ({"type": "web_search_2026", "name": "web_search"}, True, "Future Anthropic native (2026)"), + ({"type": "web_search_20270615", "name": "web_search"}, True, "Future Anthropic native (2027)"), + ({"name": "web_search", "type": "web_search_20250305"}, True, "Claude Code format"), + ({"name": "WebSearch"}, True, "Legacy WebSearch"), + ({"name": "calculator"}, False, "Non-web-search tool"), + ({"name": "some_tool", "type": "function"}, False, "Other tool with type"), + ({"type": "custom_tool"}, False, "Custom tool type"), + ] + + passed = 0 + failed = 0 + + for tool, expected, description in test_cases: + result = is_web_search_tool(tool) + if result == expected: + print(f" ✅ PASS: {description}") + passed += 1 + else: + print(f" ❌ FAIL: {description}") + print(f" Tool: {tool}") + print(f" Expected: {expected}, Got: {result}") + failed += 1 + + print(f"\n📊 Results: {passed} passed, {failed} failed") + + if failed == 0: + print("\n" + "="*80) + print("✅ ALL DETECTION TESTS PASSED!") + print("="*80) + print("✅ Detects all current formats") + print("✅ Future-proof for new web_search_* versions") + print("="*80) + return True + else: + print("\n❌ Some detection tests failed") + return False + + +async def test_pre_request_hook_modifies_request_body(): + """ + Unit test to verify async_pre_request_hook correctly modifies request body. + + Tests that: + 1. WebSearchInterceptionLogger is active + 2. Native web_search_20250305 tool is converted to litellm_web_search + 3. Stream is converted from True to False + 4. Modified parameters reach the API call + """ + import asyncio + from unittest.mock import AsyncMock, patch, MagicMock + from litellm.constants import LITELLM_WEB_SEARCH_TOOL_NAME + + litellm._turn_on_debug() + + print("\n" + "="*80) + print("UNIT TEST: Pre-Request Hook Modifies Request Body") + print("="*80) + + # Initialize WebSearchInterceptionLogger + litellm.callbacks = [ + WebSearchInterceptionLogger( + enabled_providers=[LlmProviders.BEDROCK], + search_tool_name="test-search-tool" + ) + ] + + print("✅ WebSearchInterceptionLogger initialized") + + # Track what actually gets sent to the API + captured_request = {} + + def mock_anthropic_messages_handler( + max_tokens, + messages, + model, + metadata=None, + stop_sequences=None, + stream=None, + system=None, + temperature=None, + thinking=None, + tool_choice=None, + tools=None, + top_k=None, + top_p=None, + container=None, + api_key=None, + api_base=None, + client=None, + custom_llm_provider=None, + **kwargs + ): + """Mock handler that captures the actual request parameters""" + # Capture what gets sent to the handler (after hook modifications) + captured_request['tools'] = tools + captured_request['stream'] = stream + captured_request['max_tokens'] = max_tokens + captured_request['model'] = model + + # Return a mock response (non-streaming) + from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicMessagesResponse + return AnthropicMessagesResponse( + id="msg_test", + type="message", + role="assistant", + content=[{ + "type": "text", + "text": "Test response" + }], + model="claude-sonnet-4-5", + stop_reason="end_turn", + usage={ + "input_tokens": 10, + "output_tokens": 20 + } + ) + + # Patch the anthropic_messages_handler function (called after hooks) + with patch('litellm.llms.anthropic.experimental_pass_through.messages.handler.anthropic_messages_handler', + side_effect=mock_anthropic_messages_handler): + + print("\n📝 Making request with native web_search_20250305 tool (stream=True)...") + + # Make the request with native tool format + response = await messages.acreate( + model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + messages=[{"role": "user", "content": "Test query"}], + tools=[{ + "type": "web_search_20250305", + "name": "web_search", + "max_uses": 8 + }], + max_tokens=100, + stream=True # Should be converted to False + ) + + print("\n🔍 Verifying request modifications...") + + # Verify tool was converted + tools = captured_request.get('tools') + print(f"\n Captured tools: {tools}") + + if tools and len(tools) > 0: + tool = tools[0] + tool_name = tool.get('name') + + if tool_name == LITELLM_WEB_SEARCH_TOOL_NAME: + print(f" ✅ Tool converted: web_search_20250305 → {LITELLM_WEB_SEARCH_TOOL_NAME}") + else: + print(f" ❌ Tool NOT converted: expected {LITELLM_WEB_SEARCH_TOOL_NAME}, got {tool_name}") + return False + else: + print(" ❌ No tools captured in request") + return False + + # Verify stream was converted + stream = captured_request.get('stream') + print(f" Captured stream: {stream}") + + if stream is False: + print(" ✅ Stream converted: True → False") + else: + print(f" ❌ Stream NOT converted: expected False, got {stream}") + return False + + print("\n" + "="*80) + print("✅ PRE-REQUEST HOOK TEST PASSED!") + print("="*80) + print("✅ CustomLogger is active") + print("✅ async_pre_request_hook modifies request body") + print("✅ Tool conversion works correctly") + print("✅ Stream conversion works correctly") + print("="*80) + + return True +