AsyncHTTPHandler.__del__ was closing httpx clients still in use by
AsyncOpenAI/AsyncAzureOpenAI due to independent cache lifecycles.
Restores standalone httpx client creation for OpenAI/Azure providers.
* fix: check for model_response_choices before guardrail input
* test: add tests for responses api translation
* fix: protect other guardrail translations
* refactor: remove type ignores
* anthropic request body got mutated fix
* add warning when extra_body is provided but user is non premium
* fix: resolve mypy union-attr errors in anthropic guardrail handler
Cast choices[0] to Choices type before accessing .message attribute
to satisfy mypy's union type checking for Choices | StreamingChoices.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* add logger when model response has no choices for streaming /response and /messages
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 541a2b075a.
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 716ea0caa1.
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add x-api-key header to CountTokens handler to match chat completion
authentication. Azure AI Anthropic requires this header per Microsoft's
native API format.
When using LiteLLM's Anthropic /v1/messages endpoint to route requests to
OpenAI models, requests fail if any tool name exceeds OpenAI's 64-character
limit. Anthropic API has no such limit, causing compatibility issues.
Changes:
- Add truncate_tool_name() function using {55-char-prefix}_{8-char-hash} format
- Modify translate_anthropic_tools_to_openai() to truncate and return mapping
- Modify translate_anthropic_tool_choice_to_openai() to truncate tool name
- Restore original tool names in responses using the mapping
- Support tool name restoration in streaming responses
- Add backwards-compatible API (existing methods still work)
The fix only applies when routing Anthropic requests to OpenAI models.
Native Anthropic/Claude requests pass through unchanged.
Fixes#19788
- Add `supported_regions: ["global"]` to Qwen MaaS models in model_prices_and_context_window.json
- Update `get_supported_regions()` to read directly from `model_cost` dict
- Update `get_complete_vertex_url()` to use `get_vertex_region()` for global-only models
- Update `create_vertex_url()` to generate correct URL for global location (without region prefix)
- Add tests for Qwen global endpoint support
Per review feedback, thought_signature should not be a root-level
param on ImageObject as it's not OpenAI compatible. Moved to
provider_specific_fields dict to match the pattern used in chat
completions (Message, Delta, Choices, etc).
Fixes#17184 - Gemini 3 Pro image preview model returns a thoughtSignature
field required for interactive image editing. This change:
- Adds thought_signature field to ImageObject class
- Updates Gemini and Vertex AI transformations to extract thoughtSignature
- Adds test for thought_signature in response transformation
* fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify
The hosted_vllm provider was falling through to the OpenAI catch-all path
which doesn't pass ssl_verify to the HTTP client. This adds an explicit
elif branch that routes hosted_vllm through base_llm_http_handler.completion()
which properly passes ssl_verify to the httpx client.
- Add explicit hosted_vllm branch in main.py completion()
- Add ssl_verify tests for sync and async completion
- Update existing audio_url test to mock httpx instead of OpenAI client
* feat(hosted_vllm): add embedding support with ssl_verify
- Add HostedVLLMEmbeddingConfig for embedding transformations
- Register hosted_vllm embedding config in utils.py
- Add lazy import for embedding transformation module
- Add unit test for ssl_verify parameter handling
* fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic
Fixes#19891
Vertex AI Anthropic models don't support URL sources for images. LiteLLM
already converted image URLs to base64 for user messages, but not for tool
messages (role='tool'). This caused errors when using ToolOutputImage with
image_url in tool outputs.
Changes:
- Add force_base64 parameter to convert_to_anthropic_tool_result()
- Pass force_base64 to create_anthropic_image_param() for tool message images
- Calculate force_base64 in anthropic_messages_pt() based on llm_provider
- Add unit tests for tool message image handling
* chore: remove extra comment from test file header
The regex in get_vertex_model_id_from_url() was using [^/:]+
which stopped at the first slash, truncating model names like
'gcp/google/gemini-2.5-flash' to just 'gcp'. This caused
access_groups checks to fail for custom model names.
Changed the pattern to [^:]+ to allow slashes in model names,
only stopping at the colon before the action (e.g., :generateContent).
- Add whitelist-based filtering for anthropic_beta headers
- Only allow Bedrock-supported beta flags (computer-use, tool-search, etc.)
- Filter out unsupported flags like mcp-servers, structured-outputs
- Remove output_format parameter from Bedrock Invoke requests
- Force tool-based structured outputs when response_format is used
Fixes#16726
