* fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify
The hosted_vllm provider was falling through to the OpenAI catch-all path
which doesn't pass ssl_verify to the HTTP client. This adds an explicit
elif branch that routes hosted_vllm through base_llm_http_handler.completion()
which properly passes ssl_verify to the httpx client.
- Add explicit hosted_vllm branch in main.py completion()
- Add ssl_verify tests for sync and async completion
- Update existing audio_url test to mock httpx instead of OpenAI client
* feat(hosted_vllm): add embedding support with ssl_verify
- Add HostedVLLMEmbeddingConfig for embedding transformations
- Register hosted_vllm embedding config in utils.py
- Add lazy import for embedding transformation module
- Add unit test for ssl_verify parameter handling
* fix(proxy): prevent provider-prefixed model leaks
Proxy clients should not see LiteLLM internal provider prefixes (e.g. hosted_vllm/...) in the OpenAI-compatible response model field.
This patch sanitizes the client-facing model name for both:
- Non-streaming responses returned from base_process_llm_request
- Streaming SSE chunks emitted by async_data_generator
Adds regression tests covering vLLM-style hosted_vllm routing for both streaming and non-streaming paths.
* chore(lint): suppress PLR0915 in proxy handler
Ruff started flagging ProxyBaseLLMRequestProcessing.base_process_llm_request() for too many statements after the hotpatch changes.
Add an explicit '# noqa: PLR0915' on the function definition to avoid a large refactor in a hotpatch.
* refactor(proxy): make model restamp explicit
Replace silent try/except/pass and type ignores with explicit model restamping.
- Logs an error when the downstream response model differs from the client-requested model
- Overwrites the OpenAI `model` field to the client-requested value to avoid leaking internal provider-prefixed identifiers
- Applies the same behavior to streaming chunks, logging the mismatch only once per stream
* chore(lint): drop PLR0915 suppression
The model restamping bugfix made `base_process_llm_request()` slightly exceed Ruff's
PLR0915 (too-many-statements) threshold, requiring a `# noqa` suppression.
Collapse consecutive `hidden_params` extractions into tuple unpacking so the
function falls back under the lint limit and remove the suppression.
No functional change intended; this keeps the proxy model-field bugfix intact
while aligning with project linting rules.
* chore(proxy): log model mismatches as warnings
These model-restamping logs are intentionally verbose: a mismatch is a useful signal
that an internal provider/deployment identifier may be leaking into the public
OpenAI response `model` field.
- Downgrade model mismatch logs from error -> warning
- Keep error logs only for cases where the proxy cannot read/override the model
* fix(proxy): preserve client model for streaming aliasing
Pre-call processing can rewrite request_data['model'] via model alias maps.\n\nOur streaming SSE generator was using the rewritten value when restamping chunk.model, which caused the public 'model' field to differ between streaming and non-streaming responses for alias-based requests.\n\nStash the original client model in request_data as _litellm_client_requested_model after the model has been routed, and prefer it when overriding the outgoing chunk model. Add a regression test for the alias-mapping case.
* chore(lint): satisfy PLR0915 in streaming generator
Ruff started flagging async_data_generator() for too many statements after adding model restamping logic.\n\nExtract the client-model selection + chunk restamping into small helpers to keep behavior unchanged while meeting the project's PLR0915 threshold.
The /health/services endpoint rejected datadog_llm_observability as an
unknown service, even though it was registered in the core callback
registry and __init__.py. Added it to both the Literal type hint and
the hardcoded validation list in the health endpoint.
Fixes#19478
The stream_chunk_builder function was not handling image chunks from
models like gemini-2.5-flash-image. When streaming responses were
reconstructed (e.g., for caching), images in delta.images were lost.
This adds handling for image_chunks similar to how audio, annotations,
and other delta fields are handled.
* fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic
Fixes#19891
Vertex AI Anthropic models don't support URL sources for images. LiteLLM
already converted image URLs to base64 for user messages, but not for tool
messages (role='tool'). This caused errors when using ToolOutputImage with
image_url in tool outputs.
Changes:
- Add force_base64 parameter to convert_to_anthropic_tool_result()
- Pass force_base64 to create_anthropic_image_param() for tool message images
- Calculate force_base64 in anthropic_messages_pt() based on llm_provider
- Add unit tests for tool message image handling
* chore: remove extra comment from test file header
* fix(vertex_ai): replace custom model names with actual Vertex AI model names in passthrough URLs (#19948)
When the passthrough URL already contains project and location, the code
was skipping the deployment lookup and forwarding the URL as-is to Vertex AI.
For custom model names like gcp/google/gemini-2.5-flash, Vertex AI returned
404 because it only knows the actual model name (gemini-2.5-flash).
The fix makes the deployment lookup always run, so the custom model name
gets replaced with the actual Vertex AI model name before forwarding.
* add _resolve_vertex_model_from_router
* fix: get_llm_provider
* Potential fix for code scanning alert no. 4020: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
The regex in get_vertex_model_id_from_url() was using [^/:]+
which stopped at the first slash, truncating model names like
'gcp/google/gemini-2.5-flash' to just 'gcp'. This caused
access_groups checks to fail for custom model names.
Changed the pattern to [^:]+ to allow slashes in model names,
only stopping at the colon before the action (e.g., :generateContent).
* Cleanup code for user cli auth, and make sure not to prompt user for team multiple times while polling
* Adding tests
* Cleanup normalize teams some more
When Gemini uses implicit caching, it returns cachedContentTokenCount but
NOT cacheTokensDetails. Previously, text_tokens was not adjusted in this case,
causing costs to be calculated as if all tokens were non-cached.
This fix subtracts cachedContentTokenCount from text_tokens when no
cacheTokensDetails is present (implicit caching), ensuring correct cost
calculation with the reduced cache_read pricing.
The lazy loading implementation for encoding in __getattr__ was calling
tiktoken.get_encoding() directly without first setting TIKTOKEN_CACHE_DIR.
This caused tiktoken to attempt downloading the encoding file from the
internet instead of using the local copy bundled with litellm.
This fix uses _get_default_encoding() from _lazy_imports which properly
sets TIKTOKEN_CACHE_DIR before loading tiktoken, ensuring the local cache
is used.
As indicated by https://docs.litellm.ai/docs/exception_mapping,
BadRequestError is used as the base type for multiple exceptions. As
such, it should be tested last in handling retry policies.
This updates the integration test that validates retry policies work as
expected.
Fixes#19876
- Add whitelist-based filtering for anthropic_beta headers
- Only allow Bedrock-supported beta flags (computer-use, tool-search, etc.)
- Filter out unsupported flags like mcp-servers, structured-outputs
- Remove output_format parameter from Bedrock Invoke requests
- Force tool-based structured outputs when response_format is used
Fixes#16726
Added test cases for custom model names containing slashes in Vertex AI
passthrough URLs (e.g., gcp/google/gemini-2.5-flash).
Test cases:
- gcp/google/gemini-2.5-flash
- gcp/google/gemini-3-flash-preview
- custom/model
