* fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402)
* test(proxy): add test for vertex passthrough load balancing
Add a test that verifies _base_vertex_proxy_route uses
get_available_deployment for proper load balancing instead of
get_model_list. This ensures the correct deployment is selected
from the router and vertex credentials are properly fetched.
Also refactor the implementation to:
- Use get_available_deployment instead of get_model_list
- Add error handling for deployment retrieval
- Improve code structure with try-except block
* feat(proxy): add pass-through deployment filtering methods
Add dedicated methods to filter and select deployments for pass-through endpoints:
- Implement get_available_deployment_for_pass_through() to ensure only deployments with use_in_pass_through=True are considered
- Implement async_get_available_deployment_for_pass_through() for async operations
- Add _filter_pass_through_deployments() helper method to filter by use_in_pass_through flag
- Update vertex pass-through route to use the new dedicated method
This ensures pass-through endpoints respect the use_in_pass_through configuration and apply proper load balancing strategy only to configured deployments.
Add comprehensive tests to verify filtering and load balancing behavior.
* feat(gemini): add opt-in support for responseJsonSchema
Add support for Gemini's native responseJsonSchema parameter which uses
standard JSON Schema format instead of OpenAPI-style responseSchema.
Benefits of responseJsonSchema (Gemini 2.0+ only):
- Standard JSON Schema format (lowercase types)
- Supports additionalProperties for stricter validation
- Better compatibility with Pydantic's model_json_schema()
- No propertyOrdering required
Usage:
```python
response_format={
"type": "json_schema",
"json_schema": {"schema": {...}},
"use_json_schema": True # opt-in
}
```
This is backwards compatible - existing code continues to use
responseSchema by default.
Closes#16340
* docs: add documentation for use_json_schema parameter
Document the new use_json_schema option for Gemini 2.0+ models
in the JSON Mode documentation.
* refactor(gemini): use responseJsonSchema by default for Gemini 2.0+
Remove opt-in flag `use_json_schema` and automatically detect model version:
- Gemini 2.0+: uses responseJsonSchema (standard JSON Schema, supports additionalProperties)
- Gemini 1.5: uses responseSchema (OpenAPI format, legacy)
This follows LiteLLM's philosophy of abstracting provider differences -
users write the same code regardless of model version.
* Update CLAUDE.md with qwen3 tool_calls bug fix instructions (#18922)
* fix(ollama): set finish_reason to "tool_calls" when tool_calls present
When qwen3 models return tool_calls through Ollama, the finish_reason
was incorrectly left as "stop" instead of being set to "tool_calls".
This caused clients to miss the tool_calls in the response.
Added _get_finish_reason helper method following OpenAI provider's
pattern, and fixed both streaming and non-streaming response paths.
Fixes: https://github.com/BerriAI/litellm/issues/18922
* fix(ollama): pass tools directly without model capability check
The previous code tried to check model capability via get_model_info()
which made network calls to localhost:11434. When Ollama is remote,
this fails and falls back to JSON format, breaking tool calling.
Ollama 0.4+ supports native tool calling - let Ollama handle
model capability detection instead of LiteLLM.
Fixes#18922
* fix(ollama): transform tool_calls response to OpenAI format
Ollama returns tool_calls with arguments as dict, but OpenAI format
requires arguments to be a JSON string. Also ensures 'type': 'function'
field is present.
Completes the fix for #18922
* fix(ollama): set finish_reason to "tool_calls" when tool_calls present
Fixes#18922
Two issues addressed:
1. Remove broken model capability check
- get_model_info() fails when Ollama runs on remote server
- Broken fallback triggered JSON prompt injection
- Now passes tools directly - Ollama 0.4+ handles detection
2. Set finish_reason correctly
- Was hardcoded to "stop" even with tool_calls present
- Clients use this to know how to process the response
- Now returns "tool_calls" when tool_calls are in response
Both streaming and non-streaming responses are fixed.
Tests:
- All 14 existing Ollama tests pass
- Added 3 focused tests for the fixes
* fix(text_completion): support token IDs (list of integers) as prompt
Add support for passing token IDs (list of integers) to the text_completion
endpoint for OpenAI-compatible providers (openai, azure, vllm, etc.).
Fixes#17118
* test(text_completion): replace live test with mock test for token IDs
Move token IDs test from local_testing to test_litellm with mocks
per PR review feedback.
* feat(bedrock): add OpenAI-compatible service_tier parameter translation
Translates OpenAI's service_tier parameter (string) to Bedrock's
serviceTier format (object with type field).
* docs(bedrock): add OpenAI-compatible service_tier parameter documentation
Document the automatic translation from OpenAI-style service_tier
parameter to Bedrock's native serviceTier format.
* feat(bedrock): add service_tier to response when present
According to OpenAI's API documentation, when service_tier is sent in the
request, it should be returned in the response. This commit implements
this behavior for Bedrock Converse API to maintain compatibility with
OpenAI's API.
Changes:
- Added serviceTier field to ConverseResponseBlock type definition
- Moved ServiceTierBlock definition before ConverseResponseBlock to fix
type reference order
- Added response transformation to map Bedrock serviceTier (object) to
OpenAI service_tier (string format)
- Added 4 new tests for response transformation with service_tier
The service_tier is only added to the response when present in Bedrock's
response, maintaining backward compatibility.
Fixes#18137
Similar to the fix for web_search_tool_result (#17746, #17798), this PR
preserves web_fetch_tool_result blocks in multi-turn conversations.
Changes:
- Add handling for web_fetch_tool_result in transformation.py (non-streaming)
- Add capture of web_fetch_tool_result in handler.py (streaming)
- Fix streaming tool arguments bug where empty input {} was prepended to
actual arguments by using empty string instead of str({})
- Add unit tests for web_fetch_tool_result handling
* fix(generic-guardrail-api): fix SerializationIterator error on multimodal requests
When sending multimodal messages (with images) through the Generic Guardrail API,
the `model_dump()` call fails with "Object of type SerializationIterator is not
JSON serializable" error.
Root cause: The `ChatCompletionAssistantMessage` type defines `content` as an
`Iterable` (not just `List`), and Pydantic's `model_dump()` creates a
`SerializationIterator` for iterables which is not JSON serializable.
Fix: Use `model_dump(mode="json")` which properly converts all iterables to
lists and ensures all complex objects are JSON serializable.
* fix(guardrails): pass tools (function definitions) to guardrail inputs
The unified guardrail handler was not passing the `tools` parameter
(function definitions) from the request to the guardrail inputs.
This meant guardrails could not inspect or validate tool definitions.
Added extraction of `data.get("tools")` and inclusion in the
GenericGuardrailAPIInputs passed to `apply_guardrail()`.
* test(guardrails): add tests for tools passed to guardrail
Added tests verifying that tools (function definitions) are correctly
passed to guardrails in the unified guardrail handler:
- test_tools_passed_to_guardrail
- test_multiple_tools_passed_to_guardrail
- test_no_tools_in_request
- test_tools_and_tool_calls_both_passed
* fix(azure): add logprobs support for Azure OpenAI GPT-5 models
Azure OpenAI GPT-5 models (including gpt-5.2) support logprobs
parameters, unlike OpenAI's GPT-5 reasoning models. This fix
overrides the parent class restriction to enable logprobs for Azure.
Changes:
- Override get_supported_openai_params() in AzureOpenAIGPT5Config
- Add "logprobs" and "top_logprobs" to supported params
- Add comprehensive tests for logprobs functionality
Testing:
- Verified with direct Azure API calls to gpt-5.2
- API version: 2025-01-01-preview
- Successfully returns logprobs data
Related: #7974, #4022
* refactor: restrict logprobs support to gpt-5.2 only
Only gpt-5.2 has been verified to support logprobs on Azure.
Other gpt-5 variants (gpt-5, gpt-5.1) have not been tested.
Changes:
- Add conditional check for is_model_gpt_5_2_model()
- Update tests to be specific to gpt-5.2
- Add negative tests for gpt-5 and gpt-5.1
- Update documentation to reflect gpt-5.2 specificity
Added XIAOMI_MIMO to the LlmProviders enum in types/utils.py.
The provider was already configured in providers.json but was
missing from the enum, causing "Unsupported provider" errors
when using it in Router/Proxy configurations.
Also added comprehensive unit tests to prevent regression.
