When extract_and_raise_litellm_exception tries to raise a LiteLLM exception
from an error string, it was always passing the response parameter. However,
some exceptions like APIConnectionError don't accept this parameter, causing
a TypeError.
This fix tries to raise the exception with the response parameter first,
and falls back to raising without it if a TypeError occurs.
This fixes the error:
TypeError: APIConnectionError.__init__() got an unexpected keyword argument 'response'
Which was occurring when Gemini returned UNEXPECTED_TOOL_CALL finish reason
and LiteLLM tried to convert the error to an APIConnectionError.
Fixes: cascading error when Gemini uses thinking feature (__thought__ tool calls)
* Update CLAUDE.md with qwen3 tool_calls bug fix instructions (#18922)
* fix(ollama): set finish_reason to "tool_calls" when tool_calls present
When qwen3 models return tool_calls through Ollama, the finish_reason
was incorrectly left as "stop" instead of being set to "tool_calls".
This caused clients to miss the tool_calls in the response.
Added _get_finish_reason helper method following OpenAI provider's
pattern, and fixed both streaming and non-streaming response paths.
Fixes: https://github.com/BerriAI/litellm/issues/18922
* fix(ollama): pass tools directly without model capability check
The previous code tried to check model capability via get_model_info()
which made network calls to localhost:11434. When Ollama is remote,
this fails and falls back to JSON format, breaking tool calling.
Ollama 0.4+ supports native tool calling - let Ollama handle
model capability detection instead of LiteLLM.
Fixes#18922
* fix(ollama): transform tool_calls response to OpenAI format
Ollama returns tool_calls with arguments as dict, but OpenAI format
requires arguments to be a JSON string. Also ensures 'type': 'function'
field is present.
Completes the fix for #18922
* fix(ollama): set finish_reason to "tool_calls" when tool_calls present
Fixes#18922
Two issues addressed:
1. Remove broken model capability check
- get_model_info() fails when Ollama runs on remote server
- Broken fallback triggered JSON prompt injection
- Now passes tools directly - Ollama 0.4+ handles detection
2. Set finish_reason correctly
- Was hardcoded to "stop" even with tool_calls present
- Clients use this to know how to process the response
- Now returns "tool_calls" when tool_calls are in response
Both streaming and non-streaming responses are fixed.
Tests:
- All 14 existing Ollama tests pass
- Added 3 focused tests for the fixes
Adds 'servers' field to OpenAPI schema when server_root_path is set, ensuring correct Swagger UI execute path for reverse proxies and subpath deployments. Includes tests to verify correct server URL handling for various root path formats.
* fix(text_completion): support token IDs (list of integers) as prompt
Add support for passing token IDs (list of integers) to the text_completion
endpoint for OpenAI-compatible providers (openai, azure, vllm, etc.).
Fixes#17118
* test(text_completion): replace live test with mock test for token IDs
Move token IDs test from local_testing to test_litellm with mocks
per PR review feedback.
* feat(bedrock): add OpenAI-compatible service_tier parameter translation
Translates OpenAI's service_tier parameter (string) to Bedrock's
serviceTier format (object with type field).
* docs(bedrock): add OpenAI-compatible service_tier parameter documentation
Document the automatic translation from OpenAI-style service_tier
parameter to Bedrock's native serviceTier format.
* feat(bedrock): add service_tier to response when present
According to OpenAI's API documentation, when service_tier is sent in the
request, it should be returned in the response. This commit implements
this behavior for Bedrock Converse API to maintain compatibility with
OpenAI's API.
Changes:
- Added serviceTier field to ConverseResponseBlock type definition
- Moved ServiceTierBlock definition before ConverseResponseBlock to fix
type reference order
- Added response transformation to map Bedrock serviceTier (object) to
OpenAI service_tier (string format)
- Added 4 new tests for response transformation with service_tier
The service_tier is only added to the response when present in Bedrock's
response, maintaining backward compatibility.
Fixes#18137
Similar to the fix for web_search_tool_result (#17746, #17798), this PR
preserves web_fetch_tool_result blocks in multi-turn conversations.
Changes:
- Add handling for web_fetch_tool_result in transformation.py (non-streaming)
- Add capture of web_fetch_tool_result in handler.py (streaming)
- Fix streaming tool arguments bug where empty input {} was prepended to
actual arguments by using empty string instead of str({})
- Add unit tests for web_fetch_tool_result handling
The OCI adapter now accepts both string and object formats for image_url:
- String: "image_url": "https://example.com/image.png"
- Object: "image_url": {"url": "https://example.com/image.png"}
This fixes compatibility with OpenAI Vision API format.
* fix(gemini): prevent negative text_tokens with explicit caching (#18750)
## Problem
When using Gemini with explicit caching (especially with images),
text_tokens would become negative (e.g., -3327) due to incorrectly
subtracting total cached_tokens from modality-specific text_tokens.
## Root Cause
The old code did:
```python
text_tokens = text_tokens - cached_tokens # 737 - 4064 = -3327
```
This was wrong because:
- cached_tokens includes ALL modalities (text + image + audio + video)
- text_tokens only contains text
- Subtracting total from specific caused negative values
## Solution
Parse cacheTokensDetails to get per-modality cached token breakdown:
```python
if "cacheTokensDetails" in usage_metadata:
cached_text_tokens = parse from cacheTokensDetails["TEXT"]
text_tokens = text_tokens - cached_text_tokens # Correct!
```
Now we subtract cached tokens per modality, preventing negatives.
## Changes
- Parse cacheTokensDetails field from Gemini response
- Calculate non-cached tokens per modality (text, image, audio)
- Remove incorrect global cached_tokens subtraction
- Add tests for explicit caching and implicit/no caching scenarios
## Testing
- Added test_gemini_cache_tokens_details_no_negative_values
- Added test_gemini_without_cache_tokens_details
- All existing Gemini caching tests pass
Fixes#18750
* feat: add cache_read_input_tokens to Usage object
Addresses reviewer feedback to include cached tokens at the top level
of the Usage object. This aligns with how Anthropic provider handles
cached tokens and ensures they are visible in the final usage response.
* fix: add cacheTokensDetails field to UsageMetadata TypedDict
Fixes mypy error where cacheTokensDetails was being accessed but not defined
in the UsageMetadata TypedDict type definition.
* fix(anthropic): prevent dropping thinking when any message has thinking_blocks
When Claude returns multiple assistant messages in a conversation, some may
have thinking_blocks while others may not (Claude's behavior varies). The
previous logic only checked the LAST assistant message with tool_calls,
dropping the thinking param if it had no thinking_blocks.
This caused errors when earlier messages still contained thinking_blocks:
"When thinking is disabled, an assistant message cannot contain thinking"
The fix adds a new check: only drop thinking if NO assistant messages
have thinking_blocks. If any message has thinking_blocks, we keep
thinking enabled.
Fixes#18926
* chore: re-trigger CI
* fix(generic-guardrail-api): fix SerializationIterator error on multimodal requests
When sending multimodal messages (with images) through the Generic Guardrail API,
the `model_dump()` call fails with "Object of type SerializationIterator is not
JSON serializable" error.
Root cause: The `ChatCompletionAssistantMessage` type defines `content` as an
`Iterable` (not just `List`), and Pydantic's `model_dump()` creates a
`SerializationIterator` for iterables which is not JSON serializable.
Fix: Use `model_dump(mode="json")` which properly converts all iterables to
lists and ensures all complex objects are JSON serializable.
* fix(guardrails): pass tools (function definitions) to guardrail inputs
The unified guardrail handler was not passing the `tools` parameter
(function definitions) from the request to the guardrail inputs.
This meant guardrails could not inspect or validate tool definitions.
Added extraction of `data.get("tools")` and inclusion in the
GenericGuardrailAPIInputs passed to `apply_guardrail()`.
* test(guardrails): add tests for tools passed to guardrail
Added tests verifying that tools (function definitions) are correctly
passed to guardrails in the unified guardrail handler:
- test_tools_passed_to_guardrail
- test_multiple_tools_passed_to_guardrail
- test_no_tools_in_request
- test_tools_and_tool_calls_both_passed
When using the generateContent endpoint with non-Google providers like
github_copilot, the extra_headers from model config were not being
forwarded to the underlying litellm.completion/acompletion calls.
This caused providers requiring custom headers (e.g., Editor-Version
for GitHub Copilot authentication) to reject requests with errors like
"missing Editor-Version header for IDE auth".
Changes:
- Forward extra_headers in _prepare_completion_kwargs() handler
- Pass extra_headers explicitly to adapter in generate_content()
- Pass extra_headers explicitly to adapter in agenerate_content_stream()
- Pass extra_headers explicitly to adapter in generate_content_stream()
- Add tests for extra_headers forwarding behavior
- Update existing test to expect extra_headers in passed fields
Co-authored-by: Claude <noreply@anthropic.com>
