Enables Gemini 3+ models to combine built-in tools (Google Search, etc.)
with custom functions via `include_server_side_tool_invocations=True`.
Server-side invocations are surfaced in provider_specific_fields and
automatically re-injected on subsequent turns for multi-turn coherence.
Closes#24047
- Add try/except httpx.HTTPStatusError blocks in _async_cancel_batch for
both POST cancel and GET retrieve calls, with verbose_logger error logging
- Fix endpoint extraction inconsistency: compute endpoint from URL without
:cancel suffix so it matches behaviour of create_batch/retrieve_batch
- Add explicit validation that api_base ends with ':cancel' before
stripping it, raising a descriptive error for unsupported custom proxy
URL rewriting scenarios
- Use string-based patch() in test instead of patch.object() for robustness
against import order changes
Made-with: Cursor
_get_token_and_url_context_caching() was hardcoding model=None when
calling _check_custom_proxy(), which raises ValueError when api_base
is set because Gemini proxy URLs need the model name:
{api_base}/models/{model}:cachedContents
Fixes#23846
The count_tokens handler unconditionally overrode vertex_location to
us-central1 for Claude models, ignoring the user-configured
vertex_count_tokens_location parameter. Also, us-central1 is no longer
a supported region — Google now supports us-east5, europe-west1, and
asia-southeast1.
Now vertex_count_tokens_location takes precedence, vertex_location is
used as fallback, and us-east5 is the default only when neither is set.
Fixes#23872
Models like gemini-3.1-flash-lite-preview send the final streaming chunk
with empty content (text:"") alongside finishReason:"STOP", instead of
omitting content entirely. The existing fix (PR #21577) only handled
chunks without content, so this case was missed.
Now, after processing candidates, if tool_calls were seen in earlier
chunks and a choice has finish_reason="stop", it is overridden to
"tool_calls" to match the OpenAI spec.
Fixes#22900
Add Vertex batch cancellation support in LiteLLM batch APIs, route proxy cancel fallback using request provider headers, and return post-cancel batch state via retrieve to keep response shape compatible.
Made-with: Cursor
* PR #22867 added _remove_scope_from_cache_control for Bedrock and Azure AI but omitted Vertex AI. This applies the same pattern to VertexAIPartnerModelsAnthropicMessagesConfig."
* PR #22867 added _remove_scope_from_cache_control for Bedrock and Azure AI but omitted Vertex AI. This applies the same pattern to VertexAIPartnerModelsAnthropicMessagesConfig."
* PR #22867 added _remove_scope_from_cache_control to AzureAnthropicMessagesConfig
but missed VertexAIPartnerModelsAnthropicMessagesConfi Rather than duplicating the method again, moved it up to the base AnthropicMessagesConfig so all providers
inherit it, and removed the now-redundant copy from the Azure AI subclass.
* PR #22867 added _remove_scope_from_cache_control to AzureAnthropicMessagesConfig
but missed VertexAIPartnerModelsAnthropicMessagesConfi Rather than duplicating the method again, moved it up to the base AnthropicMessagesConfig so all providers
inherit it, and removed the now-redundant copy from the Azure AI subclass.
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
The test was creating a real AsyncHTTPHandler instance and patching its
post method, but the internal code creates its own handler, bypassing
the mock. This caused real API calls to Vertex AI, resulting in 401
auth errors in CI. Switched to patching AsyncHTTPHandler at the class
level, matching the pattern used by the passing GPT-OSS test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Add missing __init__.py files in tests/test_litellm/llms/gemini/ and
subdirectories (realtime/, image_edit/) to fix ModuleNotFoundError
with pytest-xdist parallel workers.
2. Update test_transform_request_uses_dynamic_max_tokens to use
claude-3-7-sonnet-20250219 (max_output_tokens=64000) since
claude-3-5-sonnet-20241022 was removed from model_prices JSON
during deprecated model cleanup. The test assertion was outdated.
3. Update context caching TTL tests to use gemini-2.5-pro instead of
gemini-1.5-pro. The old model was removed from model_prices JSON,
causing supports_system_messages to return False, which prevented
system_instruction from appearing in the transformation output.
Co-authored-by: yuneng-jiang <yuneng-jiang@users.noreply.github.com>
Lint fixes (check_code_and_doc_quality job):
- Remove unused variable reasoning_effort in gpt_5_transformation.py (F841)
- Remove unused timezone imports in mcp_server rest_endpoints.py and server.py (F401)
- Remove unused ProxyBaseLLMRequestProcessing import in realtime endpoints.py (F401)
- Add BaseRealtimeHTTPConfig to TYPE_CHECKING block in utils.py (F821)
- Add PLR0915 per-file-ignore for mcp_server/rest_endpoints.py in ruff.toml
Test fixes (litellm_mapped_tests_llms job):
- Gemini video cost tests: pass explicit model_info to video_generation_cost()
instead of relying on gemini/veo-3.0-generate-preview being in model_prices JSON
- Anthropic max_tokens tests: mock get_max_tokens() to return expected values
instead of depending on claude-3-5-sonnet-20241022 being in model_prices JSON
- Vertex AI pydantic obj test: update from removed gemini-1.5-pro to gemini-2.5-flash,
update expected request body to use response_json_schema format
- Vertex AI/Bedrock file_content integration tests: update mocks to target
base_llm_http_handler.retrieve_file_content (the new code path via
ProviderConfigManager) instead of the old vertex_ai_files_instance/
bedrock_files_instance paths
Co-authored-by: yuneng-jiang <yuneng-jiang@users.noreply.github.com>
Keep both sets of tests: upstream's OAuth2 token injection test and
our case-insensitive tool matching tests. Use upstream's version of
the bedrock output_config test (more comprehensive).
Add verbose_logger.warning when user-specified region is overridden by
supported_regions. Remove now-unused is_global_only_vertex_model function
and its tests since get_vertex_region handles all region logic directly.
- get_vertex_region now overrides user-specified region when it's not in
the model's supported_regions list (prevents 404 for users with a
global VERTEXAI_LOCATION default hitting global-only models)
- Add supported_regions: ["global"] to glm-5-maas in both JSON files
- Update tests to cover the override behavior
- Remove redundant get_vertex_region() call in partner models main.py
(already called inside get_complete_vertex_url)
- Rewrite test mocks to use patch.dict(litellm.model_cost) instead of
patching the removed is_global_only_vertex_model symbol
- Align test assertions with new behavior: user-specified region is
preserved (not overridden) for global-only models
Keep unified _FINISH_REASON_MAP dict approach, discard upstream's
inconsistent _VALID_OPENAI_FINISH_REASONS frozenset that mapped to
values not in the OpenAIChatCompletionFinishReason Literal.
Gemini 2.0+ natively accepts JSON Schema in tool parameters, including
bare {} (TYPE_UNSPECIFIED), anyOf with null, and lowercase types. The
existing _build_vertex_schema pipeline was coercing {} to {"type": "object"},
breaking JsonValue/Any field semantics (issue #22391).
Add _build_vertex_schema_for_gemini_2() that only resolves $ref (which
Gemini doesn't support in tools) and filters unsupported fields. Use it
for Gemini 2.0+ models, keeping the full transform for Gemini 1.5.
PR #20950 added extra_body forwarding to Vertex AI Gemini. LiteLLM-internal
keys (cache, tags) were being merged into the request body, causing Vertex AI
to reject with 400: 'Unknown name "cache": Cannot find field.'
- Add _LITELLM_INTERNAL_EXTRA_BODY_KEYS frozenset (cache, tags)
- Skip these keys in _pop_and_merge_extra_body before merging
- Add regression tests for cache and tags stripping
Fixes regression from 1.79.3 → 1.81.12 when using proxy cache with
extra_body={"cache": {"use-cache": True, "ttl": 86400}}
Made-with: Cursor
* Revert "fix(vertex): drop bare {} schemas from anyOf before adding nullable=True (#23060)"
This reverts commit 3ad9a536d3.
* Revert "Merge pull request #22589 from Chesars/fix/vertex-preserve-any-type-schema"
This reverts commit da941e4261, reversing
changes made to f77f28a5f8.
When anyOf contains a mix of concrete types, bare {} (any-type), and null,
convert_anyof_null_to_nullable was adding nullable=True to the {} entry,
producing {nullable: True} with no type field. Gemini rejects this as an
anyOf entry without a concrete type, breaking tool calls that use
Optional[List[...]] or similar union types (common in LangChain/Pydantic).
Fix: strip any-type schemas from anyOf before the nullable=True pass.
If only any-type schemas remain after null removal (anyOf: [{}, null]),
collapse the anyOf entirely and set nullable=True on the parent schema
instead — correctly representing 'any nullable value' for Gemini.
Regression introduced by da941e4261.
* feat(vertex_ai): support explicit AWS credentials for WIF auth
The current Vertex AI AWS Workload Identity Federation implementation
exclusively uses google.auth.aws.Credentials.from_info(), which requires
EC2 instance metadata access to obtain AWS credentials. In environments
where the metadata service is blocked for security reasons, this makes
WIF unusable.
Add support for explicit AWS credentials by implementing a custom
AwsSecurityCredentialsSupplier (google-auth >= 2.29.0). When aws_* keys
(e.g. aws_role_name, aws_region_name) are present in the WIF credential
JSON, LiteLLM uses BaseAWSLLM.get_credentials() to obtain AWS creds via
STS AssumeRole (or any other supported AWS auth flow), wraps them in the
custom supplier, and passes them to aws.Credentials() — bypassing the
metadata service entirely.
When no aws_* keys are present, the existing from_info() flow is used
unchanged, preserving full backward compatibility.
* refactor(vertex_ai): extract AWS WIF auth to own class + add docs
Address PR review feedback:
- Move _AWS_CREDENTIAL_KEYS, _extract_aws_params(), and
_credentials_from_aws_with_explicit_auth() from VertexBase into
new VertexAIAwsWifAuth class in vertex_ai_aws_wif.py
- Add documentation for explicit AWS credentials WIF auth method
in vertex.md (supported params, JSON example, SDK/Proxy tabs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(vertex_ai): use lazy credentials provider to prevent stale STS tokens
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Vertex AI does not support the output_config parameter in its API.
This parameter is being added by Anthropic/Gemini transformations but needs
to be removed before sending requests to Vertex AI endpoints.
This fix addresses the "Extra inputs are not permitted" error (issue #22312)
when using Claude models with structured outputs on Vertex AI.
Changes:
- Drop output_config in Gemini model transformation
- Drop output_config in Anthropic partner model transformation
- Drop output_config in Anthropic experimental pass-through transformation
- Add comprehensive tests to verify output_config is dropped
Fixes: #22312
Made-with: Cursor
* fix(gemini): ensure image token accumulation in usage metadata
Fixed an issue where image tokens were being overwritten instead of accumulated in Gemini responses. Added support for both camelCase and snake_case token count keys. Fixes#22082.
* test: add regression test for image token accumulation and cleanup files
* fix(gemini): ensure consistent accumulation for responseTokensDetails
* fix(gemini): harden token count parsing and add vertex accumulation test
Parse tokenCount/token_count as int-safe values to satisfy mypy and avoid None/object arithmetic. Add regression test for duplicate modality accumulation in Vertex _calculate_usage.
The user-specified async client was being overwritten by
`litellm.module_level_aclient` in `streaming_handler.py` when using
async+streaming with Gemini.
This fix adds a `gemini_client` parameter to `make_call()` (matching
the existing pattern in `make_sync_call()`) so the user's custom client
is preserved and not overwritten.
Fixes#17148
Add global media_resolution support for Gemini 2.x models (2.0, 2.5) when
using OpenAI's detail parameter on images. Previously, the detail parameter
was only working for Gemini 3+ models (per-part) and was silently ignored
for older Gemini models.
- Add _get_highest_media_resolution() and _extract_max_media_resolution_from_messages()
to extract highest detail from all images/files in a request
- Update _transform_request_body() to add mediaResolution to generationConfig
for Gemini 2.x models only (not 1.x which doesn't support it, not 3+ which
uses per-part)
- Add mediaResolution field to GenerationConfig TypedDict
- Support detail extraction from both image_url and file content types
- Add comprehensive unit tests and update documentation
