* fix(gemini): normalize response_schema on native generateContent
The /v1beta/models/{model}:generateContent passthrough forwarded
generationConfig.response_schema verbatim, so schemas containing $defs,
$ref, anyOf-with-null, default, or title were rejected by Gemini even
though /chat/completions already handles them.
GoogleGenAIConfig.transform_generate_content_request now calls a new
_normalize_response_schema helper that mirrors the chat/completions
path: Gemini 2.0+ models get the schema promoted to responseJsonSchema
via _build_json_schema (preserving $defs/$ref natively), older models
keep responseSchema but the schema is flattened with
_build_vertex_schema. VertexAIGoogleGenAIConfig (which overrides the
transform entirely) calls the same helper before building the request.
* fix(gemini): preserve caller-supplied responseJsonSchema when responseSchema co-present
Previously, when both responseJsonSchema and responseSchema were present
on Gemini 2.0+, _normalize_response_schema processed responseJsonSchema
first (no-op normalization) then unconditionally promoted responseSchema
to responseJsonSchema, clobbering the caller-supplied value.
Now skip the promotion (and drop the redundant responseSchema) when the
caller already supplied responseJsonSchema.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore: strip restating comments from response-schema normalize
Drop the docstring on _normalize_response_schema and the two inline
comments that just restated what the surrounding code/asserts already
say. Function name + variable names carry the intent; PR description
covers the why-it-exists context.
* perf(gemini): drop redundant deepcopy on responseJsonSchema normalize
_build_json_schema is a no-op (returns its argument unchanged), so the
deepcopy + round-trip on the responseJsonSchema branch allocated a full
schema copy on every request with no observable effect. Forward the
caller's value as-is, and just move the popped responseSchema value when
promoting on Gemini 2.0+.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* style: remove unneeded comment
* fix(gemini): drop unsupported responseJsonSchema for older models
* test(gemini): add parity test between native and chat schema normalization
Per @Sameerlite review: lock the two Gemini schema-normalization paths
together. If either GoogleGenAIConfig._normalize_response_schema (native
generateContent) or VertexGeminiConfig.apply_response_schema_transformation
(/chat/completions) drifts, the parity test fails — forcing both to be
updated together.
* fix(google_genai): preserve key naming convention in _normalize_response_schema
When the input schema key is snake_case (response_schema), the promoted
JSON schema key should also be snake_case (response_json_schema) instead
of mixing in camelCase (responseJsonSchema). This matters for the Vertex
AI google_genai path which converts all keys to snake_case before
calling _normalize_response_schema.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Gemini API keys embedded in URLs as ?key= query parameters leak through
httpx error tracebacks, which are then captured by traceback.format_exc()
and forwarded to logging callbacks, Slack/Teams alerts, and HTTP client
responses.
Short-term: all httpx.HTTPStatusError handlers now raise
MaskedHTTPStatusError(...) from None, which masks the URL and breaks
exception chaining so the original error never appears in tracebacks.
Long-term: moved all Gemini/Vertex URL constructions from ?key={api_key}
to x-goog-api-key header (Google's documented auth method), so the key
is never in the URL at all. WebSocket realtime is the only exception
since WS clients cannot use custom headers.
Additionally hardened all outbound credential paths:
- WebSocket close reasons now pass through _redact_string()
- Callback pipeline (failure_handler) redacts traceback_exception and
error_str before forwarding to integrations (Langfuse, Datadog, etc.)
- Slack/Teams alert messages redacted in send_llm_exception_alert,
ProxyLogging.failure_handler, and post_call_failure_hook
- HTTP error responses in proxy SSE and health endpoints redacted
- Exception messages in exception_mapping_utils redacted
- print_verbose() stdout output redacted when set_verbose=True
- HTTPHandler.put() now has MaskedHTTPStatusError (was missing)
* litellm_fix_mapped_tests_core: fix test isolation and mock injection issues
## Problem
Four tests in litellm_mapped_tests_core were failing:
1. test_register_model_with_scientific_notation - KeyError due to test isolation issues
2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path
3. test_send_email_missing_api_key - Real API calls despite mocking
4. test_stream_transformation_error_sync - Mock not effective, real API called
## Solution
### test_register_model_with_scientific_notation
- Use unique model name to avoid conflicts with other tests
- Clear LRU caches before test to prevent stale data
- Clean up model_cost entry after test
### test_search_uses_registry_credentials
- Use patch.object() on the actual base_llm_http_handler instance
- String-based patching for instance methods can fail; direct object patching is more reliable
### test_send_email_missing_api_key
- Directly inject mock HTTP client into logger instance
- This bypasses any caching issues that could cause the fixture mock to be ineffective
### test_stream_transformation_error_sync
- Patch litellm.completion directly instead of the handler module's litellm reference
- This ensures the mock is effective regardless of import order
## Regression
These tests were affected by LRU caching added in #19606 and HTTP client caching.
* fix(test): use patch.object for container API tests to fix mock injection
## Problem
test_retrieve_container_basic tests were failing because mocks weren't
being applied correctly. The tests used string-based patching:
patch('litellm.containers.main.base_llm_http_handler')
But base_llm_http_handler is imported at module level, so the mock wasn't
intercepting the actual handler calls, resulting in real HTTP requests
to OpenAI API.
## Solution
Use patch.object() to directly mock methods on the imported handler
instance. Import base_llm_http_handler in the test file and patch like:
patch.object(base_llm_http_handler, 'container_retrieve_handler', ...)
This ensures the mock is applied to the actual object being used,
regardless of import order or caching.
* fix(test): add missing Prometheus metric labels to test_proxy_failure_metrics
Add client_ip, user_agent, model_id labels to expected metric patterns.
These labels were added in PRs #19717 and #19678 but test wasn't updated.
* fix(test_resend_email): use direct mock injection for all email tests
Extend the mock injection pattern used in test_send_email_missing_api_key
to all other tests in the file:
- test_send_email_success
- test_send_email_multiple_recipients
Instead of relying on fixture-based patching and respx mocks which can
fail due to import order and caching issues, directly inject the mock
HTTP client into the logger instance. This ensures mocks are always used
regardless of test execution order.
* fix(test): use patch.object for image_edit and vector_store tests
- test_image_edit_merges_headers_and_extra_headers: import base_llm_http_handler
and use patch.object instead of string path patching
- test_search_uses_registry_credentials: import module and patch via
module.base_llm_http_handler to ensure we patch the right instance
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
When using the generateContent endpoint with non-Google providers like
github_copilot, the extra_headers from model config were not being
forwarded to the underlying litellm.completion/acompletion calls.
This caused providers requiring custom headers (e.g., Editor-Version
for GitHub Copilot authentication) to reject requests with errors like
"missing Editor-Version header for IDE auth".
Changes:
- Forward extra_headers in _prepare_completion_kwargs() handler
- Pass extra_headers explicitly to adapter in generate_content()
- Pass extra_headers explicitly to adapter in agenerate_content_stream()
- Pass extra_headers explicitly to adapter in generate_content_stream()
- Add tests for extra_headers forwarding behavior
- Update existing test to expect extra_headers in passed fields
Co-authored-by: Claude <noreply@anthropic.com>
* Implement fix for thinking_blocks and converse API calls
This fixes Claude's models via the Converse API, which should also fix
Claude Code.
* Add thinking literal
* Fix mypy issues
* Type fix for redacted thinking
* Add voyage model integration in sagemaker
* Add config file logic
* Use already exiting voyage transformation
* refactor code as per comments
* fix merge error
* refactor code as per comments
* refactor code as per comments
* UI new build
* [Fix] router - regression when adding/removing models (#15451)
* fix(router): update model_name_to_deployment_indices on deployment removal
When a deployment is deleted, the model_name_to_deployment_indices map
was not being updated, causing stale index references. This could lead
to incorrect routing behavior when deployments with the same model_name
were dynamically removed.
Changes:
- Update _update_deployment_indices_after_removal to maintain
model_name_to_deployment_indices mapping
- Remove deleted indices and decrement indices greater than removed index
- Clean up empty entries when no deployments remain for a model name
- Update test to verify proper index shifting and cleanup behavior
* fix(router): remove redundant index building during initialization
Remove duplicate index building operations that were causing unnecessary
work during router initialization:
1. Removed redundant `_build_model_id_to_deployment_index_map` call in
__init__ - `set_model_list` already builds all indices from scratch
2. Removed redundant `_build_model_name_index` call at end of
`set_model_list` - the index is already built incrementally via
`_create_deployment` -> `_add_model_to_list_and_index_map`
Both indices (model_id_to_deployment_index_map and
model_name_to_deployment_indices) are properly maintained as lookup
indexes through existing helper methods. This change eliminates O(N)
duplicate work during initialization without any behavioral changes.
The indices continue to be correctly synchronized with model_list on
all operations (add/remove/upsert).
* fix(prometheus): Fix Prometheus metric collection in a multi-workers environment (#14929)
Co-authored-by: sotazhang <sotazhang@tencent.com>
* Add tiered pricing and cost calculation for xai
* Use generic cost calculator
* Resolve conflicts in generated HTML files
* Remove penalty params as supported params for gemini preview model (#15503)
* fix conversion of thinking block
* add application level encryption in SQS (#15512)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* add application level encryption in SQS
* add application level encryption in SQS
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
* [Feat] Bedrock Knowledgebase - return search_response when using /chat/completions API with LiteLLM (#15509)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* add AnthropicCitation
* fix async_post_call_success_deployment_hook
* fix add vector_store_custom_logger to global callbacks
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call
* async_post_call_success_deployment_hook
* add async_post_call_streaming_deployment_hook
* async def test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_streaming(setup_vector_store_registry):
* fix _call_post_streaming_deployment_hook
* fix async_post_call_streaming_deployment_hook
* test update
* docs: Accessing Search Results
* docs KB
* fix chatUI
* fix searchResults
* fix onSearchResults
* fix kb
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* [Feat] Add dynamic rate limits on LiteLLM Gateway (#15518)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* fix: KeyRequestBase
* fix rpm_limit_type
* fix dynamic rate limits
* fix use dynamic limits here
* fix _should_enforce_rate_limit
* fix _should_enforce_rate_limit
* fix counter
* test_dynamic_rate_limiting_v3
* use _create_rate_limit_descriptors
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* Add google rerank endpoint
* Add docs
* fix mypy error
* fix mypy and lint errors
* Add haiku 4.5 integration
* Add haiku 4.5 integration for other regions as well
* Handle citation field correctly
* Fix filtering headers for signature calcs
* Add haiku 4.5 integration (#15650)
---------
Co-authored-by: Leslie Cheng <leslie.cheng5@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
Co-authored-by: Lucas <10226902+LoadingZhang@users.noreply.github.com>
Co-authored-by: sotazhang <sotazhang@tencent.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
This commit adds a comprehensive suite of unit tests for the Google GenAI adapter to ensure compliance with the project's contribution guidelines.
The new tests cover four main areas:
- Request parameter translation
- Streaming response handling
- Router methods for Google GenAI
- Proxy endpoints for Google GenAI
Additionally, this commit includes minor formatting and linting fixes identified during development.
* fix(main.py): handle router custom azure model name for responses api bridge
* fix(responses/handler): ensure azure model name is stripped before sending to provider
Fixes model name error
* fix(google_genai/main.py): handle stream=true being set in kwargs
* docs: cleanup icons from sidebar
* fix(test-litellm.yml): add google-genai to test litellmyml
* fix(main.py): strip 'responses/' from bridge
* fix(main.py): fix linting errors
* fix(types/openai.py): allow item to be none
handle azure streaming response
* fix(base.py): allow extra fields + handle azure item = none value in response output item added event
* fix(main.py): correctly handle removing responses/
* test(test_main.py): add unit tests
