* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes#30926)
* style: format common_utils.py with black
* fix(anthropic): extract api_base from litellm_params in batches/files validate_environment
* fix(anthropic): scope Bearer key check to custom api_base endpoints
* fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives
The Anthropic streaming protocol emits `message_start.usage.output_tokens=1`
as a placeholder cursor; the real cumulative output count only arrives in
the final `message_delta` event. When a stream is cancelled before
`message_delta` lands (common for thinking models on long-tail prompts),
ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left
completion_tokens stuck at 1. Because 1 is truthy, the
`completion_tokens or token_counter(text=...)` fallback in
calculate_usage() never fired, and requests were billed for 1 output
token even when several thousand tokens of text had actually streamed.
Fix: track whether any chunk's completion_tokens exceeded 1
(saw_non_cursor_completion). If the only update we saw was the cursor,
reset completion_tokens to 0 so the text-based fallback estimates from
the real completion content.
Legitimate 1-token completions (model returns "Yes." etc.) are unaffected
in practice — token_counter on a 1-token completion_output also yields
~1, so billing stays approximately correct.
Tests:
- TestAnthropicCursorBug (6 cases) — pins the post-fix behavior
- TestNonAnthropicStreamingIntact (2 cases) — guards against regression on
providers without the cursor pattern
All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests
still pass.
* fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival
Addresses both Greptile P2 threads on PR #30420:
CLASS A — Anthropic-specific heuristic was applied globally
============================================================
The `completion_tokens == 1 and not saw_non_cursor_completion` reset
lived in provider-neutral `streaming_chunk_builder_utils.py`. Any
non-Anthropic provider that legitimately reports completion_tokens=1
in a single usage chunk (perfectly normal for short OpenAI / Bedrock /
Vertex single-token replies with stream_options.include_usage=true)
would have its value silently rewritten to 0 and re-billed via
token_counter — producing a different number than what the provider
actually charged.
Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved
from the first chunk's `_hidden_params` (the same field set by
streaming_handler.py:722 on the live path). Unknown / missing provider
is treated as non-Anthropic and skips the reset, so newer providers and
custom plugins are also safe by default.
CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies
============================================================
Previous condition was `usage_chunk_dict["completion_tokens"] > 1`,
which never fires for an Anthropic stream where the model legitimately
emits exactly one output token (e.g., "Yes."). Anthropic still sends
message_start (output_tokens=1, the cursor) AND message_delta
(output_tokens=1, the real value) — same value, but two distinct usage
events. The old check couldn't tell that apart from a cancelled stream
where only message_start landed.
Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion`
when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR
(2) we've seen >=2 completion-bearing usage events (positive evidence
that message_delta arrived). Cancelled cursor-only streams still have
exactly one event and still hit the reset; cache chunks with
completion_tokens=0 don't count toward the threshold.
Tests
============================================================
- _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default
"anthropic") so the gate is exercised by every existing test —
none of them needed assertion changes besides the legitimate-single-
token case, which now expects exactly 1 (was a fuzzy 0..3 range).
- New: test_anthropic_cache_only_chunks_after_message_start_still_resets
- New: test_non_anthropic_provider_completion_tokens_one_not_reset
- New: test_unknown_provider_completion_tokens_one_not_reset
11/11 tests pass.
* chore: add Co-authored-by trailer for attribution
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
* fix(anthropic): preserve messages cache usage
* style(anthropic): format messages cache usage helper
* fix(anthropic): accept integral float cache token counts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(anthropic): accept integral float cache token counts
* test(anthropic): cover cache usage edge cases
* fix(gemini): preserve thoughtSignature for server-side tool responses
When Gemini API returns toolCall and toolResponse parts, they might have
different thoughtSignatures. Previously, LiteLLM merged them into a single
dict, overwriting the response's thoughtSignature with the call's.
This fix extracts them separately and re-injects them correctly.
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* fix(gemini): address PR comments on thoughtSignature handling
- Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature
- Add missing assertions in existing tests
- Add new unit tests for orphan-response signature handling
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* feat(mcp): include server alias and server_id in mcp_info response
- Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint
- Update rest_endpoints.py to surface alias from server config
- Add test coverage in test_mcp_server.py and test_rest_endpoints.py
Fixes#31015
* fix(proxy): reject non-finite spend via validate_finite_spend
A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a
shared finite-value guard, defined above the litellm.proxy.* imports to
avoid the module-level cyclic-import warning.
* fix(proxy): require admin for any /key/update spend, reject non-finite
Gate the admin check on the presence of `spend` (not a value diff): the
DB spend lags the live cross-pod counter, so an "unchanged" spend on the
non-admin path let a key owner / team member overwrite the live counter
below real usage. Also reject NaN/+-inf spend before the DB write.
* fix(proxy): invalidate spend counter on /user/update spend change
A direct spend change on /user/update wrote the DB row but left the warm
cross-pod counter at the stale value, so enforcement kept reading the old
spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB),
and reject non-finite spend before the write.
* fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244)
The semantic cache's embedding model is a proxy Router alias whose AWS
credentials (aws_role_name, aws_session_name) live only in the Router
deployment's litellm_params. The sync embedding paths called litellm.embedding()
directly, bypassing the Router, so they could neither resolve the alias nor
assume the configured role; cross-account Bedrock semantic caching failed with
"bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup
because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding
during cache construction, while llm_router is still None.
Fix A: make the sync paths mirror the already-correct async paths. A shared,
dependency-injected helper (litellm/caching/_embedding_router.py) decides whether
to route through llm_router.embedding(...) when the model is a Router deployment,
else fall back to direct litellm.embedding(...). Redis and qdrant sync
set_cache/get_cache now precompute the embedding and pass vector= to the backend,
exactly as the async astore/acheck already do. Both async _get_async_embedding
methods are unified onto the same helper and now forward the caller's full
metadata instead of a hand-picked subset.
Fix B (Redis only): defer redisvl index construction from __init__ into a lazy,
memoized llmcache property, so the dimension-probe embedding fires on first cache
use, after llm_router is wired. A failed build is not memoized, so a transient
outage recovers on the next request.
Known limitation: resolve_embedding_router gates on an exact model-name match
(same as the shipped async path); wildcard/alias/team-public routes still fall
back to direct embedding. Tracked as a follow-up.
* fix(cache): harden embedding-router and shrink Any surface (review)
Address review feedback on the semantic-cache aws-role fix (#28244):
- resolve_embedding_router now skips deployment entries missing model_name
instead of raising KeyError on a malformed model_list (Greptile P2);
add a regression test that fails on the old direct-key access.
- Replace the `**kwargs: Any` passthrough on the four cache _get_embedding /
_get_async_embedding helpers with an explicit, typed
`metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only
ever consumed kwargs["metadata"], so this is behavior-preserving, makes the
forwarded field obvious at the call site, and removes three bare-Any
annotations (keeps the strict-rule ANN401 budget within ceiling).
- Note in _build_llmcache that redisvl's dimension-probe embedding adds one
extra billable embedding on the first cache request (Greptile P2).
* fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models
Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist"
Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved
A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop
The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities
acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash
* test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression
Adds the regression coverage codecov flagged on the two responses to completion
bridge guard lines and the bedrock route-prefix helper. The handler tests drive
both the sync and async fallback paths with litellm.completion and
litellm.acompletion mocked, and assert the forwarded kwargs carry
_skip_responses_api_bridge=True, so dropping either flag line fails the suite.
The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer
resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids
still do, exercising both branches of _model_has_route_prefix.
Also aligns update_messages_with_model_file_ids model_id to Optional[str],
matching its Responses API sibling, so the defensive model_info fallback no
longer introduces a new reportArgumentType in completion(); the file-id lookup
narrows model_id before the dict get
* chore(ui): sync generated OpenAPI types for optional test_connection mode
The test_model_connection mode body param default changed from chat to None so
the mode is auto-detected from model capabilities, which makes the field
optional in the proxy OpenAPI spec. Regenerate the committed schema so the
dashboard types match: mode becomes optional and the description and default
JSDoc follow the spec, keeping the Check UI API Types Sync gate green
* refactor(bedrock): match all explicit route prefixes at path-segment boundary
Migrates the remaining substring route checks to the existing
_model_has_route_prefix helper so every explicit route token matches only as a
leading path segment, consistent with get_bedrock_route and the mantle route.
Covers _explicit_converse_route, _explicit_claude_platform_route,
_explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route,
_explicit_converse_like_route, _explicit_async_invoke_route and
_explicit_openai_route. This also stops invoke/ from substring-matching
async_invoke/. Route precedence and order are unchanged, and a note on the
segment invariant is added to the helper docstring
* test(bedrock): cover explicit route prefix segment matching
Exercises all eight migrated _explicit_*_route helpers (converse, converse_like,
invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each
matches its token as a leading path segment and rejects the token glued to a
preceding segment, so reverting any method to the old substring check fails the
suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete
improvement of the segment-boundary migration
* test(proxy): assert negative spend is allowed (one-time grant use-case)
Negative spend is intentionally permitted so admins can grant extra
allowance for the current budget period only, without raising the
recurring budget ceiling. Cover it explicitly in validate_finite_spend
and via the /user/update invalidation test.
* fix(google_genai): forward native generateContent top-level fields
Google's native generateContent REST body carries safetySettings, toolConfig,
cachedContent and labels at the top level as siblings of generationConfig. The
proxy's :generateContent endpoint spread them into agenerate_content as loose
kwargs and then dropped them, so callers had to wrap them in extra_body for them
to take effect; safetySettings, for instance, was silently ignored
The provider config now exposes the native top-level field names and
setup_generate_content_call collects whichever are present, merging them into the
outgoing request body through the existing extra_body merge so they reach Google
verbatim. An explicit extra_body still wins on conflict. The sync
generate_content_stream path now also forwards systemInstruction, matching the
other three entry points
Fixes#12671
Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK
* fix(proxy): resolve env refs for DB-stored models
* fix(proxy): restrict DB env ref resolution
* fix(proxy): block team DB env ref resolution
* fix(lint): resolve ANN401/UP045/C901 strict-gate violations
- Replace Optional[X] with X | None (UP045) in 8 files
- Replace Any return/param types with concrete types or object (ANN401)
- Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix
Users who pass a key already prefixed with "Bearer " get Authorization: Bearer.
All other keys continue to use x-api-key, preserving backward compatibility with
custom api_base endpoints that expect x-api-key rather than Authorization.
Also consolidates get_auth_header to reuse _make_api_key_auth_header helper,
eliminating the duplicated custom-endpoint routing logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base
The backwards-compat change broke existing tests that verify the intentional
Bearer-for-custom-base behavior (Fixes#30926). Restore original logic while
keeping the _make_api_key_auth_header helper for code deduplication.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag
Previously the auth-header switch from x-api-key to Authorization: Bearer
applied unconditionally for non-sk-ant- keys on a custom api_base, silently
breaking existing deployments that proxied to gateways expecting x-api-key.
Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header,
get_anthropic_headers, and get_auth_header. validate_environment reads it from
litellm_params so callers can opt in per-model without any API surface change.
Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981)
DEL was the only Redis cache operation that skipped check_and_fix_namespace,
so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the
namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM
error on deployments with an ACL restricting DEL to the litellm:* pattern,
and a silent no-op on all other deployments since the un-prefixed key was
never stored.
* style(anthropic): reformat common_utils.py with Black (--target-version py312)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: preserve cache metadata and spend counters
* style: apply ruff format to streaming_iterator.py
* refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate
Extract Anthropic message_start cursor reset into
_reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter
invalidation into _invalidate_user_spend_counter_if_changed, keeping both
_calculate_usage_per_chunk and _update_single_user_helper under the
max-complexity ceiling. Use builtin generics in the new signatures so no
new UP006 violations are introduced. Behavior unchanged.
---------
Co-authored-by: rupak-eng <rupakji99@gmail.com>
Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com>
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com>
Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com>
Co-authored-by: Andrii Butko <booandrew23@gmail.com>
Co-authored-by: Kent <kingdooo@gmail.com>
Co-authored-by: kunal2002 <k.nayyar2002@gmail.com>
Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com>
Co-authored-by: jesco-absolut <team@srswti.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Matt Hill <mhill@dataminr.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(google_genai): preserve complete SSE events in image streaming
Use iter_lines/aiter_lines instead of byte chunking so large inlineData
base64 payloads from Vertex/Gemini streamGenerateContent are not split
across events, which caused truncated JSON and SDK parse failures.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(google_genai): buffer SSE lines until event delimiter
Assemble multi-field SSE events on blank-line boundaries instead of
terminating each field line individually.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): update google_ai_studio mocks from aiter_bytes to aiter_lines
Streaming iterator was changed to use iter_lines/aiter_lines instead of
iter_bytes/aiter_bytes. Update the two mocked streaming responses in
test_google_ai_studio.py to match.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini): normalize response_schema on native generateContent
The /v1beta/models/{model}:generateContent passthrough forwarded
generationConfig.response_schema verbatim, so schemas containing $defs,
$ref, anyOf-with-null, default, or title were rejected by Gemini even
though /chat/completions already handles them.
GoogleGenAIConfig.transform_generate_content_request now calls a new
_normalize_response_schema helper that mirrors the chat/completions
path: Gemini 2.0+ models get the schema promoted to responseJsonSchema
via _build_json_schema (preserving $defs/$ref natively), older models
keep responseSchema but the schema is flattened with
_build_vertex_schema. VertexAIGoogleGenAIConfig (which overrides the
transform entirely) calls the same helper before building the request.
* fix(gemini): preserve caller-supplied responseJsonSchema when responseSchema co-present
Previously, when both responseJsonSchema and responseSchema were present
on Gemini 2.0+, _normalize_response_schema processed responseJsonSchema
first (no-op normalization) then unconditionally promoted responseSchema
to responseJsonSchema, clobbering the caller-supplied value.
Now skip the promotion (and drop the redundant responseSchema) when the
caller already supplied responseJsonSchema.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore: strip restating comments from response-schema normalize
Drop the docstring on _normalize_response_schema and the two inline
comments that just restated what the surrounding code/asserts already
say. Function name + variable names carry the intent; PR description
covers the why-it-exists context.
* perf(gemini): drop redundant deepcopy on responseJsonSchema normalize
_build_json_schema is a no-op (returns its argument unchanged), so the
deepcopy + round-trip on the responseJsonSchema branch allocated a full
schema copy on every request with no observable effect. Forward the
caller's value as-is, and just move the popped responseSchema value when
promoting on Gemini 2.0+.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* style: remove unneeded comment
* fix(gemini): drop unsupported responseJsonSchema for older models
* test(gemini): add parity test between native and chat schema normalization
Per @Sameerlite review: lock the two Gemini schema-normalization paths
together. If either GoogleGenAIConfig._normalize_response_schema (native
generateContent) or VertexGeminiConfig.apply_response_schema_transformation
(/chat/completions) drifts, the parity test fails — forcing both to be
updated together.
* fix(google_genai): preserve key naming convention in _normalize_response_schema
When the input schema key is snake_case (response_schema), the promoted
JSON schema key should also be snake_case (response_json_schema) instead
of mixing in camelCase (responseJsonSchema). This matters for the Vertex
AI google_genai path which converts all keys to snake_case before
calling _normalize_response_schema.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Gemini API keys embedded in URLs as ?key= query parameters leak through
httpx error tracebacks, which are then captured by traceback.format_exc()
and forwarded to logging callbacks, Slack/Teams alerts, and HTTP client
responses.
Short-term: all httpx.HTTPStatusError handlers now raise
MaskedHTTPStatusError(...) from None, which masks the URL and breaks
exception chaining so the original error never appears in tracebacks.
Long-term: moved all Gemini/Vertex URL constructions from ?key={api_key}
to x-goog-api-key header (Google's documented auth method), so the key
is never in the URL at all. WebSocket realtime is the only exception
since WS clients cannot use custom headers.
Additionally hardened all outbound credential paths:
- WebSocket close reasons now pass through _redact_string()
- Callback pipeline (failure_handler) redacts traceback_exception and
error_str before forwarding to integrations (Langfuse, Datadog, etc.)
- Slack/Teams alert messages redacted in send_llm_exception_alert,
ProxyLogging.failure_handler, and post_call_failure_hook
- HTTP error responses in proxy SSE and health endpoints redacted
- Exception messages in exception_mapping_utils redacted
- print_verbose() stdout output redacted when set_verbose=True
- HTTPHandler.put() now has MaskedHTTPStatusError (was missing)
* litellm_fix_mapped_tests_core: fix test isolation and mock injection issues
## Problem
Four tests in litellm_mapped_tests_core were failing:
1. test_register_model_with_scientific_notation - KeyError due to test isolation issues
2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path
3. test_send_email_missing_api_key - Real API calls despite mocking
4. test_stream_transformation_error_sync - Mock not effective, real API called
## Solution
### test_register_model_with_scientific_notation
- Use unique model name to avoid conflicts with other tests
- Clear LRU caches before test to prevent stale data
- Clean up model_cost entry after test
### test_search_uses_registry_credentials
- Use patch.object() on the actual base_llm_http_handler instance
- String-based patching for instance methods can fail; direct object patching is more reliable
### test_send_email_missing_api_key
- Directly inject mock HTTP client into logger instance
- This bypasses any caching issues that could cause the fixture mock to be ineffective
### test_stream_transformation_error_sync
- Patch litellm.completion directly instead of the handler module's litellm reference
- This ensures the mock is effective regardless of import order
## Regression
These tests were affected by LRU caching added in #19606 and HTTP client caching.
* fix(test): use patch.object for container API tests to fix mock injection
## Problem
test_retrieve_container_basic tests were failing because mocks weren't
being applied correctly. The tests used string-based patching:
patch('litellm.containers.main.base_llm_http_handler')
But base_llm_http_handler is imported at module level, so the mock wasn't
intercepting the actual handler calls, resulting in real HTTP requests
to OpenAI API.
## Solution
Use patch.object() to directly mock methods on the imported handler
instance. Import base_llm_http_handler in the test file and patch like:
patch.object(base_llm_http_handler, 'container_retrieve_handler', ...)
This ensures the mock is applied to the actual object being used,
regardless of import order or caching.
* fix(test): add missing Prometheus metric labels to test_proxy_failure_metrics
Add client_ip, user_agent, model_id labels to expected metric patterns.
These labels were added in PRs #19717 and #19678 but test wasn't updated.
* fix(test_resend_email): use direct mock injection for all email tests
Extend the mock injection pattern used in test_send_email_missing_api_key
to all other tests in the file:
- test_send_email_success
- test_send_email_multiple_recipients
Instead of relying on fixture-based patching and respx mocks which can
fail due to import order and caching issues, directly inject the mock
HTTP client into the logger instance. This ensures mocks are always used
regardless of test execution order.
* fix(test): use patch.object for image_edit and vector_store tests
- test_image_edit_merges_headers_and_extra_headers: import base_llm_http_handler
and use patch.object instead of string path patching
- test_search_uses_registry_credentials: import module and patch via
module.base_llm_http_handler to ensure we patch the right instance
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
When using the generateContent endpoint with non-Google providers like
github_copilot, the extra_headers from model config were not being
forwarded to the underlying litellm.completion/acompletion calls.
This caused providers requiring custom headers (e.g., Editor-Version
for GitHub Copilot authentication) to reject requests with errors like
"missing Editor-Version header for IDE auth".
Changes:
- Forward extra_headers in _prepare_completion_kwargs() handler
- Pass extra_headers explicitly to adapter in generate_content()
- Pass extra_headers explicitly to adapter in agenerate_content_stream()
- Pass extra_headers explicitly to adapter in generate_content_stream()
- Add tests for extra_headers forwarding behavior
- Update existing test to expect extra_headers in passed fields
Co-authored-by: Claude <noreply@anthropic.com>
* Implement fix for thinking_blocks and converse API calls
This fixes Claude's models via the Converse API, which should also fix
Claude Code.
* Add thinking literal
* Fix mypy issues
* Type fix for redacted thinking
* Add voyage model integration in sagemaker
* Add config file logic
* Use already exiting voyage transformation
* refactor code as per comments
* fix merge error
* refactor code as per comments
* refactor code as per comments
* UI new build
* [Fix] router - regression when adding/removing models (#15451)
* fix(router): update model_name_to_deployment_indices on deployment removal
When a deployment is deleted, the model_name_to_deployment_indices map
was not being updated, causing stale index references. This could lead
to incorrect routing behavior when deployments with the same model_name
were dynamically removed.
Changes:
- Update _update_deployment_indices_after_removal to maintain
model_name_to_deployment_indices mapping
- Remove deleted indices and decrement indices greater than removed index
- Clean up empty entries when no deployments remain for a model name
- Update test to verify proper index shifting and cleanup behavior
* fix(router): remove redundant index building during initialization
Remove duplicate index building operations that were causing unnecessary
work during router initialization:
1. Removed redundant `_build_model_id_to_deployment_index_map` call in
__init__ - `set_model_list` already builds all indices from scratch
2. Removed redundant `_build_model_name_index` call at end of
`set_model_list` - the index is already built incrementally via
`_create_deployment` -> `_add_model_to_list_and_index_map`
Both indices (model_id_to_deployment_index_map and
model_name_to_deployment_indices) are properly maintained as lookup
indexes through existing helper methods. This change eliminates O(N)
duplicate work during initialization without any behavioral changes.
The indices continue to be correctly synchronized with model_list on
all operations (add/remove/upsert).
* fix(prometheus): Fix Prometheus metric collection in a multi-workers environment (#14929)
Co-authored-by: sotazhang <sotazhang@tencent.com>
* Add tiered pricing and cost calculation for xai
* Use generic cost calculator
* Resolve conflicts in generated HTML files
* Remove penalty params as supported params for gemini preview model (#15503)
* fix conversion of thinking block
* add application level encryption in SQS (#15512)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* add application level encryption in SQS
* add application level encryption in SQS
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
* [Feat] Bedrock Knowledgebase - return search_response when using /chat/completions API with LiteLLM (#15509)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* add AnthropicCitation
* fix async_post_call_success_deployment_hook
* fix add vector_store_custom_logger to global callbacks
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call
* async_post_call_success_deployment_hook
* add async_post_call_streaming_deployment_hook
* async def test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_streaming(setup_vector_store_registry):
* fix _call_post_streaming_deployment_hook
* fix async_post_call_streaming_deployment_hook
* test update
* docs: Accessing Search Results
* docs KB
* fix chatUI
* fix searchResults
* fix onSearchResults
* fix kb
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* [Feat] Add dynamic rate limits on LiteLLM Gateway (#15518)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* fix: KeyRequestBase
* fix rpm_limit_type
* fix dynamic rate limits
* fix use dynamic limits here
* fix _should_enforce_rate_limit
* fix _should_enforce_rate_limit
* fix counter
* test_dynamic_rate_limiting_v3
* use _create_rate_limit_descriptors
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* Add google rerank endpoint
* Add docs
* fix mypy error
* fix mypy and lint errors
* Add haiku 4.5 integration
* Add haiku 4.5 integration for other regions as well
* Handle citation field correctly
* Fix filtering headers for signature calcs
* Add haiku 4.5 integration (#15650)
---------
Co-authored-by: Leslie Cheng <leslie.cheng5@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
Co-authored-by: Lucas <10226902+LoadingZhang@users.noreply.github.com>
Co-authored-by: sotazhang <sotazhang@tencent.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
This commit adds a comprehensive suite of unit tests for the Google GenAI adapter to ensure compliance with the project's contribution guidelines.
The new tests cover four main areas:
- Request parameter translation
- Streaming response handling
- Router methods for Google GenAI
- Proxy endpoints for Google GenAI
Additionally, this commit includes minor formatting and linting fixes identified during development.
* fix(main.py): handle router custom azure model name for responses api bridge
* fix(responses/handler): ensure azure model name is stripped before sending to provider
Fixes model name error
* fix(google_genai/main.py): handle stream=true being set in kwargs
* docs: cleanup icons from sidebar
* fix(test-litellm.yml): add google-genai to test litellmyml
* fix(main.py): strip 'responses/' from bridge
* fix(main.py): fix linting errors
* fix(types/openai.py): allow item to be none
handle azure streaming response
* fix(base.py): allow extra fields + handle azure item = none value in response output item added event
* fix(main.py): correctly handle removing responses/
* test(test_main.py): add unit tests
