Gemini image usage carries prompt_tokens and friends as extra fields on
ResponseAPIUsage, which collided with the bridge's explicit kwargs and
raised TypeError. Exclude keys the bridge already sets explicitly.
Drop the transform overrides that swapped response.usage to the chat
shape, which broke the /v1/responses client contract. Provider extras
like server_side_tool_usage_details already survive validation via
ResponseAPIUsage extra fields, so the shared usage bridge now carries
them onto the bridged chat Usage generically. The web_search_call
output gate also reads dict output items, since items that fail SDK
validation stay plain dicts, and the chat path gains billing tests.
xAI already converts Responses usage to chat Usage so web_search_calls survive
cost tracking. The chat completions bridge then re-ran the Responses usage
transform and crashed on missing input_tokens. Pass through already-chat Usage
and chat-shaped dumps instead
Cursor's ApplyPatch is a grammar-constrained custom tool; the Responses
surface carries the grammar flat while chat completions wraps the same
fields in a grammar object, so the nested envelope from the previous
commit still 400d at OpenAI (tools[N].custom.format.grammar). Adds a
shared flat to nested format helper pair in prompt_templates/common_utils
used by the cursor messages arm and the chat-to-responses bridge, nests
flat Responses-style tool_choice objects on the cursor arm, flattens chat
custom tool_choice on the chat-to-responses bridge, and maps custom
tool_choice to function tool_choice on the responses-to-chat bridge to
match that bridge's custom-to-function tool downgrade
The Responses API (/v1/responses) usage transform rebuilt prompt token
details and dropped OpenAI's input_tokens_details.cache_write_tokens, so
gpt-5.6 cache-creation tokens were never logged or billed via that route.
Map it in the transform, and make PromptTokensDetailsWrapper keep
cache_write_tokens and cache_creation_tokens in sync on assignment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Append-append conflict at the end of test_mcp_server.py between this branch's aggregate-outcome
tests and the mode-aware preemptive-401 tests from staging; both kept
MCP tool calling worked on /v1/chat/completions and /v1/responses but not on
/v1/messages. Those are the only two surfaces with an MCP gateway entry point,
so a litellm_proxy MCP reference reached Anthropic verbatim inside tools and the
API rejected the request with "Input tag 'mcp' found using 'type' does not match
any of the expected tags". The playground never surfaced this because it dropped
the reference before sending, and disabled the MCP selector for the endpoint.
Add the third entry point in anthropic_messages_handler, ahead of the provider
branch so it covers the native path and both bridges from one place. The gateway
expands the reference against the caller's own credentials and access control,
which is the whole point of routing it through litellm rather than handing the
url to the provider.
/v1/messages needs Anthropic's own tool shape, so transform_mcp_tool_to_anthropic_tool
joins the OpenAI chat and Responses transforms alongside it. The tool loop speaks
tool_use and tool_result rather than OpenAI tool_calls, and reuses the existing
FakeAnthropicMessagesStreamIterator to re-stream the result, the same pattern the
websearch interception already uses on this route. Argument extraction moves into
the shared extractor: an Anthropic tool_use block carries its arguments under
`input`, and reading only `arguments` failed silently, executing the tool with
every argument dropped.
On the frontend the request builder declared selectedMCPTools and never read it,
so no tools key was ever sent. Wire it through a shared block builder and add the
endpoint to MCP_SUPPORTED_ENDPOINTS, which is what greys the selector out.
Resolves LIT-4517
Resolves LIT-4518
The aggregate MCP tools/list absorbed every per-server failure (upstream 401/403/5xx, timeouts,
network errors) into that server contributing zero tools, making a broken upstream indistinguishable
from a healthy server with no tools; the single-server REST list masked the same failures as
{"tools": [], "error": null, "message": "Successfully retrieved tools"}
Phase 2 of the MCP error-handling framework (LIT-4419): the manager fetch hops now raise a
classified MCPServerListError (faults/list_outcomes.py: total classifier, frozen outcome values)
instead of returning [], and each boundary applies the relay-vs-absorb policy matrix. The aggregate
keeps serving the healthy subset but records each server's outcome, surfaced on the tools/list
result _meta under litellm.ai/server_outcomes (the SDK passes a ListToolsResult through unwrapped)
and in spend logs as per_server_list_outcomes. Single-server REST requests relay truthful statuses
(unreachable/upstream_error 502, timeout 504, internal 500) and access denials now surface as real
403s instead of 200 unexpected_error bodies; upstream 403s surface through MCPUpstreamAuthError
like 401s. Outcome wire values carry category and status code only, never upstream prose
Resolves LIT-4421
When a /responses request uses a hosted MCP tool (server_url: litellm_proxy/<label>)
with store=true and the model calls a tool, the gateway auto-executes the tool and
streams one logical response stitched from several upstream responses: an interim
response whose only output is the function_call, then the post-tool answer
B1 (correctness): every streamed event was pinned to the first round's response id,
i.e. the interim response that carries the function_call but no tool output. The
client then continued the next turn from that dangling response and the provider
rejected it with "No tool output found for function call <id>", which on the
streaming path surfaced as a silent empty completion. The fix adopts each
auto-execute round's own response id (the cached id is reset when a follow-up round
starts) so the client continues from the final round, whose stored input chain
includes the function_call_output
B2 (robustness): initial and follow-up call failures were swallowed; the stream
emitted the mcp_list_tools discovery events and then closed with HTTP 200 and no
output and no error. The fix stashes the failure, makes the initial call eagerly in
aresponses_api_with_mcp so a pre-stream failure re-raises as a real 4xx before any
SSE bytes are written, and emits a terminal error event when a follow-up call fails
mid-stream
Adds regression tests covering continuation exposing the final round's response id
rather than the interim tool-call id, a follow-up failure emitting a terminal error
event, and an initial-call failure being stashed for eager re-raise
* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param
- BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each
chunk and raises litellm.APIError for type=error and type=response.failed events
so callers see an exception instead of a benign stream chunk
- rate_limit* codes map to 429; client error codes (invalid_request_error,
context_length_exceeded, etc.) map to 400; all other codes default to 500;
raw integer codes are never used as-is as HTTP status codes
- ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]]
to prevent Pydantic ValidationError on dict-typed param payloads silently
dropping error events before any type inspection
* test(responses-api): add streaming iterator error event tests to CI-covered path
* test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths
* test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk
* fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting
insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees
_handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code
usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost
dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed
* fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status
Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and
5xx in-stream error events now raise MidStreamFallbackError carrying the
mapped APIError so the router's FallbackResponsesStreamWrapper triggers
mid-stream fallback and cooldown; non-retriable 4xx still raise APIError
directly. Status mapping now reads both the OpenAI error type and code
fields, so type-classified client errors (e.g. invalid_request_error with
code invalid_prompt) map to 400 instead of falling through to 500.
* fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting
MidStreamFallbackError was always raised with generated_content="", so the
router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk
and retried with the original input, streaming duplicated content to clients
that had already received partial output. The iterators now accumulate
response.output_text.delta text (mirroring chat's response_uptil_now) and pass
it as generated_content, letting the router build a continuation input via
_build_responses_continuation_input.
* test(responses-api): pin in-stream token limit error to raised APIError
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(responses): preserve reasoning_tokens through chat->responses usage translation
Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.
* test(responses): update assertions to match Optional[int] reasoning_tokens default
* fix(responses): preserve explicit reasoning_tokens=0 in usage translation
Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(mcp): resolve tool name prefix via known server prefixes, not string match
When an MCP server's alias differs from its server_name, tool names are
listed with the alias prefix but _execute_tool_calls compared that prefix
against the server_name stored in tool_server_map. The mismatch silently
skipped prefix stripping, forwarding the fully-prefixed tool name upstream
and causing "Unknown tool" failures. Resolve the actual MCPServer object
and strip using its known prefix forms (alias, server_name, server_id)
instead.
* fix(mcp): preserve tool overrides and scope REST tool listing
Return saved tool display/description overrides from the server table API
so the edit UI reloads them, resolve display names before prefix stripping
on tool calls, and honor mcp_server_name and toolset_name filters on the
REST tools list endpoint.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): inject BYOK credentials on Playground OpenAPI tool calls
Playground and Responses API route MCP execution through call_tool, which
skipped BYOK lookup and never set the OpenAPI auth ContextVar, so upstream
calls went out unauthenticated despite a stored user credential.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(mcp): cover alias-mismatch prefix stripping and display-name reverse mapping
Regression tests for _execute_tool_calls: an MCP server whose alias differs
from its server_name must still have its tool-name prefix stripped correctly,
and a tool called by its configured display name must resolve back to the
original tool name before dispatch.
* fix(mcp): validate tool display names against Bedrock's tool-name pattern
A display name replaces the tool name sent to the LLM provider, so a value
with spaces or other special characters saves successfully but fails every
subsequent Bedrock tool call. Validate tool_name_to_display_name server-side
(create/update payload) against Bedrock's [a-zA-Z0-9_-]+ constraint, and add
matching inline validation plus a save-blocking guard in the Admin UI's
create and edit MCP server forms.
* style(mcp): fix ruff/prettier formatting on CI
No logic changes; satisfies the format checks flagged on PR #32320.
* fix(mcp): fix CI failures on PR - complexity budget and stale test mock
Extract toolset-scope resolution and query-param normalization out of
list_tool_rest_api into helpers to bring it back under the C901 complexity
budget (was 18, now within the 15 threshold).
Add the missing get_mcp_server_by_name stub to the streaming iterator test's
mock manager; the alias-fallback resolution added for tool-name-prefix
stripping calls it unconditionally when _get_mcp_server_from_tool_name misses.
* test(mcp): cover BYOK OpenAPI auth-header helpers to close codecov patch gap
_format_byok_openapi_auth_header, _openapi_forwarded_extra_headers, and
_resolve_byok_mcp_auth_header were only exercised indirectly via a mocked
call_tool test, leaving their branches (auth-type formatting, header
forwarding/stripping, missing-credential 401) uncovered.
* fix(mcp): resolve BYOK auth before queuing the during-hook task
_resolve_byok_mcp_auth_header can raise a 401 when no credential is stored.
Resolving it after during_hook_task was already queued meant a hook's
side effects (audit logging, rate-limit bookkeeping) could run and record
success for a tool call that then fails on the missing credential.
* fix: correct mcp alias routing regressions
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Streaming pass-through for native Anthropic /v1/messages and the /v1/responses
streaming iterator never set logging_obj.completion_start_time, so
_success_handler_helper_fn fell back to completion_start_time = end_time.
Downstream TTFT consumers (Prometheus, OTEL, Langfuse, Admin UI, spend logs
completionStartTime) then reported time-to-first-token equal to total request
duration.
Stamp completion_start_time on the first chunk in PassThroughStreamingHandler.
chunk_processor and BaseResponsesAPIStreamingIterator._process_chunk, mirroring
CustomStreamWrapper for /chat/completions.
Resolves LIT-4185
Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
* fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI
Convert Responses API custom tools to Chat Completions function tools and map
function_call responses back to custom_tool_call output items so Codex CLI gets
the apply_patch round-trip it expects. Preserve and validate allowed_callers
during the custom->function conversion so the Anthropic adapter's caller
allowlist is not silently dropped, which would let a tool meant to be callable
only by another tool be invoked directly by the model. Use modern type
annotations (list/dict/set/X | None) throughout to keep the ruff strict budget
within its ratcheted ceilings.
* fix(responses-bridge): address review feedback on custom tool bridge
Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam
and validate allowed_callers with a strict TypeAdapter so the two new cast()
calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only
tool types (computer_use, image_generation, namespace, shell) instead of
discarding them silently. Return output items as Pydantic models instead of
model_dump()ing every item to a dict, matching the declared return type. Apply
the same None-safe metadata pattern to the request_data paths that still used
setdefault, and drop the unused build_custom_tool_call_item helper.
* fix(responses-bridge): recover custom tool input when arguments is empty
* fix(auth): extract custom tool names for allowlist enforcement on responses route
The Responses guardrail translation handler only extracted function and mcp
tool names, so a key or team restricted by metadata.allowed_tools could invoke
a disallowed tool by declaring it with type custom now that the bridge converts
custom tools into callable Chat Completions function tools. Extract custom tool
names through the same path so check_tools_allowlist rejects them.
* fix(responses-bridge): scope input payload recovery to custom_tool_call items
Recovering tool arguments from the input field on any falsy arguments value
made plain function_call input items with empty arguments and a stray input
key get rewritten into a {"content": ...} envelope, corrupting multi-turn
replay for normal function tools. Gate the recovery on the item type so it
only applies to custom_tool_call items, which are the ones that store their
payload in input.
* fix(responses-bridge): default missing function_call arguments to empty string
With input recovery scoped to custom_tool_call items, a plain function_call
input item without an arguments key left raw_arguments as None and the
downstream str() turned it into the literal string None. Coerce to an empty
string instead, matching the pre-bridge behavior.
---------
Co-authored-by: duanhongyi <duanhongyi@doopai.com>
MCPEnhancedStreamingIterator only auto-executed one round of MCP tool calls.
When a model retried a tool (e.g. after an error) in its follow-up turn, that
second tool call was streamed but never executed, and the response ended with
no final text. Route follow-up calls back through the same completion-check
phase as the initial response, so further tool-call rounds are handled the
same way, capped at MAX_MCP_TOOL_CALL_ROUNDS to avoid an unbounded loop.
* fix(mcp): roll up MCP tool spend to user counters and usage UI
Direct REST MCP tool calls now fire success logging so spend_logs and
user/team rollups include configured mcp_server_cost_info charges.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): gate key-info enrichment to requests missing user_id; fix import order
- Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is
absent, avoiding a cache/DB lookup on every normal LLM request.
- Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure
Add api_key = ANY($2) to the MCP session aggregate query so it is
bounded by the same ownership already applied to the main page query.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix spend logs for call and list mcp tools
* Add tags in mcp logging
* Fix ruff
* fix(lint): replace List/Dict with list/dict in new annotations (UP006)
Replace the 8 new UP006 violations introduced by the mcp-tags changes:
- Optional[List[str]] → Optional[list[str]] for request_tags params
- List[str] return type → list[str] in _get_parent_request_tags
- Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError
* fix(mcp): keep final streaming chunk when draining inner stream fails
* fix: handle MCP logging edge cases
* fix: propagate MCP logging cancellation
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): drop unmappable Responses tools instead of failing the request (LIT-3858)
When an OpenAI Responses request is routed to a Bedrock Converse Anthropic model,
litellm translates the tools array into Bedrock toolConfig. Responses built-in tool
types beyond function (web_search, image_generation, namespace, tool_search, custom)
have no Bedrock equivalent, and previously caused two failures.
A web_search tool is derived into a web_search_options param. Bedrock Anthropic
models do not list web_search_options in get_supported_openai_params, so the request
raised UnsupportedParamsError (HTTP 400) even though it never needed web search. The
derived param is now dropped on the Bedrock chat-completion bridge for models that
do not support it, scoped to Bedrock so other providers are untouched and without
requiring drop_params. Nova still keeps it since it maps to a nova_grounding systemTool.
The remaining non-function tools reached _bedrock_tools_pt and were emitted as junk
litellm_unnamed_tool_N toolSpecs with empty schemas, polluting toolConfig with tools
the model could hallucinate calls to. They are now dropped because they carry neither
an OpenAI function nor an Anthropic input_schema, while mappable function and
input_schema tools survive untouched.
* refactor(responses): drop derived web_search_options via provider config
Greptile flagged that the LIT-3858 fix put Bedrock-specific logic in the
generic Responses->Chat Completion bridge: it imported AmazonConverseConfig
and branched on custom_llm_provider.startswith("bedrock").
Read web_search_options support from each provider's own
get_supported_openai_params instead, so the bridge stays provider-agnostic
and Bedrock capability knowledge lives in the Bedrock config that already
owns it. Behavior is unchanged for the cases the PR targeted (Bedrock
Anthropic drops, Bedrock Nova and OpenAI keep) and now generalizes correctly
to any provider whose config does not support the derived param.
Add a Cohere regression test proving the drop is provider-agnostic; it fails
under the old bedrock-only check and passes now.
* fix(responses): drop derived web_search_options for bedrock_converse alias
Greptile/T-Rex caught that the provider-agnostic drop regressed the
bedrock_converse route: get_supported_openai_params did not map the
bedrock_converse alias (only "bedrock"), so it returned None (unmapped) and
the derived web_search_options was forwarded for
model="bedrock/converse/us.anthropic.claude-sonnet-4-6",
custom_llm_provider="bedrock_converse" instead of being dropped. The previous
startswith("bedrock") check happened to match the alias.
Map bedrock_converse through AmazonConverseConfig in get_supported_openai_params,
mirroring the existing ["bedrock", "bedrock_converse"] pairing in
_strip_model_name. Add regression tests at both levels: the alias now drops the
derived param for Anthropic Converse models, still keeps it for Nova, and the
helper resolves identically to "bedrock".
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes#30926)
* style: format common_utils.py with black
* fix(anthropic): extract api_base from litellm_params in batches/files validate_environment
* fix(anthropic): scope Bearer key check to custom api_base endpoints
* fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives
The Anthropic streaming protocol emits `message_start.usage.output_tokens=1`
as a placeholder cursor; the real cumulative output count only arrives in
the final `message_delta` event. When a stream is cancelled before
`message_delta` lands (common for thinking models on long-tail prompts),
ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left
completion_tokens stuck at 1. Because 1 is truthy, the
`completion_tokens or token_counter(text=...)` fallback in
calculate_usage() never fired, and requests were billed for 1 output
token even when several thousand tokens of text had actually streamed.
Fix: track whether any chunk's completion_tokens exceeded 1
(saw_non_cursor_completion). If the only update we saw was the cursor,
reset completion_tokens to 0 so the text-based fallback estimates from
the real completion content.
Legitimate 1-token completions (model returns "Yes." etc.) are unaffected
in practice — token_counter on a 1-token completion_output also yields
~1, so billing stays approximately correct.
Tests:
- TestAnthropicCursorBug (6 cases) — pins the post-fix behavior
- TestNonAnthropicStreamingIntact (2 cases) — guards against regression on
providers without the cursor pattern
All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests
still pass.
* fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival
Addresses both Greptile P2 threads on PR #30420:
CLASS A — Anthropic-specific heuristic was applied globally
============================================================
The `completion_tokens == 1 and not saw_non_cursor_completion` reset
lived in provider-neutral `streaming_chunk_builder_utils.py`. Any
non-Anthropic provider that legitimately reports completion_tokens=1
in a single usage chunk (perfectly normal for short OpenAI / Bedrock /
Vertex single-token replies with stream_options.include_usage=true)
would have its value silently rewritten to 0 and re-billed via
token_counter — producing a different number than what the provider
actually charged.
Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved
from the first chunk's `_hidden_params` (the same field set by
streaming_handler.py:722 on the live path). Unknown / missing provider
is treated as non-Anthropic and skips the reset, so newer providers and
custom plugins are also safe by default.
CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies
============================================================
Previous condition was `usage_chunk_dict["completion_tokens"] > 1`,
which never fires for an Anthropic stream where the model legitimately
emits exactly one output token (e.g., "Yes."). Anthropic still sends
message_start (output_tokens=1, the cursor) AND message_delta
(output_tokens=1, the real value) — same value, but two distinct usage
events. The old check couldn't tell that apart from a cancelled stream
where only message_start landed.
Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion`
when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR
(2) we've seen >=2 completion-bearing usage events (positive evidence
that message_delta arrived). Cancelled cursor-only streams still have
exactly one event and still hit the reset; cache chunks with
completion_tokens=0 don't count toward the threshold.
Tests
============================================================
- _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default
"anthropic") so the gate is exercised by every existing test —
none of them needed assertion changes besides the legitimate-single-
token case, which now expects exactly 1 (was a fuzzy 0..3 range).
- New: test_anthropic_cache_only_chunks_after_message_start_still_resets
- New: test_non_anthropic_provider_completion_tokens_one_not_reset
- New: test_unknown_provider_completion_tokens_one_not_reset
11/11 tests pass.
* chore: add Co-authored-by trailer for attribution
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
* fix(anthropic): preserve messages cache usage
* style(anthropic): format messages cache usage helper
* fix(anthropic): accept integral float cache token counts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(anthropic): accept integral float cache token counts
* test(anthropic): cover cache usage edge cases
* fix(gemini): preserve thoughtSignature for server-side tool responses
When Gemini API returns toolCall and toolResponse parts, they might have
different thoughtSignatures. Previously, LiteLLM merged them into a single
dict, overwriting the response's thoughtSignature with the call's.
This fix extracts them separately and re-injects them correctly.
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* fix(gemini): address PR comments on thoughtSignature handling
- Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature
- Add missing assertions in existing tests
- Add new unit tests for orphan-response signature handling
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* feat(mcp): include server alias and server_id in mcp_info response
- Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint
- Update rest_endpoints.py to surface alias from server config
- Add test coverage in test_mcp_server.py and test_rest_endpoints.py
Fixes#31015
* fix(proxy): reject non-finite spend via validate_finite_spend
A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a
shared finite-value guard, defined above the litellm.proxy.* imports to
avoid the module-level cyclic-import warning.
* fix(proxy): require admin for any /key/update spend, reject non-finite
Gate the admin check on the presence of `spend` (not a value diff): the
DB spend lags the live cross-pod counter, so an "unchanged" spend on the
non-admin path let a key owner / team member overwrite the live counter
below real usage. Also reject NaN/+-inf spend before the DB write.
* fix(proxy): invalidate spend counter on /user/update spend change
A direct spend change on /user/update wrote the DB row but left the warm
cross-pod counter at the stale value, so enforcement kept reading the old
spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB),
and reject non-finite spend before the write.
* fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244)
The semantic cache's embedding model is a proxy Router alias whose AWS
credentials (aws_role_name, aws_session_name) live only in the Router
deployment's litellm_params. The sync embedding paths called litellm.embedding()
directly, bypassing the Router, so they could neither resolve the alias nor
assume the configured role; cross-account Bedrock semantic caching failed with
"bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup
because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding
during cache construction, while llm_router is still None.
Fix A: make the sync paths mirror the already-correct async paths. A shared,
dependency-injected helper (litellm/caching/_embedding_router.py) decides whether
to route through llm_router.embedding(...) when the model is a Router deployment,
else fall back to direct litellm.embedding(...). Redis and qdrant sync
set_cache/get_cache now precompute the embedding and pass vector= to the backend,
exactly as the async astore/acheck already do. Both async _get_async_embedding
methods are unified onto the same helper and now forward the caller's full
metadata instead of a hand-picked subset.
Fix B (Redis only): defer redisvl index construction from __init__ into a lazy,
memoized llmcache property, so the dimension-probe embedding fires on first cache
use, after llm_router is wired. A failed build is not memoized, so a transient
outage recovers on the next request.
Known limitation: resolve_embedding_router gates on an exact model-name match
(same as the shipped async path); wildcard/alias/team-public routes still fall
back to direct embedding. Tracked as a follow-up.
* fix(cache): harden embedding-router and shrink Any surface (review)
Address review feedback on the semantic-cache aws-role fix (#28244):
- resolve_embedding_router now skips deployment entries missing model_name
instead of raising KeyError on a malformed model_list (Greptile P2);
add a regression test that fails on the old direct-key access.
- Replace the `**kwargs: Any` passthrough on the four cache _get_embedding /
_get_async_embedding helpers with an explicit, typed
`metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only
ever consumed kwargs["metadata"], so this is behavior-preserving, makes the
forwarded field obvious at the call site, and removes three bare-Any
annotations (keeps the strict-rule ANN401 budget within ceiling).
- Note in _build_llmcache that redisvl's dimension-probe embedding adds one
extra billable embedding on the first cache request (Greptile P2).
* fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models
Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist"
Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved
A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop
The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities
acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash
* test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression
Adds the regression coverage codecov flagged on the two responses to completion
bridge guard lines and the bedrock route-prefix helper. The handler tests drive
both the sync and async fallback paths with litellm.completion and
litellm.acompletion mocked, and assert the forwarded kwargs carry
_skip_responses_api_bridge=True, so dropping either flag line fails the suite.
The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer
resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids
still do, exercising both branches of _model_has_route_prefix.
Also aligns update_messages_with_model_file_ids model_id to Optional[str],
matching its Responses API sibling, so the defensive model_info fallback no
longer introduces a new reportArgumentType in completion(); the file-id lookup
narrows model_id before the dict get
* chore(ui): sync generated OpenAPI types for optional test_connection mode
The test_model_connection mode body param default changed from chat to None so
the mode is auto-detected from model capabilities, which makes the field
optional in the proxy OpenAPI spec. Regenerate the committed schema so the
dashboard types match: mode becomes optional and the description and default
JSDoc follow the spec, keeping the Check UI API Types Sync gate green
* refactor(bedrock): match all explicit route prefixes at path-segment boundary
Migrates the remaining substring route checks to the existing
_model_has_route_prefix helper so every explicit route token matches only as a
leading path segment, consistent with get_bedrock_route and the mantle route.
Covers _explicit_converse_route, _explicit_claude_platform_route,
_explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route,
_explicit_converse_like_route, _explicit_async_invoke_route and
_explicit_openai_route. This also stops invoke/ from substring-matching
async_invoke/. Route precedence and order are unchanged, and a note on the
segment invariant is added to the helper docstring
* test(bedrock): cover explicit route prefix segment matching
Exercises all eight migrated _explicit_*_route helpers (converse, converse_like,
invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each
matches its token as a leading path segment and rejects the token glued to a
preceding segment, so reverting any method to the old substring check fails the
suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete
improvement of the segment-boundary migration
* test(proxy): assert negative spend is allowed (one-time grant use-case)
Negative spend is intentionally permitted so admins can grant extra
allowance for the current budget period only, without raising the
recurring budget ceiling. Cover it explicitly in validate_finite_spend
and via the /user/update invalidation test.
* fix(google_genai): forward native generateContent top-level fields
Google's native generateContent REST body carries safetySettings, toolConfig,
cachedContent and labels at the top level as siblings of generationConfig. The
proxy's :generateContent endpoint spread them into agenerate_content as loose
kwargs and then dropped them, so callers had to wrap them in extra_body for them
to take effect; safetySettings, for instance, was silently ignored
The provider config now exposes the native top-level field names and
setup_generate_content_call collects whichever are present, merging them into the
outgoing request body through the existing extra_body merge so they reach Google
verbatim. An explicit extra_body still wins on conflict. The sync
generate_content_stream path now also forwards systemInstruction, matching the
other three entry points
Fixes#12671
Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK
* fix(proxy): resolve env refs for DB-stored models
* fix(proxy): restrict DB env ref resolution
* fix(proxy): block team DB env ref resolution
* fix(lint): resolve ANN401/UP045/C901 strict-gate violations
- Replace Optional[X] with X | None (UP045) in 8 files
- Replace Any return/param types with concrete types or object (ANN401)
- Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix
Users who pass a key already prefixed with "Bearer " get Authorization: Bearer.
All other keys continue to use x-api-key, preserving backward compatibility with
custom api_base endpoints that expect x-api-key rather than Authorization.
Also consolidates get_auth_header to reuse _make_api_key_auth_header helper,
eliminating the duplicated custom-endpoint routing logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base
The backwards-compat change broke existing tests that verify the intentional
Bearer-for-custom-base behavior (Fixes#30926). Restore original logic while
keeping the _make_api_key_auth_header helper for code deduplication.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag
Previously the auth-header switch from x-api-key to Authorization: Bearer
applied unconditionally for non-sk-ant- keys on a custom api_base, silently
breaking existing deployments that proxied to gateways expecting x-api-key.
Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header,
get_anthropic_headers, and get_auth_header. validate_environment reads it from
litellm_params so callers can opt in per-model without any API surface change.
Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981)
DEL was the only Redis cache operation that skipped check_and_fix_namespace,
so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the
namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM
error on deployments with an ACL restricting DEL to the litellm:* pattern,
and a silent no-op on all other deployments since the un-prefixed key was
never stored.
* style(anthropic): reformat common_utils.py with Black (--target-version py312)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: preserve cache metadata and spend counters
* style: apply ruff format to streaming_iterator.py
* refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate
Extract Anthropic message_start cursor reset into
_reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter
invalidation into _invalidate_user_spend_counter_if_changed, keeping both
_calculate_usage_per_chunk and _update_single_user_helper under the
max-complexity ceiling. Use builtin generics in the new signatures so no
new UP006 violations are introduced. Behavior unchanged.
---------
Co-authored-by: rupak-eng <rupakji99@gmail.com>
Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com>
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com>
Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com>
Co-authored-by: Andrii Butko <booandrew23@gmail.com>
Co-authored-by: Kent <kingdooo@gmail.com>
Co-authored-by: kunal2002 <k.nayyar2002@gmail.com>
Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com>
Co-authored-by: jesco-absolut <team@srswti.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Matt Hill <mhill@dataminr.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415)
* fix(pricing): add GitHub Copilot MAI Code Flash pricing
Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(pricing): cover GitHub Copilot MAI Code Flash pricing
Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213)
* fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210)
#28990 added ownership recording for streaming /v1/responses via
_wrap_responses_stream_for_container_ownership, which reads
`getattr(stream_response, 'completed_response', None)` to extract the
ResponsesAPIResponse. The unit test bypassed the Router, so it never
exercised the production wrapping path.
Through the Router (every proxy deployment), the stream is wrapped by
FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set
`self.completed_response = None` and __anext__ only forwarded chunks
— the inner source iterator's terminal event never bubbled up to the
attribute the ownership hook reads, so the hook silently recorded
nothing and every follow-up /v1/containers/<id>/files call returned
403 for non-admin keys.
This commit:
- router.py: pre-resolves the responses-API terminal event tuple
(response.completed / .incomplete / .failed) once per
_aresponses_streaming_iterator call, and has the wrapper's __anext__
sniff each forwarded chunk's .type. First terminal event hit gets
stored on the wrapper's completed_response. Iterator-agnostic — works
for source_iterator AND any future wrapper.
- common_request_processing.py: when _extract_completed_responses_response
returns None we now warn instead of silently skipping. Reporter on
#30210 lost a day to this exact silent skip; the warning surfaces
future regressions of the same shape directly in operator logs.
Fixes#30210
* fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning
CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments
in FallbackResponsesStreamWrapper.__init__:
router.py:2564 self.response = getattr(source_iterator, 'response', None)
router.py:2565 self.model = getattr(source_iterator, 'model', None)
router.py:2566 self.logging_obj = getattr(..., None)
Those lines also exist on litellm_internal_staging and pass mypy there.
Adding the typed terminal-event tuple above the class made the function
body more narrowable, which surfaced the pre-existing mismatch — base
class declares non-Optional types but the bridge path
(LiteLLMCompletionStreamingIterator) legitimately omits these. Keep
the None fallback and silence with type: ignore[assignment].
Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter
which misleads operators when a non-code_interpreter stream aborts.
Generalize to 'any tool container (e.g. code_interpreter)'.
* fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201)
* fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198)
get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0
when they are absent from the raw entry (the price-unknown and free cases
share the same representation). register_model then merges that result back
into litellm.model_cost, which flips a sparse entry from 'no cost keys'
(priced via model name) to 'cost keys = 0' (free).
That defeats _is_cost_explicitly_configured (#24949) on re-registration:
_is_model_cost_zero returns True, common_checks skips every tag / key /
team / user / org budget check for the group, and over-budget traffic
keeps returning 200. Spend keeps recording because cost calc still resolves
by model name, so the symptom is silent and only triggers on the second
register_model pass (router rebuild, /model/update, config sync).
Mirror the existing litellm_provider-None guard one block above and pop
the cost fields from the synthesized result when they are absent from the
raw entry and not in the caller's value. Caller-provided zeros (genuinely
free models, BYOK overrides) are preserved.
Fixes#30198
* fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion
Greptile #30201 review notes:
- the `or`-chain in the raw-entry lookup treated an empty dict (a key
with no fields) as falsy and fell through to the second arm — replace
with explicit `is None` checks so a present-but-empty entry is still
taken at face value.
- the first assertion in `test_router_double_init_keeps_db_model_entry_sparse`
used `in (None, 0)` which passes under the bug condition (cost = 0
matches the tuple); the strong follow-up assertion already covers
every shape, so drop the dead branch.
* fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426)
* fix(bedrock mantle): use unique function-call id for responses->chat tool calls
...
* fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id
The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved.
* fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241)
* fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235)
Router.get_deployment_credentials_with_provider re-validates a
deployment's litellm_params through CredentialLiteLLMParams before
handing them to file/batch/passthrough callers:
return CredentialLiteLLMParams(
**deployment.litellm_params.model_dump(exclude_none=True)
).model_dump(exclude_none=True)
Any field NOT declared on CredentialLiteLLMParams gets silently dropped
on the way through. azure_ad_token was undeclared, so Azure deployments
using OAuth/M2M (azure_ad_token instead of a static api_key) silently
lost their token at the files endpoint and the proxy returned:
Missing credentials. Please pass one of api_key, azure_ad_token,
azure_ad_token_provider, ...
Declare azure_ad_token on CredentialLiteLLMParams alongside api_key /
api_base / api_version so it rides through the round-trip. Static-key
deployments stay unaffected (Optional, default None, dropped by
exclude_none=True). Provider-callable (azure_ad_token_provider) is a
separate concern and out of scope here.
Fixes#30235
* fix(ui-types): regenerate schema.d.ts for new azure_ad_token field
CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check
auto-detected the new field and emitted the exact diff to apply.
Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams,
both get the new azure_ad_token marker next to it.
* fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247)
When the UI sends the callers own user_id (as it does for non-Admin
global roles), _enforce_list_team_v2_access now nulls it out for org
admins so _build_team_list_where_conditions scopes by organization_id
only -- matching the legacy /team/list behavior and the documented intent.
Fixes#30215
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707)
litellm_internal_staging already routes the cachedContents URL through
get_vertex_base_url, fixing the multi-region 404 reported in #29571 —
but carries no test coverage for the actual regression scenario (eu/us
must resolve to the REP host aiplatform.{geo}.rep.googleapis.com).
Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host
assertions (including absence of the old broken {geo}-aiplatform host),
plus regional (us-central1) and global no-regression checks.
* fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245)
* fix(proxy): close upstream LLM stream when client disconnects mid-stream
When a streaming client disconnects, Starlette abandons the response
body iterator without calling aclose(), so the proxy's connection to
the upstream backend stays open until garbage collection, which may
never come. The backend (e.g. vLLM) keeps generating into a dead pipe:
small responses drain invisibly into TCP buffers while large ones block
the backend on a full send buffer indefinitely (observed via lsof as an
ESTABLISHED proxy->backend connection minutes after the client left)
create_response now returns a StreamingResponse subclass that closes
both its body iterator and the wrapped upstream-facing generator in a
shielded finally. The upstream generator is closed directly rather than
through a cascade because aclose() on a never-started generator skips
its body, which would make the cascade a no-op when the client
disconnects before the first chunk is sent.
async_streaming_data_generator also gains the same shielded
finally-aclose that async_data_generator in proxy_server.py already
had, covering the Anthropic and Google SSE paths
With this, killing a streaming client causes the backend to observe the
abort within about a second and free its slot, while completed streams
are unaffected. No flag is needed, unlike the non-streaming opt-in
cancel in #30223: this only releases resources after the client is
already gone and does not change any response a client can observe
Fixes#30244
* fix(proxy): close upstream even when body iterator aclose raises BaseException
Addresses the Greptile finding on #30245: the cleanup loop caught only
Exception while the generator-level cleanup catches BaseException, so a
CancelledError or GeneratorExit escaping body_iterator.aclose() would
skip closing the upstream generator. Both sites now use the same scope
and a regression test pins that the upstream is closed even when the
body iterator explodes with a BaseException
* fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection
The response-level close added for #30244 only worked for SDK-based
providers (e.g. openai), whose streams expose aclose all the way down.
Providers served by base_llm_http_handler (hosted_vllm and most modern
transformation-based providers) wrap a bare response.aiter_lines()
generator in BaseModelResponseIterator, which had no aclose or close at
all, and nothing retained the httpx response object; so
CustomStreamWrapper.aclose() silently did nothing and the upstream
connection stayed open. Verified with a vLLM-style mock: with
hosted_vllm/ the backend streamed all 100 chunks to completion after
the client disconnected, while openai/ aborted at chunk 6
BaseModelResponseIterator now carries an optional http_response and an
aclose() that closes it; make_async_call_stream_helper attaches the
response after building the iterator. With this, hosted_vllm aborts the
backend within ~1.6s of the client dropping, and completed streams are
unaffected
---------
Co-authored-by: kursad <kursad.lacin@brado.net>
* feat(anthropic): surface compaction usage iterations data (#27065)
* feat(anthropic): surface compaction usage iterations data
* style: apply black formatting to fix lint checks
* fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422)
* fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock
* fix(usage): optimize test imports
* feat: add fastCRW search provider (#30434)
* feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203)
* feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider
* libertai: update served endpoints backup + add mode/matrix tests
Addresses review feedback:
- Add libertai to litellm/provider_endpoints_support_backup.json, the file
actually served by GET /public/supported_endpoints (the root
provider_endpoints_support.json already had it).
- Add tests asserting bge-m3 normalizes to mode='embedding' and that the
served matrix lists libertai. embeddings stays false: the JSON-configured
provider path only wires chat routing (OpenAILike embedding handler is
reached only for literal openai_like/llamafile/lm_studio), matching the
llamagate precedent; bge-m3 remains in the cost map for metadata.
---------
Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com>
* feat(provider): add ModelScope as an OpenAI-compatible provider (#28460)
* add ModelScope API support
* add modelscope api support
* update modelscope model list
* add image-genetation support
* update test and multimodal
* fix: address PR review feedback for modelscope provider
* update README
* fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849)
* fix(customer_endpoints): restrict /customer/daily/activity to admin-only
* fix(customer_endpoints): check role before prisma_client guard
* fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563)
* fix(fallbacks): preserve fallback model in SDK fallback responses (#28260)
* fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks
* fix(fallbacks): gate x-litellm-* passthrough to trusted callers only
The previous patch unconditionally let `x-litellm-*` keys bypass the
`llm_provider-` prefix in `process_response_headers`. That function is
also called on raw upstream-provider response headers (e.g. from
`llm_http_handler.py`), so a malicious provider could return
`x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker,
bypassing the proxy model-override guard.
Add a `preserve_litellm_internal_headers` flag (default False). Only
`response_metadata.py`, which re-processes the already-built
`_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes
True. Raw provider header callsites keep the default False, so upstream
`x-litellm-*` still gets the `llm_provider-` prefix.
Adds a regression test for the spoofing case and renames the existing
preserve test to make the trusted-path semantics explicit.
* fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs
* style(core_helpers): apply black formatting
* fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): apply black formatting to modelscope chat transformation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): remove unused AllMessageValues import
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert: restore base_model_iterator.py to original PR state
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget
The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813.
* fix(lint): add @override to modelscope image generation overrides
Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913.
---------
Co-authored-by: Joel Tony <github@jaytau.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com>
Co-authored-by: Nahrin <nahrin@nahrinoda.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Humphrey <a739376838@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com>
Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com>
Co-authored-by: Recep S <22618852+us@users.noreply.github.com>
Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com>
Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com>
Co-authored-by: Rongkun Yan <2493404415@qq.com>
Co-authored-by: Varshith <kvarshithgowda@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
