Gemini image usage carries prompt_tokens and friends as extra fields on
ResponseAPIUsage, which collided with the bridge's explicit kwargs and
raised TypeError. Exclude keys the bridge already sets explicitly.
Drop the transform overrides that swapped response.usage to the chat
shape, which broke the /v1/responses client contract. Provider extras
like server_side_tool_usage_details already survive validation via
ResponseAPIUsage extra fields, so the shared usage bridge now carries
them onto the bridged chat Usage generically. The web_search_call
output gate also reads dict output items, since items that fail SDK
validation stay plain dicts, and the chat path gains billing tests.
xAI already converts Responses usage to chat Usage so web_search_calls survive
cost tracking. The chat completions bridge then re-ran the Responses usage
transform and crashed on missing input_tokens. Pass through already-chat Usage
and chat-shaped dumps instead
Cursor's ApplyPatch is a grammar-constrained custom tool; the Responses
surface carries the grammar flat while chat completions wraps the same
fields in a grammar object, so the nested envelope from the previous
commit still 400d at OpenAI (tools[N].custom.format.grammar). Adds a
shared flat to nested format helper pair in prompt_templates/common_utils
used by the cursor messages arm and the chat-to-responses bridge, nests
flat Responses-style tool_choice objects on the cursor arm, flattens chat
custom tool_choice on the chat-to-responses bridge, and maps custom
tool_choice to function tool_choice on the responses-to-chat bridge to
match that bridge's custom-to-function tool downgrade
A second mutation batch scored the previously unmapped mirror files on
current staging. These four generate mutants for the module they are
named after, yet no test in the file executes any of them; their
test-context coverage lands on generic shared machinery or, for the
guardrail translation handler remainder, on no litellm line at all.
Eight sibling findings that do exercise a different real module are
kept for retargeting instead of removal.
The Responses API (/v1/responses) usage transform rebuilt prompt token
details and dropped OpenAI's input_tokens_details.cache_write_tokens, so
gpt-5.6 cache-creation tokens were never logged or billed via that route.
Map it in the transform, and make PromptTokensDetailsWrapper keep
cache_write_tokens and cache_creation_tokens in sync on assignment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Append-append conflict at the end of test_mcp_server.py between this branch's aggregate-outcome
tests and the mode-aware preemptive-401 tests from staging; both kept
MCP tool calling worked on /v1/chat/completions and /v1/responses but not on
/v1/messages. Those are the only two surfaces with an MCP gateway entry point,
so a litellm_proxy MCP reference reached Anthropic verbatim inside tools and the
API rejected the request with "Input tag 'mcp' found using 'type' does not match
any of the expected tags". The playground never surfaced this because it dropped
the reference before sending, and disabled the MCP selector for the endpoint.
Add the third entry point in anthropic_messages_handler, ahead of the provider
branch so it covers the native path and both bridges from one place. The gateway
expands the reference against the caller's own credentials and access control,
which is the whole point of routing it through litellm rather than handing the
url to the provider.
/v1/messages needs Anthropic's own tool shape, so transform_mcp_tool_to_anthropic_tool
joins the OpenAI chat and Responses transforms alongside it. The tool loop speaks
tool_use and tool_result rather than OpenAI tool_calls, and reuses the existing
FakeAnthropicMessagesStreamIterator to re-stream the result, the same pattern the
websearch interception already uses on this route. Argument extraction moves into
the shared extractor: an Anthropic tool_use block carries its arguments under
`input`, and reading only `arguments` failed silently, executing the tool with
every argument dropped.
On the frontend the request builder declared selectedMCPTools and never read it,
so no tools key was ever sent. Wire it through a shared block builder and add the
endpoint to MCP_SUPPORTED_ENDPOINTS, which is what greys the selector out.
Resolves LIT-4517
Resolves LIT-4518
The aggregate MCP tools/list absorbed every per-server failure (upstream 401/403/5xx, timeouts,
network errors) into that server contributing zero tools, making a broken upstream indistinguishable
from a healthy server with no tools; the single-server REST list masked the same failures as
{"tools": [], "error": null, "message": "Successfully retrieved tools"}
Phase 2 of the MCP error-handling framework (LIT-4419): the manager fetch hops now raise a
classified MCPServerListError (faults/list_outcomes.py: total classifier, frozen outcome values)
instead of returning [], and each boundary applies the relay-vs-absorb policy matrix. The aggregate
keeps serving the healthy subset but records each server's outcome, surfaced on the tools/list
result _meta under litellm.ai/server_outcomes (the SDK passes a ListToolsResult through unwrapped)
and in spend logs as per_server_list_outcomes. Single-server REST requests relay truthful statuses
(unreachable/upstream_error 502, timeout 504, internal 500) and access denials now surface as real
403s instead of 200 unexpected_error bodies; upstream 403s surface through MCPUpstreamAuthError
like 401s. Outcome wire values carry category and status code only, never upstream prose
Resolves LIT-4421
When a /responses request uses a hosted MCP tool (server_url: litellm_proxy/<label>)
with store=true and the model calls a tool, the gateway auto-executes the tool and
streams one logical response stitched from several upstream responses: an interim
response whose only output is the function_call, then the post-tool answer
B1 (correctness): every streamed event was pinned to the first round's response id,
i.e. the interim response that carries the function_call but no tool output. The
client then continued the next turn from that dangling response and the provider
rejected it with "No tool output found for function call <id>", which on the
streaming path surfaced as a silent empty completion. The fix adopts each
auto-execute round's own response id (the cached id is reset when a follow-up round
starts) so the client continues from the final round, whose stored input chain
includes the function_call_output
B2 (robustness): initial and follow-up call failures were swallowed; the stream
emitted the mcp_list_tools discovery events and then closed with HTTP 200 and no
output and no error. The fix stashes the failure, makes the initial call eagerly in
aresponses_api_with_mcp so a pre-stream failure re-raises as a real 4xx before any
SSE bytes are written, and emits a terminal error event when a follow-up call fails
mid-stream
Adds regression tests covering continuation exposing the final round's response id
rather than the interim tool-call id, a follow-up failure emitting a terminal error
event, and an initial-call failure being stashed for eager re-raise
* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param
- BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each
chunk and raises litellm.APIError for type=error and type=response.failed events
so callers see an exception instead of a benign stream chunk
- rate_limit* codes map to 429; client error codes (invalid_request_error,
context_length_exceeded, etc.) map to 400; all other codes default to 500;
raw integer codes are never used as-is as HTTP status codes
- ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]]
to prevent Pydantic ValidationError on dict-typed param payloads silently
dropping error events before any type inspection
* test(responses-api): add streaming iterator error event tests to CI-covered path
* test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths
* test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk
* fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting
insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees
_handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code
usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost
dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed
* fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status
Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and
5xx in-stream error events now raise MidStreamFallbackError carrying the
mapped APIError so the router's FallbackResponsesStreamWrapper triggers
mid-stream fallback and cooldown; non-retriable 4xx still raise APIError
directly. Status mapping now reads both the OpenAI error type and code
fields, so type-classified client errors (e.g. invalid_request_error with
code invalid_prompt) map to 400 instead of falling through to 500.
* fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting
MidStreamFallbackError was always raised with generated_content="", so the
router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk
and retried with the original input, streaming duplicated content to clients
that had already received partial output. The iterators now accumulate
response.output_text.delta text (mirroring chat's response_uptil_now) and pass
it as generated_content, letting the router build a continuation input via
_build_responses_continuation_input.
* test(responses-api): pin in-stream token limit error to raised APIError
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(responses): preserve reasoning_tokens through chat->responses usage translation
Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.
* test(responses): update assertions to match Optional[int] reasoning_tokens default
* fix(responses): preserve explicit reasoning_tokens=0 in usage translation
Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(mcp): resolve tool name prefix via known server prefixes, not string match
When an MCP server's alias differs from its server_name, tool names are
listed with the alias prefix but _execute_tool_calls compared that prefix
against the server_name stored in tool_server_map. The mismatch silently
skipped prefix stripping, forwarding the fully-prefixed tool name upstream
and causing "Unknown tool" failures. Resolve the actual MCPServer object
and strip using its known prefix forms (alias, server_name, server_id)
instead.
* fix(mcp): preserve tool overrides and scope REST tool listing
Return saved tool display/description overrides from the server table API
so the edit UI reloads them, resolve display names before prefix stripping
on tool calls, and honor mcp_server_name and toolset_name filters on the
REST tools list endpoint.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): inject BYOK credentials on Playground OpenAPI tool calls
Playground and Responses API route MCP execution through call_tool, which
skipped BYOK lookup and never set the OpenAPI auth ContextVar, so upstream
calls went out unauthenticated despite a stored user credential.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(mcp): cover alias-mismatch prefix stripping and display-name reverse mapping
Regression tests for _execute_tool_calls: an MCP server whose alias differs
from its server_name must still have its tool-name prefix stripped correctly,
and a tool called by its configured display name must resolve back to the
original tool name before dispatch.
* fix(mcp): validate tool display names against Bedrock's tool-name pattern
A display name replaces the tool name sent to the LLM provider, so a value
with spaces or other special characters saves successfully but fails every
subsequent Bedrock tool call. Validate tool_name_to_display_name server-side
(create/update payload) against Bedrock's [a-zA-Z0-9_-]+ constraint, and add
matching inline validation plus a save-blocking guard in the Admin UI's
create and edit MCP server forms.
* style(mcp): fix ruff/prettier formatting on CI
No logic changes; satisfies the format checks flagged on PR #32320.
* fix(mcp): fix CI failures on PR - complexity budget and stale test mock
Extract toolset-scope resolution and query-param normalization out of
list_tool_rest_api into helpers to bring it back under the C901 complexity
budget (was 18, now within the 15 threshold).
Add the missing get_mcp_server_by_name stub to the streaming iterator test's
mock manager; the alias-fallback resolution added for tool-name-prefix
stripping calls it unconditionally when _get_mcp_server_from_tool_name misses.
* test(mcp): cover BYOK OpenAPI auth-header helpers to close codecov patch gap
_format_byok_openapi_auth_header, _openapi_forwarded_extra_headers, and
_resolve_byok_mcp_auth_header were only exercised indirectly via a mocked
call_tool test, leaving their branches (auth-type formatting, header
forwarding/stripping, missing-credential 401) uncovered.
* fix(mcp): resolve BYOK auth before queuing the during-hook task
_resolve_byok_mcp_auth_header can raise a 401 when no credential is stored.
Resolving it after during_hook_task was already queued meant a hook's
side effects (audit logging, rate-limit bookkeeping) could run and record
success for a tool call that then fails on the missing credential.
* fix: correct mcp alias routing regressions
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Streaming pass-through for native Anthropic /v1/messages and the /v1/responses
streaming iterator never set logging_obj.completion_start_time, so
_success_handler_helper_fn fell back to completion_start_time = end_time.
Downstream TTFT consumers (Prometheus, OTEL, Langfuse, Admin UI, spend logs
completionStartTime) then reported time-to-first-token equal to total request
duration.
Stamp completion_start_time on the first chunk in PassThroughStreamingHandler.
chunk_processor and BaseResponsesAPIStreamingIterator._process_chunk, mirroring
CustomStreamWrapper for /chat/completions.
Resolves LIT-4185
Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
* fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI
Convert Responses API custom tools to Chat Completions function tools and map
function_call responses back to custom_tool_call output items so Codex CLI gets
the apply_patch round-trip it expects. Preserve and validate allowed_callers
during the custom->function conversion so the Anthropic adapter's caller
allowlist is not silently dropped, which would let a tool meant to be callable
only by another tool be invoked directly by the model. Use modern type
annotations (list/dict/set/X | None) throughout to keep the ruff strict budget
within its ratcheted ceilings.
* fix(responses-bridge): address review feedback on custom tool bridge
Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam
and validate allowed_callers with a strict TypeAdapter so the two new cast()
calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only
tool types (computer_use, image_generation, namespace, shell) instead of
discarding them silently. Return output items as Pydantic models instead of
model_dump()ing every item to a dict, matching the declared return type. Apply
the same None-safe metadata pattern to the request_data paths that still used
setdefault, and drop the unused build_custom_tool_call_item helper.
* fix(responses-bridge): recover custom tool input when arguments is empty
* fix(auth): extract custom tool names for allowlist enforcement on responses route
The Responses guardrail translation handler only extracted function and mcp
tool names, so a key or team restricted by metadata.allowed_tools could invoke
a disallowed tool by declaring it with type custom now that the bridge converts
custom tools into callable Chat Completions function tools. Extract custom tool
names through the same path so check_tools_allowlist rejects them.
* fix(responses-bridge): scope input payload recovery to custom_tool_call items
Recovering tool arguments from the input field on any falsy arguments value
made plain function_call input items with empty arguments and a stray input
key get rewritten into a {"content": ...} envelope, corrupting multi-turn
replay for normal function tools. Gate the recovery on the item type so it
only applies to custom_tool_call items, which are the ones that store their
payload in input.
* fix(responses-bridge): default missing function_call arguments to empty string
With input recovery scoped to custom_tool_call items, a plain function_call
input item without an arguments key left raw_arguments as None and the
downstream str() turned it into the literal string None. Coerce to an empty
string instead, matching the pre-bridge behavior.
---------
Co-authored-by: duanhongyi <duanhongyi@doopai.com>
MCPEnhancedStreamingIterator only auto-executed one round of MCP tool calls.
When a model retried a tool (e.g. after an error) in its follow-up turn, that
second tool call was streamed but never executed, and the response ended with
no final text. Route follow-up calls back through the same completion-check
phase as the initial response, so further tool-call rounds are handled the
same way, capped at MAX_MCP_TOOL_CALL_ROUNDS to avoid an unbounded loop.
* fix(mcp): roll up MCP tool spend to user counters and usage UI
Direct REST MCP tool calls now fire success logging so spend_logs and
user/team rollups include configured mcp_server_cost_info charges.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): gate key-info enrichment to requests missing user_id; fix import order
- Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is
absent, avoiding a cache/DB lookup on every normal LLM request.
- Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure
Add api_key = ANY($2) to the MCP session aggregate query so it is
bounded by the same ownership already applied to the main page query.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix spend logs for call and list mcp tools
* Add tags in mcp logging
* Fix ruff
* fix(lint): replace List/Dict with list/dict in new annotations (UP006)
Replace the 8 new UP006 violations introduced by the mcp-tags changes:
- Optional[List[str]] → Optional[list[str]] for request_tags params
- List[str] return type → list[str] in _get_parent_request_tags
- Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError
* fix(mcp): keep final streaming chunk when draining inner stream fails
* fix: handle MCP logging edge cases
* fix: propagate MCP logging cancellation
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): drop unmappable Responses tools instead of failing the request (LIT-3858)
When an OpenAI Responses request is routed to a Bedrock Converse Anthropic model,
litellm translates the tools array into Bedrock toolConfig. Responses built-in tool
types beyond function (web_search, image_generation, namespace, tool_search, custom)
have no Bedrock equivalent, and previously caused two failures.
A web_search tool is derived into a web_search_options param. Bedrock Anthropic
models do not list web_search_options in get_supported_openai_params, so the request
raised UnsupportedParamsError (HTTP 400) even though it never needed web search. The
derived param is now dropped on the Bedrock chat-completion bridge for models that
do not support it, scoped to Bedrock so other providers are untouched and without
requiring drop_params. Nova still keeps it since it maps to a nova_grounding systemTool.
The remaining non-function tools reached _bedrock_tools_pt and were emitted as junk
litellm_unnamed_tool_N toolSpecs with empty schemas, polluting toolConfig with tools
the model could hallucinate calls to. They are now dropped because they carry neither
an OpenAI function nor an Anthropic input_schema, while mappable function and
input_schema tools survive untouched.
* refactor(responses): drop derived web_search_options via provider config
Greptile flagged that the LIT-3858 fix put Bedrock-specific logic in the
generic Responses->Chat Completion bridge: it imported AmazonConverseConfig
and branched on custom_llm_provider.startswith("bedrock").
Read web_search_options support from each provider's own
get_supported_openai_params instead, so the bridge stays provider-agnostic
and Bedrock capability knowledge lives in the Bedrock config that already
owns it. Behavior is unchanged for the cases the PR targeted (Bedrock
Anthropic drops, Bedrock Nova and OpenAI keep) and now generalizes correctly
to any provider whose config does not support the derived param.
Add a Cohere regression test proving the drop is provider-agnostic; it fails
under the old bedrock-only check and passes now.
* fix(responses): drop derived web_search_options for bedrock_converse alias
Greptile/T-Rex caught that the provider-agnostic drop regressed the
bedrock_converse route: get_supported_openai_params did not map the
bedrock_converse alias (only "bedrock"), so it returned None (unmapped) and
the derived web_search_options was forwarded for
model="bedrock/converse/us.anthropic.claude-sonnet-4-6",
custom_llm_provider="bedrock_converse" instead of being dropped. The previous
startswith("bedrock") check happened to match the alias.
Map bedrock_converse through AmazonConverseConfig in get_supported_openai_params,
mirroring the existing ["bedrock", "bedrock_converse"] pairing in
_strip_model_name. Add regression tests at both levels: the alias now drops the
derived param for Anthropic Converse models, still keeps it for Nova, and the
helper resolves identically to "bedrock".
