An empty content list, or one holding only opaque blocks, still lets the
provider-bound branch replay the summary text. The inspection path treated
any non-None content as final, so that replayed text stayed invisible to
guardrails and token counting.
A reasoning input item that carries only summary text is replayed to the
provider as reasoning_content, so inspection-only callers must see that
text too. They used to fall through to the generic content branch, which
reads content and drops a summary-only item, leaving guardrails and token
counters blind to text the model still receives.
The session lookup reads spend logs straight out of the database, so a
follow-up sent seconds after the turn it chains off found nothing while the
row was still queued in the worker that served it, and the conversation was
dropped without an error. Responses calls now ask the spend-log writer to
flush on its next pass instead of waiting out its poll interval, and the
lookup gives a just-finished turn a short second chance.
Replaying a session also accepted `input` only as a string or a single dict,
so the standard list shape dropped every user turn and left the model with
assistant messages alone.
Write the fallback reasoning item id back to the cache so the
reasoning-done path and the completed snapshot cannot drift apart, and
cover the shared delta id and the snapshot alignment with tests.
Guardrails, token counting and rate limiting share the input transform with
the provider path, so moving reasoning onto reasoning_content hid it from
them. Provider-bound callers opt in with replay_reasoning.
Align the response.completed item IDs by copying each output item rather than
writing to it in place, and move the regression cases into the existing
completion-response and image-generation test modules.
Streaming /v1/responses over the completion bridge minted a fresh resp_{uuid4}
for every response, while spend tracking stored the inner chat completion id as
request_id. The session lookup queries on request_id, so a follow-up sent with
that response id matched no rows and the prior conversation was silently
dropped. The iterator now pulls the first upstream chunk before emitting
response.created, so created, in_progress and completed all carry the same
encoded chat completion id.
Two more ways the same history went missing:
- The session lookup only read spend logs already written to the DB, so a
follow-up sent inside the batch writer's window found nothing. It now also
reads the rows still queued in memory.
- Input was only accepted as a string or a single dict, so the list shape the
Responses API actually sends dropped every user turn from the reconstructed
history.
encrypted_content on a reasoning input item is written by LiteLLM's own
_encode_thinking_blocks as a JSON array of Anthropic/Bedrock thinking
blocks, so decode it back and replay the signed blocks on the assistant
message instead of dropping them. Providers without a native
ResponsesAPIConfig now keep the verifiable chain-of-thought across turns,
and prior-turn reasoning stops reaching the provider as visible
assistant text.
The Chat Completions -> Responses bridge stamped the upstream chatcmpl-*
ID onto message output items, so replaying bridged history into native
OpenAI Responses failed with "Expected an ID that begins with 'msg'".
Image generation calls were minted as chatcmpl-*_img_N instead of ig_*,
and reasoning items used a salted hash() that is not stable across
processes.
Streaming minted msg_* for its incremental events but rebuilt the
response.completed snapshot through the same broken transform, so the
snapshot contradicted the events it had just sent and streaming clients
hit the same 400. The snapshot now reuses the IDs already streamed.
Fixes#27333
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
An unknown reasoning split now falls back to reasoning_tokens=0 in the
chat-to-responses usage translation, since the OpenAI SDK requires
output_tokens_details with an int reasoning_tokens, and the streaming
chunk builder caps the tokenized reasoning estimate at completion_tokens
and fills text_tokens with the remainder
Cursor's ApplyPatch is a grammar-constrained custom tool; the Responses
surface carries the grammar flat while chat completions wraps the same
fields in a grammar object, so the nested envelope from the previous
commit still 400d at OpenAI (tools[N].custom.format.grammar). Adds a
shared flat to nested format helper pair in prompt_templates/common_utils
used by the cursor messages arm and the chat-to-responses bridge, nests
flat Responses-style tool_choice objects on the cursor arm, flattens chat
custom tool_choice on the chat-to-responses bridge, and maps custom
tool_choice to function tool_choice on the responses-to-chat bridge to
match that bridge's custom-to-function tool downgrade
A second mutation batch scored the previously unmapped mirror files on
current staging. These four generate mutants for the module they are
named after, yet no test in the file executes any of them; their
test-context coverage lands on generic shared machinery or, for the
guardrail translation handler remainder, on no litellm line at all.
Eight sibling findings that do exercise a different real module are
kept for retargeting instead of removal.
* fix(responses): preserve reasoning_tokens through chat->responses usage translation
Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.
* test(responses): update assertions to match Optional[int] reasoning_tokens default
* fix(responses): preserve explicit reasoning_tokens=0 in usage translation
Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI
Convert Responses API custom tools to Chat Completions function tools and map
function_call responses back to custom_tool_call output items so Codex CLI gets
the apply_patch round-trip it expects. Preserve and validate allowed_callers
during the custom->function conversion so the Anthropic adapter's caller
allowlist is not silently dropped, which would let a tool meant to be callable
only by another tool be invoked directly by the model. Use modern type
annotations (list/dict/set/X | None) throughout to keep the ruff strict budget
within its ratcheted ceilings.
* fix(responses-bridge): address review feedback on custom tool bridge
Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam
and validate allowed_callers with a strict TypeAdapter so the two new cast()
calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only
tool types (computer_use, image_generation, namespace, shell) instead of
discarding them silently. Return output items as Pydantic models instead of
model_dump()ing every item to a dict, matching the declared return type. Apply
the same None-safe metadata pattern to the request_data paths that still used
setdefault, and drop the unused build_custom_tool_call_item helper.
* fix(responses-bridge): recover custom tool input when arguments is empty
* fix(auth): extract custom tool names for allowlist enforcement on responses route
The Responses guardrail translation handler only extracted function and mcp
tool names, so a key or team restricted by metadata.allowed_tools could invoke
a disallowed tool by declaring it with type custom now that the bridge converts
custom tools into callable Chat Completions function tools. Extract custom tool
names through the same path so check_tools_allowlist rejects them.
* fix(responses-bridge): scope input payload recovery to custom_tool_call items
Recovering tool arguments from the input field on any falsy arguments value
made plain function_call input items with empty arguments and a stray input
key get rewritten into a {"content": ...} envelope, corrupting multi-turn
replay for normal function tools. Gate the recovery on the item type so it
only applies to custom_tool_call items, which are the ones that store their
payload in input.
* fix(responses-bridge): default missing function_call arguments to empty string
With input recovery scoped to custom_tool_call items, a plain function_call
input item without an arguments key left raw_arguments as None and the
downstream str() turned it into the literal string None. Coerce to an empty
string instead, matching the pre-bridge behavior.
---------
Co-authored-by: duanhongyi <duanhongyi@doopai.com>
* fix(bedrock): drop unmappable Responses tools instead of failing the request (LIT-3858)
When an OpenAI Responses request is routed to a Bedrock Converse Anthropic model,
litellm translates the tools array into Bedrock toolConfig. Responses built-in tool
types beyond function (web_search, image_generation, namespace, tool_search, custom)
have no Bedrock equivalent, and previously caused two failures.
A web_search tool is derived into a web_search_options param. Bedrock Anthropic
models do not list web_search_options in get_supported_openai_params, so the request
raised UnsupportedParamsError (HTTP 400) even though it never needed web search. The
derived param is now dropped on the Bedrock chat-completion bridge for models that
do not support it, scoped to Bedrock so other providers are untouched and without
requiring drop_params. Nova still keeps it since it maps to a nova_grounding systemTool.
The remaining non-function tools reached _bedrock_tools_pt and were emitted as junk
litellm_unnamed_tool_N toolSpecs with empty schemas, polluting toolConfig with tools
the model could hallucinate calls to. They are now dropped because they carry neither
an OpenAI function nor an Anthropic input_schema, while mappable function and
input_schema tools survive untouched.
* refactor(responses): drop derived web_search_options via provider config
Greptile flagged that the LIT-3858 fix put Bedrock-specific logic in the
generic Responses->Chat Completion bridge: it imported AmazonConverseConfig
and branched on custom_llm_provider.startswith("bedrock").
Read web_search_options support from each provider's own
get_supported_openai_params instead, so the bridge stays provider-agnostic
and Bedrock capability knowledge lives in the Bedrock config that already
owns it. Behavior is unchanged for the cases the PR targeted (Bedrock
Anthropic drops, Bedrock Nova and OpenAI keep) and now generalizes correctly
to any provider whose config does not support the derived param.
Add a Cohere regression test proving the drop is provider-agnostic; it fails
under the old bedrock-only check and passes now.
* fix(responses): drop derived web_search_options for bedrock_converse alias
Greptile/T-Rex caught that the provider-agnostic drop regressed the
bedrock_converse route: get_supported_openai_params did not map the
bedrock_converse alias (only "bedrock"), so it returned None (unmapped) and
the derived web_search_options was forwarded for
model="bedrock/converse/us.anthropic.claude-sonnet-4-6",
custom_llm_provider="bedrock_converse" instead of being dropped. The previous
startswith("bedrock") check happened to match the alias.
Map bedrock_converse through AmazonConverseConfig in get_supported_openai_params,
mirroring the existing ["bedrock", "bedrock_converse"] pairing in
_strip_model_name. Add regression tests at both levels: the alias now drops the
derived param for Anthropic Converse models, still keeps it for Nova, and the
helper resolves identically to "bedrock".
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes#30926)
* style: format common_utils.py with black
* fix(anthropic): extract api_base from litellm_params in batches/files validate_environment
* fix(anthropic): scope Bearer key check to custom api_base endpoints
* fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives
The Anthropic streaming protocol emits `message_start.usage.output_tokens=1`
as a placeholder cursor; the real cumulative output count only arrives in
the final `message_delta` event. When a stream is cancelled before
`message_delta` lands (common for thinking models on long-tail prompts),
ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left
completion_tokens stuck at 1. Because 1 is truthy, the
`completion_tokens or token_counter(text=...)` fallback in
calculate_usage() never fired, and requests were billed for 1 output
token even when several thousand tokens of text had actually streamed.
Fix: track whether any chunk's completion_tokens exceeded 1
(saw_non_cursor_completion). If the only update we saw was the cursor,
reset completion_tokens to 0 so the text-based fallback estimates from
the real completion content.
Legitimate 1-token completions (model returns "Yes." etc.) are unaffected
in practice — token_counter on a 1-token completion_output also yields
~1, so billing stays approximately correct.
Tests:
- TestAnthropicCursorBug (6 cases) — pins the post-fix behavior
- TestNonAnthropicStreamingIntact (2 cases) — guards against regression on
providers without the cursor pattern
All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests
still pass.
* fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival
Addresses both Greptile P2 threads on PR #30420:
CLASS A — Anthropic-specific heuristic was applied globally
============================================================
The `completion_tokens == 1 and not saw_non_cursor_completion` reset
lived in provider-neutral `streaming_chunk_builder_utils.py`. Any
non-Anthropic provider that legitimately reports completion_tokens=1
in a single usage chunk (perfectly normal for short OpenAI / Bedrock /
Vertex single-token replies with stream_options.include_usage=true)
would have its value silently rewritten to 0 and re-billed via
token_counter — producing a different number than what the provider
actually charged.
Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved
from the first chunk's `_hidden_params` (the same field set by
streaming_handler.py:722 on the live path). Unknown / missing provider
is treated as non-Anthropic and skips the reset, so newer providers and
custom plugins are also safe by default.
CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies
============================================================
Previous condition was `usage_chunk_dict["completion_tokens"] > 1`,
which never fires for an Anthropic stream where the model legitimately
emits exactly one output token (e.g., "Yes."). Anthropic still sends
message_start (output_tokens=1, the cursor) AND message_delta
(output_tokens=1, the real value) — same value, but two distinct usage
events. The old check couldn't tell that apart from a cancelled stream
where only message_start landed.
Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion`
when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR
(2) we've seen >=2 completion-bearing usage events (positive evidence
that message_delta arrived). Cancelled cursor-only streams still have
exactly one event and still hit the reset; cache chunks with
completion_tokens=0 don't count toward the threshold.
Tests
============================================================
- _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default
"anthropic") so the gate is exercised by every existing test —
none of them needed assertion changes besides the legitimate-single-
token case, which now expects exactly 1 (was a fuzzy 0..3 range).
- New: test_anthropic_cache_only_chunks_after_message_start_still_resets
- New: test_non_anthropic_provider_completion_tokens_one_not_reset
- New: test_unknown_provider_completion_tokens_one_not_reset
11/11 tests pass.
* chore: add Co-authored-by trailer for attribution
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
* fix(anthropic): preserve messages cache usage
* style(anthropic): format messages cache usage helper
* fix(anthropic): accept integral float cache token counts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(anthropic): accept integral float cache token counts
* test(anthropic): cover cache usage edge cases
* fix(gemini): preserve thoughtSignature for server-side tool responses
When Gemini API returns toolCall and toolResponse parts, they might have
different thoughtSignatures. Previously, LiteLLM merged them into a single
dict, overwriting the response's thoughtSignature with the call's.
This fix extracts them separately and re-injects them correctly.
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* fix(gemini): address PR comments on thoughtSignature handling
- Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature
- Add missing assertions in existing tests
- Add new unit tests for orphan-response signature handling
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* feat(mcp): include server alias and server_id in mcp_info response
- Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint
- Update rest_endpoints.py to surface alias from server config
- Add test coverage in test_mcp_server.py and test_rest_endpoints.py
Fixes#31015
* fix(proxy): reject non-finite spend via validate_finite_spend
A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a
shared finite-value guard, defined above the litellm.proxy.* imports to
avoid the module-level cyclic-import warning.
* fix(proxy): require admin for any /key/update spend, reject non-finite
Gate the admin check on the presence of `spend` (not a value diff): the
DB spend lags the live cross-pod counter, so an "unchanged" spend on the
non-admin path let a key owner / team member overwrite the live counter
below real usage. Also reject NaN/+-inf spend before the DB write.
* fix(proxy): invalidate spend counter on /user/update spend change
A direct spend change on /user/update wrote the DB row but left the warm
cross-pod counter at the stale value, so enforcement kept reading the old
spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB),
and reject non-finite spend before the write.
* fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244)
The semantic cache's embedding model is a proxy Router alias whose AWS
credentials (aws_role_name, aws_session_name) live only in the Router
deployment's litellm_params. The sync embedding paths called litellm.embedding()
directly, bypassing the Router, so they could neither resolve the alias nor
assume the configured role; cross-account Bedrock semantic caching failed with
"bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup
because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding
during cache construction, while llm_router is still None.
Fix A: make the sync paths mirror the already-correct async paths. A shared,
dependency-injected helper (litellm/caching/_embedding_router.py) decides whether
to route through llm_router.embedding(...) when the model is a Router deployment,
else fall back to direct litellm.embedding(...). Redis and qdrant sync
set_cache/get_cache now precompute the embedding and pass vector= to the backend,
exactly as the async astore/acheck already do. Both async _get_async_embedding
methods are unified onto the same helper and now forward the caller's full
metadata instead of a hand-picked subset.
Fix B (Redis only): defer redisvl index construction from __init__ into a lazy,
memoized llmcache property, so the dimension-probe embedding fires on first cache
use, after llm_router is wired. A failed build is not memoized, so a transient
outage recovers on the next request.
Known limitation: resolve_embedding_router gates on an exact model-name match
(same as the shipped async path); wildcard/alias/team-public routes still fall
back to direct embedding. Tracked as a follow-up.
* fix(cache): harden embedding-router and shrink Any surface (review)
Address review feedback on the semantic-cache aws-role fix (#28244):
- resolve_embedding_router now skips deployment entries missing model_name
instead of raising KeyError on a malformed model_list (Greptile P2);
add a regression test that fails on the old direct-key access.
- Replace the `**kwargs: Any` passthrough on the four cache _get_embedding /
_get_async_embedding helpers with an explicit, typed
`metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only
ever consumed kwargs["metadata"], so this is behavior-preserving, makes the
forwarded field obvious at the call site, and removes three bare-Any
annotations (keeps the strict-rule ANN401 budget within ceiling).
- Note in _build_llmcache that redisvl's dimension-probe embedding adds one
extra billable embedding on the first cache request (Greptile P2).
* fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models
Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist"
Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved
A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop
The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities
acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash
* test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression
Adds the regression coverage codecov flagged on the two responses to completion
bridge guard lines and the bedrock route-prefix helper. The handler tests drive
both the sync and async fallback paths with litellm.completion and
litellm.acompletion mocked, and assert the forwarded kwargs carry
_skip_responses_api_bridge=True, so dropping either flag line fails the suite.
The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer
resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids
still do, exercising both branches of _model_has_route_prefix.
Also aligns update_messages_with_model_file_ids model_id to Optional[str],
matching its Responses API sibling, so the defensive model_info fallback no
longer introduces a new reportArgumentType in completion(); the file-id lookup
narrows model_id before the dict get
* chore(ui): sync generated OpenAPI types for optional test_connection mode
The test_model_connection mode body param default changed from chat to None so
the mode is auto-detected from model capabilities, which makes the field
optional in the proxy OpenAPI spec. Regenerate the committed schema so the
dashboard types match: mode becomes optional and the description and default
JSDoc follow the spec, keeping the Check UI API Types Sync gate green
* refactor(bedrock): match all explicit route prefixes at path-segment boundary
Migrates the remaining substring route checks to the existing
_model_has_route_prefix helper so every explicit route token matches only as a
leading path segment, consistent with get_bedrock_route and the mantle route.
Covers _explicit_converse_route, _explicit_claude_platform_route,
_explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route,
_explicit_converse_like_route, _explicit_async_invoke_route and
_explicit_openai_route. This also stops invoke/ from substring-matching
async_invoke/. Route precedence and order are unchanged, and a note on the
segment invariant is added to the helper docstring
* test(bedrock): cover explicit route prefix segment matching
Exercises all eight migrated _explicit_*_route helpers (converse, converse_like,
invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each
matches its token as a leading path segment and rejects the token glued to a
preceding segment, so reverting any method to the old substring check fails the
suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete
improvement of the segment-boundary migration
* test(proxy): assert negative spend is allowed (one-time grant use-case)
Negative spend is intentionally permitted so admins can grant extra
allowance for the current budget period only, without raising the
recurring budget ceiling. Cover it explicitly in validate_finite_spend
and via the /user/update invalidation test.
* fix(google_genai): forward native generateContent top-level fields
Google's native generateContent REST body carries safetySettings, toolConfig,
cachedContent and labels at the top level as siblings of generationConfig. The
proxy's :generateContent endpoint spread them into agenerate_content as loose
kwargs and then dropped them, so callers had to wrap them in extra_body for them
to take effect; safetySettings, for instance, was silently ignored
The provider config now exposes the native top-level field names and
setup_generate_content_call collects whichever are present, merging them into the
outgoing request body through the existing extra_body merge so they reach Google
verbatim. An explicit extra_body still wins on conflict. The sync
generate_content_stream path now also forwards systemInstruction, matching the
other three entry points
Fixes#12671
Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK
* fix(proxy): resolve env refs for DB-stored models
* fix(proxy): restrict DB env ref resolution
* fix(proxy): block team DB env ref resolution
* fix(lint): resolve ANN401/UP045/C901 strict-gate violations
- Replace Optional[X] with X | None (UP045) in 8 files
- Replace Any return/param types with concrete types or object (ANN401)
- Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix
Users who pass a key already prefixed with "Bearer " get Authorization: Bearer.
All other keys continue to use x-api-key, preserving backward compatibility with
custom api_base endpoints that expect x-api-key rather than Authorization.
Also consolidates get_auth_header to reuse _make_api_key_auth_header helper,
eliminating the duplicated custom-endpoint routing logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base
The backwards-compat change broke existing tests that verify the intentional
Bearer-for-custom-base behavior (Fixes#30926). Restore original logic while
keeping the _make_api_key_auth_header helper for code deduplication.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag
Previously the auth-header switch from x-api-key to Authorization: Bearer
applied unconditionally for non-sk-ant- keys on a custom api_base, silently
breaking existing deployments that proxied to gateways expecting x-api-key.
Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header,
get_anthropic_headers, and get_auth_header. validate_environment reads it from
litellm_params so callers can opt in per-model without any API surface change.
Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981)
DEL was the only Redis cache operation that skipped check_and_fix_namespace,
so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the
namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM
error on deployments with an ACL restricting DEL to the litellm:* pattern,
and a silent no-op on all other deployments since the un-prefixed key was
never stored.
* style(anthropic): reformat common_utils.py with Black (--target-version py312)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: preserve cache metadata and spend counters
* style: apply ruff format to streaming_iterator.py
* refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate
Extract Anthropic message_start cursor reset into
_reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter
invalidation into _invalidate_user_spend_counter_if_changed, keeping both
_calculate_usage_per_chunk and _update_single_user_helper under the
max-complexity ceiling. Use builtin generics in the new signatures so no
new UP006 violations are introduced. Behavior unchanged.
---------
Co-authored-by: rupak-eng <rupakji99@gmail.com>
Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com>
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com>
Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com>
Co-authored-by: Andrii Butko <booandrew23@gmail.com>
Co-authored-by: Kent <kingdooo@gmail.com>
Co-authored-by: kunal2002 <k.nayyar2002@gmail.com>
Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com>
Co-authored-by: jesco-absolut <team@srswti.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Matt Hill <mhill@dataminr.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415)
* fix(pricing): add GitHub Copilot MAI Code Flash pricing
Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(pricing): cover GitHub Copilot MAI Code Flash pricing
Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213)
* fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210)
#28990 added ownership recording for streaming /v1/responses via
_wrap_responses_stream_for_container_ownership, which reads
`getattr(stream_response, 'completed_response', None)` to extract the
ResponsesAPIResponse. The unit test bypassed the Router, so it never
exercised the production wrapping path.
Through the Router (every proxy deployment), the stream is wrapped by
FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set
`self.completed_response = None` and __anext__ only forwarded chunks
— the inner source iterator's terminal event never bubbled up to the
attribute the ownership hook reads, so the hook silently recorded
nothing and every follow-up /v1/containers/<id>/files call returned
403 for non-admin keys.
This commit:
- router.py: pre-resolves the responses-API terminal event tuple
(response.completed / .incomplete / .failed) once per
_aresponses_streaming_iterator call, and has the wrapper's __anext__
sniff each forwarded chunk's .type. First terminal event hit gets
stored on the wrapper's completed_response. Iterator-agnostic — works
for source_iterator AND any future wrapper.
- common_request_processing.py: when _extract_completed_responses_response
returns None we now warn instead of silently skipping. Reporter on
#30210 lost a day to this exact silent skip; the warning surfaces
future regressions of the same shape directly in operator logs.
Fixes#30210
* fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning
CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments
in FallbackResponsesStreamWrapper.__init__:
router.py:2564 self.response = getattr(source_iterator, 'response', None)
router.py:2565 self.model = getattr(source_iterator, 'model', None)
router.py:2566 self.logging_obj = getattr(..., None)
Those lines also exist on litellm_internal_staging and pass mypy there.
Adding the typed terminal-event tuple above the class made the function
body more narrowable, which surfaced the pre-existing mismatch — base
class declares non-Optional types but the bridge path
(LiteLLMCompletionStreamingIterator) legitimately omits these. Keep
the None fallback and silence with type: ignore[assignment].
Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter
which misleads operators when a non-code_interpreter stream aborts.
Generalize to 'any tool container (e.g. code_interpreter)'.
* fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201)
* fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198)
get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0
when they are absent from the raw entry (the price-unknown and free cases
share the same representation). register_model then merges that result back
into litellm.model_cost, which flips a sparse entry from 'no cost keys'
(priced via model name) to 'cost keys = 0' (free).
That defeats _is_cost_explicitly_configured (#24949) on re-registration:
_is_model_cost_zero returns True, common_checks skips every tag / key /
team / user / org budget check for the group, and over-budget traffic
keeps returning 200. Spend keeps recording because cost calc still resolves
by model name, so the symptom is silent and only triggers on the second
register_model pass (router rebuild, /model/update, config sync).
Mirror the existing litellm_provider-None guard one block above and pop
the cost fields from the synthesized result when they are absent from the
raw entry and not in the caller's value. Caller-provided zeros (genuinely
free models, BYOK overrides) are preserved.
Fixes#30198
* fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion
Greptile #30201 review notes:
- the `or`-chain in the raw-entry lookup treated an empty dict (a key
with no fields) as falsy and fell through to the second arm — replace
with explicit `is None` checks so a present-but-empty entry is still
taken at face value.
- the first assertion in `test_router_double_init_keeps_db_model_entry_sparse`
used `in (None, 0)` which passes under the bug condition (cost = 0
matches the tuple); the strong follow-up assertion already covers
every shape, so drop the dead branch.
* fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426)
* fix(bedrock mantle): use unique function-call id for responses->chat tool calls
...
* fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id
The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved.
* fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241)
* fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235)
Router.get_deployment_credentials_with_provider re-validates a
deployment's litellm_params through CredentialLiteLLMParams before
handing them to file/batch/passthrough callers:
return CredentialLiteLLMParams(
**deployment.litellm_params.model_dump(exclude_none=True)
).model_dump(exclude_none=True)
Any field NOT declared on CredentialLiteLLMParams gets silently dropped
on the way through. azure_ad_token was undeclared, so Azure deployments
using OAuth/M2M (azure_ad_token instead of a static api_key) silently
lost their token at the files endpoint and the proxy returned:
Missing credentials. Please pass one of api_key, azure_ad_token,
azure_ad_token_provider, ...
Declare azure_ad_token on CredentialLiteLLMParams alongside api_key /
api_base / api_version so it rides through the round-trip. Static-key
deployments stay unaffected (Optional, default None, dropped by
exclude_none=True). Provider-callable (azure_ad_token_provider) is a
separate concern and out of scope here.
Fixes#30235
* fix(ui-types): regenerate schema.d.ts for new azure_ad_token field
CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check
auto-detected the new field and emitted the exact diff to apply.
Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams,
both get the new azure_ad_token marker next to it.
* fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247)
When the UI sends the callers own user_id (as it does for non-Admin
global roles), _enforce_list_team_v2_access now nulls it out for org
admins so _build_team_list_where_conditions scopes by organization_id
only -- matching the legacy /team/list behavior and the documented intent.
Fixes#30215
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707)
litellm_internal_staging already routes the cachedContents URL through
get_vertex_base_url, fixing the multi-region 404 reported in #29571 —
but carries no test coverage for the actual regression scenario (eu/us
must resolve to the REP host aiplatform.{geo}.rep.googleapis.com).
Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host
assertions (including absence of the old broken {geo}-aiplatform host),
plus regional (us-central1) and global no-regression checks.
* fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245)
* fix(proxy): close upstream LLM stream when client disconnects mid-stream
When a streaming client disconnects, Starlette abandons the response
body iterator without calling aclose(), so the proxy's connection to
the upstream backend stays open until garbage collection, which may
never come. The backend (e.g. vLLM) keeps generating into a dead pipe:
small responses drain invisibly into TCP buffers while large ones block
the backend on a full send buffer indefinitely (observed via lsof as an
ESTABLISHED proxy->backend connection minutes after the client left)
create_response now returns a StreamingResponse subclass that closes
both its body iterator and the wrapped upstream-facing generator in a
shielded finally. The upstream generator is closed directly rather than
through a cascade because aclose() on a never-started generator skips
its body, which would make the cascade a no-op when the client
disconnects before the first chunk is sent.
async_streaming_data_generator also gains the same shielded
finally-aclose that async_data_generator in proxy_server.py already
had, covering the Anthropic and Google SSE paths
With this, killing a streaming client causes the backend to observe the
abort within about a second and free its slot, while completed streams
are unaffected. No flag is needed, unlike the non-streaming opt-in
cancel in #30223: this only releases resources after the client is
already gone and does not change any response a client can observe
Fixes#30244
* fix(proxy): close upstream even when body iterator aclose raises BaseException
Addresses the Greptile finding on #30245: the cleanup loop caught only
Exception while the generator-level cleanup catches BaseException, so a
CancelledError or GeneratorExit escaping body_iterator.aclose() would
skip closing the upstream generator. Both sites now use the same scope
and a regression test pins that the upstream is closed even when the
body iterator explodes with a BaseException
* fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection
The response-level close added for #30244 only worked for SDK-based
providers (e.g. openai), whose streams expose aclose all the way down.
Providers served by base_llm_http_handler (hosted_vllm and most modern
transformation-based providers) wrap a bare response.aiter_lines()
generator in BaseModelResponseIterator, which had no aclose or close at
all, and nothing retained the httpx response object; so
CustomStreamWrapper.aclose() silently did nothing and the upstream
connection stayed open. Verified with a vLLM-style mock: with
hosted_vllm/ the backend streamed all 100 chunks to completion after
the client disconnected, while openai/ aborted at chunk 6
BaseModelResponseIterator now carries an optional http_response and an
aclose() that closes it; make_async_call_stream_helper attaches the
response after building the iterator. With this, hosted_vllm aborts the
backend within ~1.6s of the client dropping, and completed streams are
unaffected
---------
Co-authored-by: kursad <kursad.lacin@brado.net>
* feat(anthropic): surface compaction usage iterations data (#27065)
* feat(anthropic): surface compaction usage iterations data
* style: apply black formatting to fix lint checks
* fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422)
* fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock
* fix(usage): optimize test imports
* feat: add fastCRW search provider (#30434)
* feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203)
* feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider
* libertai: update served endpoints backup + add mode/matrix tests
Addresses review feedback:
- Add libertai to litellm/provider_endpoints_support_backup.json, the file
actually served by GET /public/supported_endpoints (the root
provider_endpoints_support.json already had it).
- Add tests asserting bge-m3 normalizes to mode='embedding' and that the
served matrix lists libertai. embeddings stays false: the JSON-configured
provider path only wires chat routing (OpenAILike embedding handler is
reached only for literal openai_like/llamafile/lm_studio), matching the
llamagate precedent; bge-m3 remains in the cost map for metadata.
---------
Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com>
* feat(provider): add ModelScope as an OpenAI-compatible provider (#28460)
* add ModelScope API support
* add modelscope api support
* update modelscope model list
* add image-genetation support
* update test and multimodal
* fix: address PR review feedback for modelscope provider
* update README
* fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849)
* fix(customer_endpoints): restrict /customer/daily/activity to admin-only
* fix(customer_endpoints): check role before prisma_client guard
* fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563)
* fix(fallbacks): preserve fallback model in SDK fallback responses (#28260)
* fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks
* fix(fallbacks): gate x-litellm-* passthrough to trusted callers only
The previous patch unconditionally let `x-litellm-*` keys bypass the
`llm_provider-` prefix in `process_response_headers`. That function is
also called on raw upstream-provider response headers (e.g. from
`llm_http_handler.py`), so a malicious provider could return
`x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker,
bypassing the proxy model-override guard.
Add a `preserve_litellm_internal_headers` flag (default False). Only
`response_metadata.py`, which re-processes the already-built
`_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes
True. Raw provider header callsites keep the default False, so upstream
`x-litellm-*` still gets the `llm_provider-` prefix.
Adds a regression test for the spoofing case and renames the existing
preserve test to make the trusted-path semantics explicit.
* fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs
* style(core_helpers): apply black formatting
* fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): apply black formatting to modelscope chat transformation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): remove unused AllMessageValues import
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert: restore base_model_iterator.py to original PR state
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget
The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813.
* fix(lint): add @override to modelscope image generation overrides
Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913.
---------
Co-authored-by: Joel Tony <github@jaytau.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com>
Co-authored-by: Nahrin <nahrin@nahrinoda.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Humphrey <a739376838@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com>
Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com>
Co-authored-by: Recep S <22618852+us@users.noreply.github.com>
Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com>
Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com>
Co-authored-by: Rongkun Yan <2493404415@qq.com>
Co-authored-by: Varshith <kvarshithgowda@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes#29665) (#29788)
* feat(responses): add default no-op sign_request to BaseResponsesAPIConfig
* feat(responses): call sign_request after body is final, send signed bytes when signed
* feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path)
* test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency
* feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer
* docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route
* test(responses): cover fake-stream signing order and mantle bearer arg/env precedence
* fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance
* fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked
* fix(bedrock_mantle): sign the compact Responses route too, not just create
* fix(github-copilot): route per-model on /v1/responses based on model info (#29747)
* feat(focus): add GCS destination for FOCUS export (#29751)
* test: add failing tests for FocusGCSDestination
* feat: add FocusGCSDestination reusing GCSBucketBase auth
* feat: register FocusGCSDestination in factory; export from __init__
* fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config
* style: apply Black formatting to gcs_destination and tests
* style: apply Black formatting to factory.py
* fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565)
Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input
request when additionalModelRequestFields: {} or system: [] are present in the
payload. Both fields are optional in CommonRequestObject (total=False) and must
be omitted rather than sent as empty structures.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730)
* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible
Azure OpenAI resources created via the newer "Azure AI Foundry" /
Cognitive Services pathway live on `*.cognitiveservices.azure.com`
subdomains, not the older `openai.azure.com`. Both are valid Azure
OpenAI surfaces in production today.
The OpenAI pass-through cost-tracking handler hard-codes only the older
hostname in five places (four `is_openai_*_route` methods on
OpenAIPassthroughLoggingHandler, plus is_openai_route on
PassThroughEndpointLogging). As a result, calls from newer Azure
deployments are silently classified as "not an OpenAI route", the
dispatch into the cost-tracking handler is skipped, and tokens/cost
never get extracted into LiteLLM_SpendLogs — the row gets written with
prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'.
Reproduced 2026-06-04 against a real Azure OpenAI deployment on
`*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0.
Fix: factor the hostname check into a single helper
`_is_openai_compatible_host` listing all three recognized surfaces
(api.openai.com, openai.azure.com, cognitiveservices.azure.com), and
have all five call sites delegate to it. Purely additive — never
weakens recognition for the originally-supported hostnames.
Adds a test
`test_is_openai_route_recognizes_cognitiveservices_azure_com` that
exercises all four `is_openai_*_route` static methods against
`*.cognitiveservices.azure.com` URLs (positive cases per route + a
small cross-route negative to confirm route-specific path matching
still works on the new hostname).
Out of scope for this PR (separate followup):
- `openai_passthrough_handler` calls chat/completions
`transform_response` on Responses API payloads (`output:` not
`choices:`), which throws inside the dispatch and drops the
SpendLogs row entirely. Recognized + tracked separately.
* ci: trigger fresh run
Empty commit to re-run checks. The previous auth-and-jwt failure was
a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads
in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to
this PR's scope (hostname recognition in pass-through cost tracking).
No code change.
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812)
The Responses API forces a specific function with a top-level name
({"type": "function", "name": "X"}), but _transform_tool_choice only handled the
nested Chat Completions shape and fell through to returning "required" for the flat
form, silently dropping the function name and degrading a forced function call to
force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the
"required" fallback when no name is present.
* Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584)
* Preserve x-anthropic-billing-header system blocks for first-party Anthropic
PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for
every Anthropic target. That block is how the first-party Anthropic API recognizes
Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry
only that block, such as the auto-mode tool-safety classifier, fail with a
misleading 429 rate_limit_error; normal turns still work because they also carry
the "You are Claude Code" identity block.
Gate the strip behind should_strip_billing_metadata(), defaulting to False on the
first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and
overridden to True on the providers that reach these transforms and reject the
block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek
for the messages path). Behavior for those providers is unchanged.
* Strip billing header on Bedrock invoke and Vertex messages pass-through
Two more subclasses reach the gated strip but inherited keep-by-default.
AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request,
which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig
(Vertex messages pass-through) calls super().transform_anthropic_messages_request.
Override should_strip_billing_metadata() to True on both.
Add a parametrized test asserting the flag for every first-party base (False) and
provider subclass (True), covering all overrides, plus a translate_system_message
regression test for the Bedrock invoke path.
* fix(cache): log hashed cache keys (#29890)
* fix(ui): save routing groups as list (#29889)
* Revert "fix(ui): save routing groups as list (#29889)" (#29928)
This reverts commit 9b1f78ffa7.
* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider (#29842)
* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider
Registers parasail in the openai_like JSON provider loader with both
/v1/chat/completions and /v1/responses support. Parasail's Responses API
rejects store:true and any request that omits store, so the loader gains a
force_store_false special_handling flag; the parasail entry sets it and
the generated Responses config overrides store=false on every call. This
keeps callers from hitting "State storage not supported" and matches what
Parasail's docs require.
Adds the PARASAIL enum value, listing under openai_compatible_providers,
provider documentation at docs/my-website/docs/providers/parasail.md, and
a focused unit test file under tests/test_litellm/llms/parasail/ that
covers JSON registration, chat URL construction, Responses URL
construction with PARASAIL_API_BASE override, and the force_store_false
regression in both the caller-sent-store=true and caller-omitted cases.
* fix(parasail): register in provider_endpoints_support, drop in-repo docs
Greptile review feedback. The provider doc belongs in the litellm-docs
repo, not this one's docs/my-website tree; removing it here. Adds the
parasail entry to provider_endpoints_support.json so the
check_provider_folders_documented.py CI check passes (chat_completions
and responses true; others false).
* fix: normalize Anthropic passthrough server tool usage (#29827)
* test(anthropic): cover server_tool_use dict cost tracking
* fix: normalize Anthropic server tool usage
(cherry picked from commit 982f726bed)
* fix: keep server tool usage subscriptable
(cherry picked from commit 70280b9b27)
---------
Co-authored-by: Genmin <joey@joeyroth.com>
* fix(proxy): fix typo generic_role_mappoings -> generic_role_mappings in ui_sso.py (#29753)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(proxy): add disable_budget_reservation general setting (#27639) (#29493)
* feat(proxy): add disable_budget_reservation general setting (#27639)
* feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639)
* docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639)
* ci: re-trigger flaky docker build (prisma generate ECONNRESET)
* fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639)
* feat(gemini_tts): adding support to Gemini TTS languageCode parameters (#29623)
* Adding support to Gemini TTS Language Code parameters
* Mapping Gemini TTS languageCode param in Docstring
* Use snake_case for language_code input keyMapping Gemini TTS languageCode param in Docstring
* Restoring files modified under enterprise/litellm_enterprise due to lint/formatting checks
---------
Co-authored-by: João Garrido <joaogarrido@google.com>
* feat(guardrails): capture user and model metadata in CrowdStrike AIDR (#29517)
* fix(proxy): require OpenAI path segment for shared Azure Cognitive Services domains
Address Greptile review: the `*.cognitiveservices.azure.com` /
`*.openai.azure.com` domains are shared by every Azure Cognitive Service
(Speech, Vision, Language, ...), so a hostname-only substring match
misclassified non-OpenAI Azure traffic as OpenAI routes.
- Replace the substring host test with suffix matching (rejects look-alike
domains like cognitiveservices.azure.com.attacker.example).
- Add `_is_openai_compatible_url` that requires an OpenAI-style path marker
(`/openai/` or `/v1/`) on the shared Azure domains, and use it in
PassThroughEndpointLogging.is_openai_route (previously hostname-only).
- Add negative tests for Azure Speech/Vision paths and look-alike domains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: support Responses input in Redis semantic cache (#29581)
* fix: support responses input in redis semantic cache
* test: cover redis semantic prompt extraction
* test: handle blank redis semantic text fallbacks
* chore: remove async cache dead statement
* test: cover redis semantic cache miss paths
* fix: filter sensitive cache lookup kwargs
* chore: rerun ci after huggingface rate limit
* chore(ui): regenerate dashboard API types (npm run gen:api)
Sync src/lib/http/schema.d.ts with the proxy OpenAPI spec: adds the
disable_budget_reservation general-settings field and picks up the
RateLimitError docstring reindent. Fixes the gen:api CI drift check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(bedrock): assert empty additionalModelRequestFields is omitted
The Converse transformer now drops an empty additionalModelRequestFields
block instead of sending it as `{}`. Update test_bedrock_top_k_param so
models without top_k support (llama3) assert the key is absent rather than
equal to an empty dict.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: codgician <15964984+codgician@users.noreply.github.com>
Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com>
Co-authored-by: Roi <roytev@gmail.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Liam Scott <liam@uilliam.com>
Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
Co-authored-by: Ceder Dens <cederdens@gmail.com>
Co-authored-by: 冯基魁 <56265583+fengjikui@users.noreply.github.com>
Co-authored-by: Kai Huang <kaihuang724@gmail.com>
Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com>
Co-authored-by: Genmin <joey@joeyroth.com>
Co-authored-by: Arnav Bhilwariya <arnavbhilwariya0408@gmail.com>
Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com>
Co-authored-by: João Garrido <48538534+johngarrido@users.noreply.github.com>
Co-authored-by: João Garrido <joaogarrido@google.com>
Co-authored-by: Kenan Yildirim <kenan@kenany.me>
Co-authored-by: Dávid Balatoni <balcsida@gmail.com>
* fix(responses): preserve cache_control in Responses API -> Chat Completion transformation
cache_control injected by AnthropicCacheControlHook was silently dropped when
_transform_responses_api_content_to_chat_completion_content rebuilt content blocks
with only {type, text}. Now copies cache_control through so Anthropic prompt caching
works correctly when using client.responses.create with cache_control_injection_points.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): preserve cache_control for input_image and input_file blocks
Extends the cache_control fix to image and file content blocks, which were
also silently dropping cache_control during the Responses API -> Chat Completion
transformation. Adds tests for all three content block types.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Babysitter <claude@anthropic.com>
* fix(responses): map refusal stop_reason to incomplete status in streaming
Fixes streaming responses API translation where Anthropic's stop_reason="refusal"
was incorrectly translated to status="completed" instead of "incomplete".
Root cause: build_base_response was unconditionally overwriting finish_reason
with None from later chunks, losing the terminal content_filter value.
Changes:
- streaming_chunk_builder_utils: skip None finish_reason values in build_base_response
- streaming_iterator: snapshot chunks before returning pending events (sync path)
- streaming_handler: treat usage-only chunks as meaningful content
- transformation: map finish_reason=refusal to status=incomplete
- tests: add regression tests for refusal handling
Made-with: Cursor
* Fix test
