* fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed
* fix(proxy): gate the zero-cost 499 row on the success log being enqueued, not merely claimed
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47652 to stable/1.106.x
Cherry-picked from ffb7133d32 (main)
* fix(proxy): scope the search free-tier counter to the request's provider
* fix(proxy): read the provider with get so partial spend payloads still get the free tier
* test(proxy): give the grounded spend payload fixture its provider
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47646 to stable/1.106.x
Cherry-picked from 640f2d865b (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47647 to stable/1.106.x
Cherry-picked from 70ece3501f (main)
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata
* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47639 to stable/1.106.x
Cherry-picked from 2a268951c4 (main)
* fix(proxy): bill partial Responses API streams when the client disconnects
Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.
* fix(proxy): satisfy lint gates for partial Responses API disconnect billing
Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.
* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect
The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.
* Potential fix for pull request finding 'CodeQL / Log Injection'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Backport of #47611 to stable/1.106.x
Cherry-picked from f0ae445cc4 (main)
Every /v1/responses spend log asked its worker to flush the queue right away so a
chained previous_response_id could find the row. Only the chat-completions bridge
reads that row. Native Responses providers hold the conversation upstream, so their
rows now wait for the batch write like chat rows do
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47614 to stable/1.106.x
Cherry-picked from 423362bab2 (main)
* feat(proxy): opt-in free-tier allowance for Google Search grounding spend
* fix(proxy): declare the search free tier counter's service target
* test(cost_map): allow search_free_requests in the cost map schema
* fix(proxy): skip cache hits and never free more than the billed cost in the search free tier
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47494 to stable/1.106.x
Cherry-picked from e63aec029a (main)
_failure_spend_rows yields the final row plus one re-keyed row per lifted billed_failed_attempts entry (<request_id>-attempt-<position>, the attempt's usage, cost, retry count and deployment id), each written through DBSpendUpdateWriter.update_database so it also deducts from key, team and user spend
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47416 to stable/1.106.x
Cherry-picked from ad8ca533ca (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47571 to stable/1.106.x
Cherry-picked from ccd0d4ea49 (main)
* feat(proxy): lift every earlier billed timed-out retry attempt onto the failed request
_billed_earlier_attempts reads the router's previous_models records, keeps the Timeout ones other than the final attempt (matched on deployment_id plus attempted_retries, since the count restarts after a fallback hop and the fallback handler re-logs the exhausting attempt) and prices each at the deployment that served it. post_call_failure_hook lifts the tuple as billed_failed_attempts; nothing writes it yet. previous_models joins the router-reserved metadata fields so a client-planted record never reaches it
* fix(proxy): price earlier timed-out attempts from the prompt estimate only
The final row may lift a mid-stream recovered usage (prompt plus delivered chunks) or a failed_attempts_usage sum; earlier attempts never produced output, so they are priced from _estimated_prompt_usage, and when failed_attempts_usage already bills them into the final row none is lifted again
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47490 to stable/1.106.x
Cherry-picked from 7a4a255d1d (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47564 to stable/1.106.x
Cherry-picked from ce63f48ed2 (main)
PR #47394 withheld cache_control_injection_points from the OpenAI SDK extra_body for
every provider on that path, which also covers litellm_proxy. The downstream LiteLLM
proxy applies the point itself, so litellm_proxy keeps forwarding it while OpenAI and the
other OpenAI-compatible providers still never receive it
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47570 to stable/1.106.x
Cherry-picked from 04e9ed4c52 (main)
get_standard_logging_object_payload decided custom_pricing from
litellm_params alone, so on an Azure PTU spillover response
model_map_information kept the deployment's zeroed PTU entry while
response_cost already used the standard model rate. Reuse the billing
side LitellmLogging._custom_pricing_for so both read the same decision
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47567 to stable/1.106.x
Cherry-picked from 4222523261 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47437 to stable/1.106.x
Cherry-picked from fb555b00dc (main)
* fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row
* fix(proxy): read the provider handoff before the proxy-only error handler stamps it so a pre-handoff 499 is not priced
* fix(proxy): drop the pre-logging handoff read now that pre_call no longer stamps proxy-only failures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47436 to stable/1.106.x
Cherry-picked from 0e7c4908bb (main)
* fix(batches): apply the Vertex regional endpoint uplift to batch cost
batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* test(logging): point the resolver tests at resolve_vertex_location_for_cost
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
Backport of #47466 to stable/1.106.x
Cherry-picked from c6c303a618 (main)
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47460 to stable/1.106.x
Cherry-picked from d29200fa05 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47456 to stable/1.106.x
Cherry-picked from 3fa1587f43 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47435 to stable/1.106.x
Cherry-picked from 7b76b2b15f (main)
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
Backport of #47394 to stable/1.106.x
Cherry-picked from 93b79d2587 (main)
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Backport of #47373 to stable/1.106.x
Cherry-picked from 09e9193514 (main)
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
Backport of #47406 to stable/1.106.x
Cherry-picked from 62ce515d15 (main)
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47183 to stable/1.106.x
Cherry-picked from 53821f6654 (main)
The #47389 adaptation moved the handler's return type to dict[str, object] and kept the Any import, so ruff check reports F401 on this file at 1.106.1 and not at 1.106.0
e46904b546 decided by resolved provider name, so the OpenAI-compatible
/v1/messages passthrough and Bedrock Claude Platform /v1/messages, which
inherit the "anthropic" provider name, and the Bedrock Mantle chat and
/v1/messages configs, which inherit the Bedrock Invoke override, still sent
block_binding where 1.106.0 sent {"type": "adaptive"}. Each in-scope config
now opts in for its own class only, and the Bedrock Invoke display: updates
and clear_thinking rebuilds keep block_binding only when the config carries it
The #46342 pick spread every caller key from a legacy thinking.type=enabled
request into the rebuilt adaptive param, so block_binding and any unknown key
newly reached Bedrock Converse, Vertex AI and Databricks, where 1.106.0 sent
only {"type": "adaptive"}. The rebuild now carries block_binding alone, and
only for Anthropic, Azure AI and Bedrock Invoke. The Bedrock Invoke
display: updates and clear_thinking rebuilds keep block_binding and drop the
other caller keys, as 1.106.0 did
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit e979bd0ecf)
* fix(vector-stores): keep proxy context out of provider vector store metadata
The proxy merged its request context (user_api_key_* fields, endpoint,
user_agent, requester_ip_address, litellm_api_version) and the router's
deployment info into the vector store request's metadata. For vector
stores that field is also OpenAI's own metadata, which is capped at 16
keys, so a store created with a team key was full on creation and every
later metadata update returned 400.
Vector store routes now keep that context in litellm_metadata, like
batches, files and responses, and the router does the same for
avector_store_create. Provider-side metadata holds only what the caller
sent.
Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in
script that removes the proxy-written keys from stores created before
this change. It is a dry run unless --apply is passed.
* fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values
Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values.
* fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata
Vector store routes now keep proxy context in litellm_metadata, but the v3
limiter only read agent_id from metadata, and logging let provider metadata
tags or spend_logs_metadata replace the proxy's values. A string tags value
also crashed get_request_tags.
The limiter now prefers the proxy-resolved agent_id in litellm_metadata.
Logging combines list tags from both buckets (metadata first, deduplicated)
and merges dict spend_logs_metadata with litellm_metadata winning, ignoring
values of the wrong type.
* fix(vector-stores): narrow the logging fallback to mistyped caller values
The tag union and spend_logs_metadata merge changed logged tags and
spend_logs_metadata on every route that has both metadata buckets, deduped
tags quadratically, and stopped background interaction polling from keeping
string tags as sent. Caller values now win exactly as before, and the
proxy's litellm_metadata tags or spend_logs_metadata only replace a caller
value of the wrong type.
The limiter reads agent ids through a pydantic model instead of a cast and
ignores non-string ids.
* fix(vector-stores): log the proxy's full tag list on litellm_metadata routes
On routes that keep proxy context in litellm_metadata, the proxy builds
litellm_metadata.tags from the caller's tags plus key, team and header
tags, but get_request_tags preferred the caller's metadata.tags. A caller
sending metadata.tags on a vector store call dropped the key's tags from
request_tags, which tag spend tracking reads. get_request_tags now prefers
a non-empty litellm_metadata tag list, and falls back to metadata.tags as
before.
* fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets
Moving vector store proxy context into litellm_metadata broke shared readers
(agent limits, request tags, merged logging metadata), and fixing them
widened the change to every litellm_metadata route. This reverts the bucket
move, the router change and the reader fixes, so proxy context stays in
metadata exactly as on main.
Vector store create and update now drop the keys LiteLLM itself writes
(user_api_key*, requester and deployment keys, agent ids) from the metadata
sent to the provider, in the shared optional-param helper both paths use.
The cleanup script reuses the same key list.
* fix(vector-stores): pass non-string metadata keys through the provider metadata filter
* fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls
* fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy
* fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies
* fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed
(cherry picked from commit 4846cfabc7)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts ad9e861ae7. Anthropic, Bedrock Invoke and Vertex all accept a
forced tool_choice next to thinking.type=adaptive and only reject it with
type=enabled, so counting adaptive as enabled fixed no live failure. It did
drop the forced JSON tool, so Claude answered response_format requests in
prose, and it downgraded a required tool_choice to auto on Bedrock Converse
* test(e2e): drop the adaptive thinking tool round trip case
With adaptive thinking the model often skips thinking on the weather prompt, so
turn 1 carries no signed empty block and the case fails before the replay it is
meant to cover. Anthropic also answers the replay with a 200 whether or not the
block is forwarded, so the case could not tell the fix from main. The factory
unit tests pin the outgoing shape instead
* fix(anthropic): only skip the thinking-strip retry for a binding mismatch 400
prefix_mismatch_behavior: error asks Anthropic to reject a block bound to
another conversation. It says nothing about unrelated bad blocks such as a
null signature, so those still get the strip-and-retry self-heal. Also trims
the empty-thinking docstrings to what the code guarantees and types the new
test parameters
* refactor(anthropic): narrow thinking fields with the mapping validator instead of typing.cast
typing.cast is banned by the strict ruff gate (TID251), so the three new casts pushed that rule one over its merge-base count. The existing pydantic mapping validator gives the same typed view without an unchecked cast, and it now has a generic name since it is no longer only used for Claude Code payloads
* refactor(anthropic): read thinking settings through the validated mapping so basedpyright stays at its merge-base count
The Bedrock clear_thinking rewrite still narrowed with a cast and an untyped dict fallback, which added two reportUnknownVariableType errors. It now uses the same pydantic mapping validator, made public for that reuse, and the retry hook validates the request body itself so the handler passes a typed dict instead of an untyped one
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
(cherry picked from commit f470e418ea)
The request-side sanitizer on /v1/messages dropped every thinking block whose
text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1
return exactly that shape by default (thinking.display omitted): the signature
is the preserved reasoning. Replaying the assistant turn as returned therefore
lost the model's reasoning on every multi-turn request, silently (the provider
answers 200 without it), and left preserved-thinking enforcement
(thinking.block_binding) nothing to bind.
Anthropic accepts an empty thinking block that carries a signature and binds it
to its prefix; only an empty block with no signature (the LIT-6357 case, a turn
a non-Anthropic model served through the bridge) is rejected with "each
thinking block must contain thinking". Use the signature-aware predicate
is_empty_unsigned_thinking_block, which #38809 already applied on the emit
side, for the request side too. The unsigned case is still dropped.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 897bfea38d)
A JSON-RPC tools/call on /{server_name}/mcp (or /mcp with x-mcp-servers naming one server) that uses the upstream's bare tool name answered "Tool not found" on every proxy process that had not yet listed that server, because the per-process tool-name map stays empty for a server whose upstream only lists tools with the caller's own credentials, and mcp SDK 2 no longer lists before an uncached call. call_mcp_tool now hands the single scoped server to execute_mcp_tool as requested_server_id, so the existing REST contract lists the server with the caller's headers and routes the bare name to it, and a bare name shared by two servers goes to the one the path names
Co-authored-by: aayushbaluni <73417844+aayushbaluni@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ec28acf322)
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4b7b8efd4c)
Takes the main version of each file, which is the rc copy plus one main commit:
- test_image_generation_e2e.py from #46998 (Nova Canvas reached end of life)
- test_bedrock_converse.py from #45530 (ask an uncached model once more)
- migratedPages.spec.ts from #45863 (/ui/ lands on Home since #45858)
* fix(caching): give RedisClusterCache one async client per event loop (#45784)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit cb73bad48a)
* fix(caching): make Redis Cluster commands wait for a pool slot instead of tripping the circuit breaker (#46488)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ca2f99edac)
---------
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check previous_response_id ownership on /v1/responses/compact
* test(proxy): cover the unmanaged response id escape hatches on every response id route
* test(proxy): audit compact previous_response_id ownership
Add integration cells for the native /v1/responses/compact routes covering owner,
admin, cross-tenant, unmanaged, malformed, duplicated and retained ids, provider
errors, the response cache, id-security settings, an upstream outage, a held
upstream, a worker kill and a proxy restart. Unskip the provider wire cell whose
bug this branch fixes.
* test(proxy): cover the no signing key path on every response id route
Backport of #46408 to rc/1.106.0
Cherry-picked from 389403fede (main)
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers
* fix(cost): drop the typesafe/jev-1.14.0 entry main already prices
* fix(typesafe): send images only with the images header and box questions only to models that answer them
TypeSafe takes assets and input_image parts only when the request carries
TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like
any other provider that cannot take images. Box support now comes from a
supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so
jev-1.13.0 and jev-latest refuse or drop a box question before the call. A
malformed provider_specific_header answers 400 instead of 500.
* fix(typesafe): allow null box confidence and validate asset references and base64
* fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through
---------
Backport of #46596 to rc/1.106.0
Cherry-picked from 3fec5075fa (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price typesafe/jev-1.14.0
TypeSafe released jev-1.14.0 on 2026-10-14 at the same price as jev-1.13.0
($0.042 per 1M input tokens, output free; https://docs.typesafe.ai/models).
Without an entry, calls through the TypeSafe pass-through log a cost of 0:
TypeSafePassthroughLoggingHandler prices from litellm.model_cost["typesafe/<model>"],
and a deployment's model_info price is registered under its model id only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): add max_input_tokens to the typesafe jev rows
---------
Backport of #46594 to rc/1.106.0
Cherry-picked from 9d72bae336 (main)
Co-authored-by: Mike Hyams <241361536+hyamie@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): send images to Cloudflare Clef
OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise.
* fix(decisions): follow drop_params for images and use the call timeout for image downloads
* fix(decisions): drop a malformed images entry on its own under drop_params
* fix(decisions): refuse images next to input whatever drop_params says
* fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs
Refuse more than 4 images for Cloudflare clef before any download, keep the
caller's timeout as one deadline across image fetch retries, accept images
next to input under drop_params, validate image strings and objects, and pass
the caller's image URLs (not inlined bytes) to logging callbacks.
* fix(decisions): check input_image URLs and clef image sizes
* fix(decisions): accept data URL media type parameters and check clef image types
* fix(decisions): count clef images after dropping unsupported types and flag clef vision
---------
Backport of #45734 to rc/1.106.0
Cherry-picked from 2f3ccbe7c7 (main)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): support Clef-omni audio and video
* fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes
The clip patterns now mirror Cloudflare's input schema, so a data URL or
content_type carrying parameters like audio/webm;codecs=opus is accepted.
The four proxy routes get media tests of their own, the backup cost map
is synced with the new Clef-omni rows, and the docs page moves to the
docs repo as a follow-up.
* fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot
Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields
* fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation
* fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models
* fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers
* test(decisions): cover the Fireworks System One router in the media refusal cells
---------
Backport of #45811 to rc/1.106.0
Cherry-picked from a86ad9ebf8 (main)
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(decisions): rename the evaluation model mode to decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): canonicalize the health check mode without rebinding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): replace canonical_model_mode with is_decisions_model_mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): preserve evaluation mode compatibility
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop the rebuilt dashboard bundle from this PR
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(dashboard): format decision mode helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Backport of #45474 to rc/1.106.0
Cherry-picked from 4c532a5713 (main)
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): skip custom metadata labels a metric already carries
A custom metadata label whose sanitized name matches one of a metric's
built-in labels was appended a second time, so every .labels() call on
that metric raised ValueError. With custom_prometheus_metadata_labels:
[project_alias], the per-project per-model rate limit gauges raised on
every request from a project key with per-model limits, and the rest of
the success event (latency, deployment success, total requests) was
skipped.
* test(prometheus): cover shared custom labels on every endpoint, failure, metadata and outage burst
* test(prometheus): drain the shared upstream at the start of each integration test
* test(prometheus): assert the forwarded route, model and stream per endpoint cell
---------
Backport of #46267 to rc/1.106.0
Cherry-picked from b581ebc405 (main)
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Backport of #46086 to rc/1.106.0
Cherry-picked from 0c04d0bdb7 (main)
Adapted to the line: the list_value test helper comes from main (#43981, not on rc/1.106.0), and the pipeline test keeps only its buffered cases, because the explicitly unbuffered case needs #45516, which is not on this line
