Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47571 to stable/1.105.x.
Cherry-picked from ccd0d4ea49 (main).
* feat(proxy): lift every earlier billed timed-out retry attempt onto the failed request
_billed_earlier_attempts reads the router's previous_models records, keeps the Timeout ones other than the final attempt (matched on deployment_id plus attempted_retries, since the count restarts after a fallback hop and the fallback handler re-logs the exhausting attempt) and prices each at the deployment that served it. post_call_failure_hook lifts the tuple as billed_failed_attempts; nothing writes it yet. previous_models joins the router-reserved metadata fields so a client-planted record never reaches it
* fix(proxy): price earlier timed-out attempts from the prompt estimate only
The final row may lift a mid-stream recovered usage (prompt plus delivered chunks) or a failed_attempts_usage sum; earlier attempts never produced output, so they are priced from _estimated_prompt_usage, and when failed_attempts_usage already bills them into the final row none is lifted again
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47490 to stable/1.105.x.
Cherry-picked from 7a4a255d1d (main).
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47564 to stable/1.105.x.
Cherry-picked from ce63f48ed2 (main).
PR #47394 withheld cache_control_injection_points from the OpenAI SDK extra_body for
every provider on that path, which also covers litellm_proxy. The downstream LiteLLM
proxy applies the point itself, so litellm_proxy keeps forwarding it while OpenAI and the
other OpenAI-compatible providers still never receive it
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47570 to stable/1.105.x.
Cherry-picked from 04e9ed4c52 (main).
get_standard_logging_object_payload decided custom_pricing from
litellm_params alone, so on an Azure PTU spillover response
model_map_information kept the deployment's zeroed PTU entry while
response_cost already used the standard model rate. Reuse the billing
side LitellmLogging._custom_pricing_for so both read the same decision
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47567 to stable/1.105.x.
Cherry-picked from 4222523261 (main).
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47437 to stable/1.105.x.
Cherry-picked from fb555b00dc (main).
* fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row
* fix(proxy): read the provider handoff before the proxy-only error handler stamps it so a pre-handoff 499 is not priced
* fix(proxy): drop the pre-logging handoff read now that pre_call no longer stamps proxy-only failures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47436 to stable/1.105.x.
Cherry-picked from 0e7c4908bb (main).
* fix(batches): apply the Vertex regional endpoint uplift to batch cost
batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* test(logging): point the resolver tests at resolve_vertex_location_for_cost
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
Backport of #47466 to stable/1.105.x.
Cherry-picked from c6c303a618 (main).
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47460 to stable/1.105.x.
Cherry-picked from d29200fa05 (main).
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47456 to stable/1.105.x.
Cherry-picked from 3fa1587f43 (main).
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47435 to stable/1.105.x.
Cherry-picked from 7b76b2b15f (main).
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
Backport of #47394 to stable/1.105.x.
Cherry-picked from 93b79d2587 (main).
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Backport of #47373 to stable/1.105.x.
Cherry-picked from 09e9193514 (main).
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
Backport of #47406 to stable/1.105.x.
Cherry-picked from 62ce515d15 (main).
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47183 to stable/1.105.x.
Cherry-picked from 53821f6654 (main).
Backport of #47419 to stable/1.105.x.
Cherry-picked from 3124797399 (main).
Adapted to the line, on top of the ported helpers in the previous commit:
- runtime_model_cost_registrations reads a per-cost-map store of the fields each register_model call set (_RegisteredPricingFields), which stands in for main's AliasedCostEntry.registered_fields (#41523) so alias entries stay plain dicts. A catalog reload resets it, as main's reload drops registered_fields.
- An inference-profile ARN in model_id is resolved to its profile name inside _bedrock_pricing_info with the line's extract_model_name_from_bedrock_arn, instead of main's get_potential_model_names ARN handling.
- completion_cost keeps the line's candidate list and passes the profile info through; optional_params is read directly (main's _LOGGED_LITELLM_PARAMS is not on the line).
- The router adapters _MODEL_INFO_ADAPTER and _LITELLM_PARAMS_ADAPTER are defined here.
- Tests use the line's private names. Dropped: the AliasedCostEntry routing test, the router cache-only pricing test (main-only fixtures), and the ignore_stamped_cost assertion (main-only parameter).
Prerequisite for #47419 on stable/1.105.x. Hand-ports from main (#41523, #46647):
ModelInfoLookupMiss and lookup_model_info_for_registration, the flat-price override helpers
(_without_overridden_price_variants and its rate-table helpers), and runtime_model_cost_registrations.
runtime_model_cost_registrations reads the line's _runtime_registered_model_cost store instead of
main's AliasedCostEntry.registered_fields, so cost-map alias entries stay plain dicts. Nothing on the
line calls these until #47419 lands, and register_model's behavior is unchanged.
Backport of #47625 to stable/1.105.x.
Cherry-picked from 6564a490f4 (main).
Adapted to the line: the router budget limiter keeps the line's clock and window TTL (budget_window_seconds from #45797 is not on the line) and its websocket skip, the charge-once helpers move to batch_utils from where the line already defines them, the write-started check drops apply_search_free_tier (#47494, not on the line) and its test, and tests use the line's _ProxyDBLogger name with only this PR's tests appended.
Backport of #47190 to stable/1.105.x.
Cherry-picked from 0ede0e10e7 (main).
Adapted to the line: the created_at/updated_at protocol fields go on the line's _ManagedObjectRow (managed_object_attribution.py is not on the line), the TestBatchLineItemsFromThePoller tests for #41691 are dropped, and the integration test change is dropped since its helpers (prisma_clients, write_rows) are not on the line.
Backport of #47182 to stable/1.105.x.
Cherry-picked from 1212260ab1 (main).
Adapted to the line: calls the line's _select_model_name_for_cost_calc (public on main) and the test uses the line's _response_cost_calculator and a local payload import; the test is appended since its main-side neighbours are not on the line.
Prerequisite for #43935 on stable/1.105.x. Ports the supports_none_reasoning_effort=false flags from #46377 onto the azure and azure_ai gpt-6.1-sol rows this line has, and the minimal, max and xhigh flags azure/eu/gpt-6-astra carries on main (#45018), so the chat and Responses gates #43935 adds drop the levels Azure answers with 400.
Backport of #43935 to stable/1.105.x.
Cherry-picked from d7620c77a1 (main).
Adapted to the line: the Responses bridge change lands in litellm/main.py (the bridge check moved to litellm/responses/bridge_check.py later on main), the line's _get_effort_level and _supports_reasoning_effort_level names are kept, and the test files keep the line's helpers with only this PR's hunks applied.
Backport of #47184 to stable/1.105.x.
Cherry-picked from 3fb95c8fe3 (main).
Adapted to the line: keeps the line's _ManagedObjectRow name and adds its status property, ports postgres_storable_json into managed_files for the claim write (the rest of managed_files keeps the line's code), and drops the S3 batch line-item call that #41691 added on main. Tests converted from MagicMock responses to LiteLLMBatch where the claim now copies the response.
Backport of #46102 to stable/1.105.x.
Cherry-picked from 5b042c0db6 (main).
Adapted to the line: only the anthropic Messages transform is ported (shared by azure_ai, whose beta-map cell supports the beta here), as a minimal change since the line has no BetaProvider mapping; the claude_platform, deepseek and vertex Rust transforms and the Rust integration tests are not on the line and are dropped. Not compiled locally.
Backport of #43955 to stable/1.105.x.
Cherry-picked from eb5295b20c (main).
Handler import conflict resolved by keeping the line's imports plus aload_beta_headers_config; the CLI-beta wire assertion change is dropped since that test is not on the line.
Backport of #45686 to stable/1.105.x.
Cherry-picked from 1aee1a21ed (main).
Test-file import conflict resolved by keeping the line's imports plus this PR's own.
Backport of #45659 to stable/1.105.x.
Cherry-picked from fa7ad80a62 (main).
Test-file conflict resolved by keeping the line's file plus this PR's own additions.
Backport of #45151 to stable/1.105.x.
Cherry-picked from 46d440ae96 (main).
Conflicts resolved by keeping the line's fields and adding only the over-100k fields this PR introduces; cost-map entries replaced per model.
Backport of #45108 to stable/1.105.x.
Cherry-picked from 5cd1126870 (main).
Prerequisite for #45151: adds the claude-haiku-5-5 entries that fix corrects. Cost-map conflicts resolved by appending the new entries.
The #46342 backport carried block_binding through every subclass of the
Anthropic and Bedrock Invoke configs, so Bedrock Mantle, Claude Platform and
OpenAI-compatible Anthropic endpoints started sending it. Each of the six chat
and messages configs now opts in by exact class, and the Bedrock Invoke
display updates and clear_thinking rebuilds carry block_binding only when the
config opts in. Every other config sends the same thinking as 1.105.1
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit e979bd0ecf)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts ad9e861ae7. Anthropic, Bedrock Invoke and Vertex all accept a
forced tool_choice next to thinking.type=adaptive and only reject it with
type=enabled, so counting adaptive as enabled fixed no live failure. It did
drop the forced JSON tool, so Claude answered response_format requests in
prose, and it downgraded a required tool_choice to auto on Bedrock Converse
* test(e2e): drop the adaptive thinking tool round trip case
With adaptive thinking the model often skips thinking on the weather prompt, so
turn 1 carries no signed empty block and the case fails before the replay it is
meant to cover. Anthropic also answers the replay with a 200 whether or not the
block is forwarded, so the case could not tell the fix from main. The factory
unit tests pin the outgoing shape instead
* fix(anthropic): only skip the thinking-strip retry for a binding mismatch 400
prefix_mismatch_behavior: error asks Anthropic to reject a block bound to
another conversation. It says nothing about unrelated bad blocks such as a
null signature, so those still get the strip-and-retry self-heal. Also trims
the empty-thinking docstrings to what the code guarantees and types the new
test parameters
* refactor(anthropic): narrow thinking fields with the mapping validator instead of typing.cast
typing.cast is banned by the strict ruff gate (TID251), so the three new casts pushed that rule one over its merge-base count. The existing pydantic mapping validator gives the same typed view without an unchecked cast, and it now has a generic name since it is no longer only used for Claude Code payloads
* refactor(anthropic): read thinking settings through the validated mapping so basedpyright stays at its merge-base count
The Bedrock clear_thinking rewrite still narrowed with a cast and an untyped dict fallback, which added two reportUnknownVariableType errors. It now uses the same pydantic mapping validator, made public for that reuse, and the retry hook validates the request body itself so the handler passes a typed dict instead of an untyped one
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
(cherry picked from commit f470e418ea)
The request-side sanitizer on /v1/messages dropped every thinking block whose
text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1
return exactly that shape by default (thinking.display omitted): the signature
is the preserved reasoning. Replaying the assistant turn as returned therefore
lost the model's reasoning on every multi-turn request, silently (the provider
answers 200 without it), and left preserved-thinking enforcement
(thinking.block_binding) nothing to bind.
Anthropic accepts an empty thinking block that carries a signature and binds it
to its prefix; only an empty block with no signature (the LIT-6357 case, a turn
a non-Anthropic model served through the bridge) is rejected with "each
thinking block must contain thinking". Use the signature-aware predicate
is_empty_unsigned_thinking_block, which #38809 already applied on the emit
side, for the request side too. The unsigned case is still dropped.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 897bfea38d)
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4b7b8efd4c)
* revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295)
This reverts commit ca1994e403.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): merge bridged tool calls into the same choice as the text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5724117116)
* fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): streamed alias matching a capability rule bills the deployment price
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert every streamed chunk carries the client alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): log the client alias on the priced streamed response, the same as non-streamed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 7b432d78d2)
* fix(proxy): stop queued registry read-throughs spending the resync budget
RegistryReadThrough.attempt serializes misses behind one lock, but every
request that was queued behind the first one still spent a unit of the
20-per-5s resync budget and re-ran the DB resync, even though the first
request had already loaded the object. A burst of more than 20 requests
for a model created on another worker therefore exhausted the budget and
the rest got 400 "Invalid model name".
attempt now checks whether the key is already loaded once it holds the
lock and returns early without touching the budget. Models check the
router's model names and deployment ids, guardrails and agents reuse
their existing registry lookups.
* test(proxy): gate the queued read-through test on events and cover each registry's loaded check
The queued-requests test now holds the first resync on an asyncio.Event instead of
a timed sleep and records calls in a recorder with tuple and frozenset state. New
tests show the model, guardrail and agent read-throughs each answer an object that
is already loaded without reading the database, so rewiring any registry's loaded
check now fails a test
* test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture
(cherry picked from commit b877a38e5f)
