Commit graph

228 commits

Author SHA1 Message Date
Yuneng Jiang
2b2e960848
Merge stable/1.104.x (v1.104.6 prep) into patch_d351e54e
Keeps stable/1.104.x's 1.104.6 version in pyproject.toml and uv.lock, so this branch no longer bumps the version. In litellm_logging.py both async success helpers stay, the base's enqueued flag and the claim that #44508 adds, matching main. The two test files keep both sides' new tests
2026-10-20 23:54:09 -07:00
Yuneng Jiang
bd382afc17
Merge stable/1.104.x (#47718) into litellm_backport_p0_batch_translation_cost_stable_1_104_x
# Conflicts:
#	litellm/model_prices_and_context_window_backup.json
#	litellm/proxy/hooks/proxy_track_cost_callback.py
#	litellm/responses/litellm_completion_transformation/streaming_iterator.py
#	model_prices_and_context_window.json
#	tests/test_litellm/proxy/hooks/test_proxy_track_cost_callback.py
#	tests/unit/test_utils.py
2026-10-20 23:07:37 -07:00
Yuneng Jiang
b465d97ead
Merge stable/1.104.x (v1.104.5) into litellm_backport_lit145_stable_1_104_x
# Conflicts:
#	uv.lock
2026-10-20 22:42:07 -07:00
Mrinal Chanshetty
c4b5c45196 Merge branch 'stable/1.104.x' into litellm_backport_p0_batch_translation_cost_stable_1_104_x 2026-10-20 18:40:29 -07:00
Mrinal Chanshetty
9c28744ad5 test: retarget cost-map tests to the rows mirrored from main 2026-10-20 18:24:13 -07:00
Kerry Lu
2313d343ba fix(logging): spilled over PTU requests without base_model log the served model's rates (#47655)
* fix(logging): report pay-as-you-go rates for spilled over PTU requests without base_model

* fix(logging): keep the provider response model so spilled over PTU payloads price the served model

* fix(logging): keep a provider_response_model the response already carries

* fix(logging): read the carried provider_response_model through a typed adapter

---------

Backport of #47655 to stable/1.104.x
Cherry-picked from 6b1f4ed46f (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
5a34391482 fix(proxy): price earlier timed-out retry attempts at their own deployment's model rate (#47648)
* fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate

* fix(router): read the retry record's base_model through the typed metadata helper

* fix(router): keep the four-record retry breadcrumb cap

* fix(router): carry a deployment's litellm_params base_model on its retry record

* chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry

---------

Backport of #47648 to stable/1.104.x
Cherry-picked from 27efc387f7 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
a8807fb0bb fix(router): price cost-based routing through model_info.base_model (#47617)
* fix(router): price cost-based routing through model_info.base_model

* fix(router): keep cost-based routing pricing helpers fully typed

* fix(router): drop typing.cast from cost-based routing pricing helpers

* fix(router): read the cost map through a typed accessor in cost-based routing

* fix(router): avoid a Final rebind inside the cost-based routing loop

* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing

* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing

* refactor(router): inline the deployment pricing-model wrapper in cost-based routing

* refactor(router): read deployment pricing models through one shared helper

deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names

* refactor(router): share the str-keyed mapping validator with cost-based routing

* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers

---------

Backport of #47617 to stable/1.104.x
Cherry-picked from 0c6731eb5c (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
b2ed8bc1c3 fix(proxy): scope the search free-tier counter to the request's provider (#47646)
* fix(proxy): scope the search free-tier counter to the request's provider

* fix(proxy): read the provider with get so partial spend payloads still get the free tier

* test(proxy): give the grounded spend payload fixture its provider

---------

Backport of #47646 to stable/1.104.x
Cherry-picked from 640f2d865b (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
13779e98bd fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata (#47639)
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata

* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType

---------

Backport of #47639 to stable/1.104.x
Cherry-picked from 2a268951c4 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
d388cb8253 fix(proxy): bill partial Responses API streams when the client disconnects (#47611)
* fix(proxy): bill partial Responses API streams when the client disconnects

Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.

* fix(proxy): satisfy lint gates for partial Responses API disconnect billing

Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.

* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect

The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.

* Potential fix for pull request finding 'CodeQL / Log Injection'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Backport of #47611 to stable/1.104.x
Cherry-picked from f0ae445cc4 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
de1e76bd13 feat(proxy): opt-in free-tier allowance for Google Search grounding spend (#47494)
* feat(proxy): opt-in free-tier allowance for Google Search grounding spend

* fix(proxy): declare the search free tier counter's service target

* test(cost_map): allow search_free_requests in the cost map schema

* fix(proxy): skip cache hits and never free more than the billed cost in the search free tier

---------

Backport of #47494 to stable/1.104.x
Cherry-picked from e63aec029a (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
e2af1d25fe fix(decisions): log non-UTF-8 bytes as base64 so the request reaches validation and gets a 400 (#47564)
Backport of #47564 to stable/1.104.x
Cherry-picked from ce63f48ed2 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
bd0b38b4aa fix(litellm_proxy): forward tool_config cache_control_injection_points to the downstream proxy (#47570)
PR #47394 withheld cache_control_injection_points from the OpenAI SDK extra_body for
every provider on that path, which also covers litellm_proxy. The downstream LiteLLM
proxy applies the point itself, so litellm_proxy keeps forwarding it while OpenAI and the
other OpenAI-compatible providers still never receive it

Backport of #47570 to stable/1.104.x
Cherry-picked from 04e9ed4c52 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
b462563d14 fix(logging): report the pay-as-you-go entry billing used on Azure PTU spillover (#47567)
get_standard_logging_object_payload decided custom_pricing from
litellm_params alone, so on an Azure PTU spillover response
model_map_information kept the deployment's zeroed PTU entry while
response_cost already used the standard model rate. Reuse the billing
side LitellmLogging._custom_pricing_for so both read the same decision

Backport of #47567 to stable/1.104.x
Cherry-picked from 4222523261 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:14:11 +00:00
Kerry Lu
f585b558e9 fix(batches): apply the Vertex regional endpoint uplift to batch cost (#47466)
* fix(batches): apply the Vertex regional endpoint uplift to batch cost

batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down

Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>

* test(logging): point the resolver tests at resolve_vertex_location_for_cost

---------

Backport of #47466 to stable/1.104.x
Cherry-picked from c6c303a618 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
2026-10-21 01:14:10 +00:00
joshua-berri
712dc21192 fix(batches): retry confirmed pre-accounting failures (#47625)
Backport of #47625 to stable/1.104.x.
Cherry-picked from 6564a490f4 (main).
Ported by hand onto the line's spend and budget code: the charge-once helpers move from model_max_budget_limiter into router_utils.batch_utils as on main; RouterBudgetLimiting keeps the line's websocket skip and uses the line's duration_in_seconds window for the batch marker TTL; the write_started mark covers budget reservations only, since the search free-tier allowance (#47494) is not on this line. Tests adapted to the line's _ProxyDBLogger name; the search-allowance test is dropped for the same reason.
2026-10-20 17:21:34 -07:00
daqiangganjun
ec25d80617 fix(router): count provider budget spend on every API surface (#38172)
Backport of #38172 to stable/1.104.x.
Cherry-picked from 8327cd6d47 (main).
Prerequisite for #47625: RouterBudgetLimiting reads the provider from the logging payload, so the batch poller's aretrieve_batch events reach router budgets at all. Test file placed at tests/unit/router_strategy/ on this line.
2026-10-20 17:21:34 -07:00
joshua-berri
f5049dba86 fix(bedrock): resolve profile pricing before applying custom rates (#47419)
Backport of #47419 to stable/1.104.x.
Cherry-picked from 3124797399 (main).
Ported onto the line's register_model and router startup path without #41523's aliased-entry rewrite: registered fields come from the side table added in the previous commit. Tests adapted to the line: private names (_replay_model_cost_registrations, _response_cost_calculator), the custom-pricing test helpers from main's test_cost_calculator.py, CONVERSE_ANSWER inlined, and the alias-propagation and bedrock gpt-6.1-sol/ARN cases removed since those depend on #41523 and #46128, which are not on this line.
2026-10-20 17:21:34 -07:00
joshua-berri
21a84f9e8f fix(batch): preserve recoverable costs and prevent poll starvation (#47190)
Backport of #47190 to stable/1.104.x.
Cherry-picked from 0ede0e10e7 (main).
Ported by hand onto the line's poller (the ManagedObjectRow protocol is the line's local _ManagedObjectRow; _finalize_unbilled_terminal_job keeps the line's None return). Test conflicts take this PR's assertions; the line-item poller tests from main are not on this line.
2026-10-20 17:21:34 -07:00
Kerry Lu
96684a2570 fix(vertex_ai): bill per-character text embeddings on Google's billableCharacterCount (#47460)
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.

Backport of #47460 to stable/1.104.x
Cherry-picked from d29200fa05 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 00:05:41 +00:00
Kerry Lu
744f63b8db fix(logging): do not stamp a provider handoff when pre_call runs for a proxy-only failure (#47456)
Backport of #47456 to stable/1.104.x
Cherry-picked from 3fa1587f43 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 00:04:36 +00:00
Kerry Lu
811cecdc3b fix(anthropic): apply tool_config cache-control points on chat and /v1/messages (#47394)
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages

A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.

Co-authored-by: Tan Nguyen <tan@pressingly.net>

* refactor(anthropic): validate tool_config inputs instead of casting them

Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.

* fix(openai): keep tool_config cache points out of extra_body, add integration test

The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.

* test(integration): cover tool_config cache points on bridged /v1/messages

Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig

---------

Backport of #47394 to stable/1.104.x
Cherry-picked from 93b79d2587 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
2026-10-21 00:01:31 +00:00
Kerry Lu
f0fb999131 fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists (#47373)
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists

Co-authored-by: shivam <shivam@berri.ai>

* style(cost): ruff format the modality cost calls

---------

Backport of #47373 to stable/1.104.x
Cherry-picked from 09e9193514 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
2026-10-20 23:57:26 +00:00
Kerry Lu
d64ac3998c feat(router): let a deployment set a timeout per service_tier (#47406)
* feat(router): let a deployment set a timeout per service_tier

Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged

Co-authored-by: adc77 <amandogra2016@gmail.com>

* style(router): sort the request_timeout_resolver import

* fix(router): read the service tier maps without the untyped params get

* fix(router): look the service tier timeout up through a typed get

---------

Backport of #47406 to stable/1.104.x
Cherry-picked from 62ce515d15 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
2026-10-20 23:56:28 +00:00
Kerry Lu
826f404735 fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions (#47183)
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions

* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built

* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request

* fix(decisions): return the single logged message so no mutable list type is annotated

* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded

* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON

* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too

* test(decisions): assert the exact BadRequestError for the rejected-questions case

---------

Backport of #47183 to stable/1.104.x
Cherry-picked from 53821f6654 (main)

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:54:07 +00:00
Kerry Lu
5684267b12 fix(logging): report the cost-map entry billing used in model_map_information (#47182)
Backport of #47182 to stable/1.104.x.
Cherry-picked from 1212260ab1 (main).
Adapted to the line's private helper names (_select_model_name_for_cost_calc, _response_cost_calculator); the new test is appended to the line's test file.
2026-10-20 16:12:01 -07:00
Paul El Khoury
54f9259b40 fix(openai): drop unsupported reasoning effort for gpt-6.1-sol (#43935)
Backport of #43935 to stable/1.104.x.
Cherry-picked from d7620c77a1 (main).
Adapted to the line: the bridge check lives in litellm/main.py rather than litellm/responses/bridge_check.py, the effort helpers keep the line's private names, and the azure gpt-6.1-sol rows get supports_none_reasoning_effort=false (from #46377 on main) since this fix and its tests key on it. Test files keep the line's content plus this PR's own hunks.
2026-10-20 16:12:01 -07:00
joshua-berri
f7b0b09e75 fix(batches): preserve spend across malformed output and metadata failures (#47184)
Backport of #47184 to stable/1.104.x.
Cherry-picked from 3fb95c8fe3 (main).
Ported by hand onto the line's poller, whose surrounding code predates several main-only batch features: the claim now writes the terminal status and NUL-stripped file_object, a failed spend-log write restores both, and the managed-files postgres_storable_json sanitizer (from #45703 on main) is added to the line since this fix needs it.
2026-10-20 16:12:01 -07:00
joshua-berri
bb1e40c3e5 fix(batches): handle empty models when logging batch cost (#47155)
Backport of #47155 to stable/1.104.x.
Cherry-picked from 4143e0869e (main).
2026-10-20 16:12:01 -07:00
devin-ai-integration[bot]
2ac8b35a05 fix(model_prices): consolidate claude-haiku-5-5 over-100k pricing and capability flags (#45151)
Backport of #45151 to stable/1.104.x.
Cherry-picked from 46d440ae96 (main).
Conflicts resolved by keeping the line's fields and adding only the over-100k fields this PR introduces; cost-map entries replaced per model.
2026-10-20 16:12:00 -07:00
Krrish Dholakia
d3366b93b4 feat(model_prices): add claude-haiku-5-5 model pricing (#45108)
Backport of #45108 to stable/1.104.x.
Cherry-picked from 5cd1126870 (main).
Prerequisite for #45151: adds the claude-haiku-5-5 entries that fix corrects. Cost-map conflicts resolved by appending the new entries.
2026-10-20 16:12:00 -07:00
devin-ai-integration[bot]
61c5d953ee fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards (#43955)
Backport of #43955 to stable/1.104.x.
Cherry-picked from eb5295b20c (main).
Adapted to the line: the transformation lives under anthropic/experimental_pass_through, the integration tests live under tests/integration/providers, and the CLI-beta wire assertion change is dropped since that test is not on the line.
2026-10-20 16:12:00 -07:00
mrinal-berri
2f19d8ed87 fix(databricks): send a single plain text block as a string for non-Claude models (#45686)
Backport of #45686 to stable/1.104.x.
Cherry-picked from 1aee1a21ed (main).
Test-file import conflict resolved by keeping the line's imports plus this PR's own.
2026-10-20 15:32:54 -07:00
mrinal-berri
6ba8151a2d fix(databricks): keep #/$defs refs in json_schema response_format (#45659)
Backport of #45659 to stable/1.104.x.
Cherry-picked from fa7ad80a62 (main).
Test-file conflict resolved by keeping the line's file plus this PR's own additions.
2026-10-20 15:32:30 -07:00
devin-ai-integration[bot]
cfdfb63a4c fix(responses): report truncated bridged output as incomplete and echo request params (#44460)
Backport of #44460 to stable/1.104.x.
Cherry-picked from f81a3f5243 (main).
2026-10-20 15:30:01 -07:00
mrinal-berri
ef3c50f2d9 fix(count_tokens): fall back to Mantle on Bedrock 404 (Bedrock part of #46430)
Backport of #46430 to stable/1.104.x.
Cherry-picked from 08a24d391a (main), Bedrock changes only.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 17:46:21 +00:00
devin-ai-integration[bot]
f9a1d5ea7a fix(bedrock): count tokens on bedrock-mantle when bedrock-runtime cannot count a Claude model (#45317)
Backport of #45317 to stable/1.104.x.
Cherry-picked from 577d74c1ce (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 17:45:51 +00:00
devin-ai-integration[bot]
eb1d2fb4ee
fix(responses): fold bridged Responses output into one chat choice (#46099)
* fix(responses): fold bridged Responses output into one chat choice

A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): drop mutable annotations and the extra cast in the choice fold

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): remove a cast-ok marker that suppressed nothing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): join folded message texts with a newline

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* revert(responses): join folded message texts with no separator so stream and non-stream match

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4b7b8efd4c)
2026-10-19 23:36:47 -07:00
Hektor Jacynycz Garcia
87ddb03ef3
fix(logging): log spend once for large non-streaming requests on the chat to Responses bridge (#44508)
* fix(logging): claim the async success dedup flag before the first await

Nested @client wrappers on the chat -> Responses bridge schedule two async
success tasks on one logging object. The dedup flag was checked at the top
of _async_success_handler_body but only set after several awaits; the
worker-thread base64 offload for messages >= 256 KiB yields there, so the
second task passed the same check and the request was logged twice
(two SpendLogs rows, key/team/daily spend counted twice).

Fixes #44500

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(logging): serialise async success handlers per logging object instead of claiming early

Claiming the dedup flag before the first await (previous commit) let a first
handler that is cancelled before its callbacks run (e.g. by the logging
worker's per-coroutine timeout) suppress the second one, losing the request's
success accounting. Serialise non-streaming handlers on the same Logging
object with an asyncio.Lock kept in a module-level WeakKeyDictionary: the
second handler waits, then skips if the first logged, or logs itself if the
first raised or was cancelled first. Streaming chunk logging is not locked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(integration): pin one spend row and one key charge for large bridged chat completions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): wait for the large request's spend row before asserting

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(logging): use fixed timestamps in async success dedup tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(logging): trim the async success dedup lock comment

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(logging): reset background interaction dedupe under the success dedup lock

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(logging): use a fixed start time in the interactions logging helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: claim the async success log at schedule time instead of locking the handler

Nested @client wrappers (chat over the Responses bridge, Anthropic Messages over the chat adapter) exit with one shared Logging object and each queued their own async success handler. The handler checked has_logged_async_success before its awaited work and set it after, so both could pass the check and the request was logged and billed twice.

_schedule_async_success_logging now claims the log on the Logging object synchronously through claim_async_success_log, and a later wrapper returns without enqueuing. There is no await between the check and the decision, so no lock is needed. The per-object asyncio.Lock, the WeakKeyDictionary that held it and the locked block in async_log_background_interaction_completion are gone.

Tests: the regression test moves to tests/unit/test_utils.py and drives two _dispatch_success_logging exits with one Logging object through the logging worker, asserting one log with the inner result. It fails without the claim. The cancelled-handler test is dropped since a second wrapper no longer retries. The background interaction completion test stays.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(logging): pin the background completion event order and name the innermost wrapper in the claim docstrings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): send one bridged request and assert the inner Responses result is the logged one

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 157d96dcd0)
2026-10-19 23:36:47 -07:00
devin-ai-integration[bot]
f5d123c495
fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model (#44341)
* fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(spend): streamed alias matching a capability rule bills the deployment price

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(spend): assert every streamed chunk carries the client alias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(logging): log the client alias on the priced streamed response, the same as non-streamed

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 7b432d78d2)
2026-10-19 23:36:47 -07:00
devin-ai-integration[bot]
8860121434
fix(anthropic): forward the dangerous-tool-use beta to Azure AI Foundry (#43934)
Map dangerous-tool-use-2026-09-03 for azure_ai in the beta header config so the
Claude Code auto mode beta reaches Foundry instead of being stripped, matching
the anthropic, bedrock, and vertex_ai entries

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit ed4caebb65)
2026-10-19 22:46:46 -07:00
devin-ai-integration[bot]
a2816d7eb7
fix(router): stream /v1/messages lifecycle frames live when no fallback can take over (#43600)
* fix(router): stream anthropic messages lifecycle frames live when no fallback can take over

The /v1/messages streaming wrapper buffered message_start and
content_block_start until the first content_block_delta and dropped
pings behind buffered frames unconditionally, even for requests no
fallback could ever recover. With adaptive thinking on Bedrock or
Vertex the client saw no bytes for the whole thinking pass and hit
read timeouts.

Buffering now applies only while a fallback can still take over
(generic or refusal chain resolving), and a ping is always forwarded
live since it carries no lifecycle and keeps the connection alive.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): mirror every dispatcher fallback path in the anthropic stream gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): skip already-tried order levels in the anthropic stream gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): keep a transport-split ping behind buffered lifecycle frames instead of forwarding its head live

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(router): credit the #39566 branch this fix supersedes

Co-authored-by: Radu Swigler <radu.porumba@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Radu Swigler <radu.porumba@gmail.com>
(cherry picked from commit d46304900f)
2026-10-19 22:40:20 -07:00
yuneng-jiang
79213e254b
fix(caching): stand default cache points down when extra_body hides a direct client mark (#43341)
* fix(caching): stand default cache points down when extra_body hides a direct client mark

On native /v1/messages the extra_body envelope is dropped, so a client tool mark or root cache_control reaches Anthropic even when extra_body overrides it. The stand-down check only counted the envelope-merged view and injected two default marks on top of the client's.

* fix(caching): keep chat completions on the envelope-merged mark count for the default stand-down

Chat completions merge extra_body over the request, so a direct tool mark that extra_body replaces never reaches the provider there. Only /v1/messages, where the native transforms drop the envelope, needs to count marks on both sides.

(cherry picked from commit b2e82cf3be)
2026-10-19 22:25:41 -07:00
Joshua Valluru
e109f8977d fix(proxy): restore legacy token count input precedence 2026-10-19 17:50:09 -07:00
mateo-berri
5e71466355 fix(count_tokens): normalize Google system parts and null tools 2026-10-19 16:53:38 -07:00
mateo-berri
3f3fb2ac60 fix(count_tokens): omit local PDF parsing from stable backport 2026-10-19 15:20:52 -07:00
mateo-berri
704a15720a test(count_tokens): type PDF budget regression cases 2026-10-19 15:14:43 -07:00
mateo-berri
c881f7cfc1 fix(count_tokens): bound PDF parsing and preserve legacy fallback 2026-10-19 15:10:51 -07:00
mateo-berri
498095ee1b fix(count_tokens): retain legacy translated content fallback 2026-10-19 14:38:30 -07:00