* fix(batches): apply the Vertex regional endpoint uplift to batch cost
batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* test(logging): point the resolver tests at resolve_vertex_location_for_cost
---------
Backport of #47466 to stable/1.104.x
Cherry-picked from c6c303a618 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.
Backport of #47460 to stable/1.104.x
Cherry-picked from d29200fa05 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47456 to stable/1.104.x
Cherry-picked from 3fa1587f43 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47435 to stable/1.104.x
Cherry-picked from 7b76b2b15f (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Backport of #47394 to stable/1.104.x
Cherry-picked from 93b79d2587 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Backport of #47373 to stable/1.104.x
Cherry-picked from 09e9193514 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Backport of #47406 to stable/1.104.x
Cherry-picked from 62ce515d15 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Backport of #47183 to stable/1.104.x
Cherry-picked from 53821f6654 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Defer the breaking error policy from #46121. Adapt the #46777 count settings flow to attempt a labelled local estimate for provider failures across providers and configured count models.
Keep provider errors when local counting is disabled or cannot estimate the request, and test the shared behavior across HTTP error statuses.
Keep #43900, #45301, #46121, #46784, #46759, #46767, #46772, #46815, #47057 and #46777. Port only the Gemini prerequisites from #46430.
Remove the deferred Bedrock Mantle behavior and unrelated picks from the final tree. Restore the PDF test provider fixture and deployment, and retain required test imports.
Hand-ported to stable/1.104.x from main (bedrock_runtime_peer, provider, _cache_control_marks_support verbatim, graceful_stop_seconds at the line's 30s stop wait).
Backport of #47064 to stable/1.104.x.
Cherry-picked from ae6f7f6613 (main).
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #46430 to stable/1.104.x.
Cherry-picked from 08a24d391a (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #46121 to stable/1.104.x.
Cherry-picked from 719c44db3f (main).
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Backport of #45463 to stable/1.104.x.
Cherry-picked from d34ad35281 (main).
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Backport of #45301 to stable/1.104.x.
Cherry-picked from 50ab9a3211 (main).
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Backport of #43900 to stable/1.104.x.
Cherry-picked from 6d73fa6b49 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Backport of #43417 to stable/1.104.x.
Cherry-picked from 8e6d99d74a (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The tests picked from #47067 annotate module constants with Final, which main's copy imports and this line's did not, so the module failed to import on Python 3.12.
Backport of #47067 to stable/1.104.x.
Cherry-picked from e979bd0ecf (main).
Conflict: main renamed the model access helper to check_model_access_helper, while stable/1.104.x still names it _check_model_access_helper, so the new owning-team check calls it by the line's name.
Line adaptation: the line's test_auth_utils.py had no module-level Router import, which main's copy has and the picked helper uses, so it is added.
