Realtime cost is computed from *_tokens_details after the usage round-trips
through the Responses shape, and the input half of that shape carried audio
only, so image and video prompt tokens stopped being billable as themselves.
Vertex splits prompt tokens by modality, so a session sending camera frames
arrives with image_tokens set. Those were folded into text_tokens and lost
their attribution. The amount happens not to move today, because the
calculator falls back to input_cost_per_token when no per-modality rate is
set, but the tokens have to survive before any such rate can ever apply.
InputTokensDetails now declares image_tokens and video_tokens instead of
leaning on pydantic extras, the repeated per-field copying is a loop over the
modality names so adding a modality no longer adds a branch, and the read-back
in ResponseAPILoggingUtils picks up video_tokens, which
PromptTokensDetailsWrapper already declared.
The output half of the original change is dropped: 449c091391 landed the same
OutputTokensDetails.audio_tokens fix upstream, with its own coverage in
test_gemini_realtime_transformation.py, and it always sets
output_tokens_details rather than only when non-empty. That structure is kept
as upstream wrote it.
The proxy-endpoints shard failed with KeyError: 'model' because the new Anthropic post-call context translation reached translate_anthropic_to_openai with request data that only carried messages and guardrail metadata.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Response-side guardrail scans on OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses now carry structured_messages (the request turns scoped exactly like the pre-call scan, closed by the model's reply as an assistant turn) and tools (the request's function definitions), in addition to texts, images, and tool_calls.
Guardrails that used structured_messages or tools as a response-side signal (akto, crowdstrike_aidr, hiddenlayer, openai moderations, promptguard, qualifire, straiker) keep their previous response payloads. Logging-only scans whose output translation differs from the input translation get a chat-shaped request so the context survives.
Resolves LIT-6628
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The session usage collapsed duplicate query strings across turns while the
price was per turn, so two turns asking the same question paid two fees yet
reported web_search_requests 1. Sum each turn's grounding requests so the
counter matches the bill; duplicates within one turn still collapse.
A converted-stream request whose cache entry is a plain (non-stream) object is
replayed as that plain object, so nothing later fires the success callbacks.
Decide deferral from the replayed result's type instead of the request kwargs.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A deployment hook (Headroom, code interpreter, web search) can downgrade
kwargs["stream"] to False while the caller still expects to iterate the
result. The cache handler keyed stream replay and callback deferral off
the raw flag, so a cache hit returned a plain object to a caller that
iterates, and the Responses iterator never persisted the converted
stream in the first place. Key both off the conversion marker as well
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Deployment hooks such as Headroom downgrade stream=True to a non-streaming provider call and the agentic loop then hands back a CustomStreamWrapper (or MockResponsesAPIStreamingIterator for Responses). wrapper_async still saw kwargs["stream"] is False, so it took the non-streaming success path with a lazy stream object: no standard_logging_object was built, the proxy cost callback raised failed_tracking_spend, and the wrapper's own end-of-stream dispatch was deduped away. Treat a lazy stream result as streaming for logging regardless of the downgraded kwarg. Regression in v1.99.0 via #35017
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Anthropic /v1/messages and Bedrock /converse resolve their upstream timeout
through resolve_llm_passthrough_timeout, which only reads timeout /
request_timeout and then falls back to the 600s pass_through default. A
stream_timeout set on the deployment or in router_settings was never
consulted on that route, while /chat/completions honors it through
Router._get_stream_timeout.
For a streaming call the resolver now checks stream_timeout at each level
before the non-stream key (kwargs -> litellm_params -> router), mirroring
_get_stream_timeout; non-streaming resolution is unchanged. The router
passes its stream_timeout alongside the explicit timeout.
Anthropic's preserved-thinking controls (`thinking.block_binding`, Claude
Fable 5.1) are only accepted alongside the beta header
`thinking-binding-controls-2026-08-01`. The proxy forwards the body field
untouched but `filter_and_transform_beta_headers` drops the header because
it has no entry in `anthropic_beta_headers_config.json`, so Bedrock and
Vertex reject the request with
"thinking.adaptive.block_binding: Extra inputs are not permitted".
Map the header for anthropic, bedrock, bedrock_converse, vertex_ai and
databricks (same beta name on all of them per Anthropic's docs). azure_ai is
left null pending verification on Foundry.
A guardrail that rewrites text and hands back tool_use arguments that are
not a JSON object used to leave the text rewrite applied when the request
was rejected, so failure logging saw a half-rewritten request. Every
rejection now happens before any write to system or messages.
The router clamps a negative request_retry_count found in request metadata before counting a failure, and the proxy strips a client-supplied request_retry_count with the other router-reserved metadata fields. The rust OCR lifecycle test that trips the per-request cap now plants request_retry_count instead of attempted_retries, which the cap no longer reads since the previous commit
num_retries_per_request has always capped the retries of one request with its fallback hops included. #40930 started reading the per-hop attempted_retries counter instead, and every fallback hop restarts that counter at zero, so a request could spend a fresh retry budget on each hop and the legacy fallback cap test started seeing the hop run.
Router.log_retry now also keeps request_retry_count on the request metadata, incremented on every retry and fallback hop and never truncated the way previous_models is, and max_retries_per_request_hit reads that count. The flat retry records, the litellm_metadata coverage and caps above four from #40930 stay as they are, and the legacy test goes back to its previous_models == 0 assertion.
httpx.Headers.items() comma-joins repeated header names, so the rebuilt
response iterates multi_items() and keeps every value, matching what the
raw openai client exposes on e.response.headers
The streaming bridge restored the namespace before deciding whether a tool call was a custom tool, so a namespaced function sharing a short name with a nested custom tool streamed back as a custom_tool_call. Classify on the raw chat tool name first, the way the non-streaming path already does.
The guardrail merge only stripped the namespace prefix and grammar suffix from the ends of the edited description, so a guardrail appending text after the grammar block left the block in the member description and the chat conversion appended it a second time. Strip the first occurrence of each instead.
