The proxy-endpoints shard failed with KeyError: 'model' because the new Anthropic post-call context translation reached translate_anthropic_to_openai with request data that only carried messages and guardrail metadata.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Response-side guardrail scans on OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses now carry structured_messages (the request turns scoped exactly like the pre-call scan, closed by the model's reply as an assistant turn) and tools (the request's function definitions), in addition to texts, images, and tool_calls.
Guardrails that used structured_messages or tools as a response-side signal (akto, crowdstrike_aidr, hiddenlayer, openai moderations, promptguard, qualifire, straiker) keep their previous response payloads. Logging-only scans whose output translation differs from the input translation get a chat-shaped request so the context survives.
Resolves LIT-6628
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The session usage collapsed duplicate query strings across turns while the
price was per turn, so two turns asking the same question paid two fees yet
reported web_search_requests 1. Sum each turn's grounding requests so the
counter matches the bill; duplicates within one turn still collapse.
A converted-stream request whose cache entry is a plain (non-stream) object is
replayed as that plain object, so nothing later fires the success callbacks.
Decide deferral from the replayed result's type instead of the request kwargs.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A deployment hook (Headroom, code interpreter, web search) can downgrade
kwargs["stream"] to False while the caller still expects to iterate the
result. The cache handler keyed stream replay and callback deferral off
the raw flag, so a cache hit returned a plain object to a caller that
iterates, and the Responses iterator never persisted the converted
stream in the first place. Key both off the conversion marker as well
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Deployment hooks such as Headroom downgrade stream=True to a non-streaming provider call and the agentic loop then hands back a CustomStreamWrapper (or MockResponsesAPIStreamingIterator for Responses). wrapper_async still saw kwargs["stream"] is False, so it took the non-streaming success path with a lazy stream object: no standard_logging_object was built, the proxy cost callback raised failed_tracking_spend, and the wrapper's own end-of-stream dispatch was deduped away. Treat a lazy stream result as streaming for logging regardless of the downgraded kwarg. Regression in v1.99.0 via #35017
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A guardrail that rewrites text and hands back tool_use arguments that are
not a JSON object used to leave the text rewrite applied when the request
was rejected, so failure logging saw a half-rewritten request. Every
rejection now happens before any write to system or messages.
The router clamps a negative request_retry_count found in request metadata before counting a failure, and the proxy strips a client-supplied request_retry_count with the other router-reserved metadata fields. The rust OCR lifecycle test that trips the per-request cap now plants request_retry_count instead of attempted_retries, which the cap no longer reads since the previous commit
Add 72 model price entries for the aihubmix openai_like provider so
cost tracking and budgets work for aihubmix/* model calls. The
provider is already registered in llms/openai_like/providers.json
but model_prices_and_context_window.json had zero entries for it.
The Anthropic-family entries (claude-fable-5, claude-haiku-4-5,
claude-opus-4-8, claude-opus-5, claude-sonnet-5) carry the same
supports_adaptive_thinking, thinking_always_on,
supports_sampling_params, and prompt_cache_min_tokens flags already
used by this repo's other Anthropic re-exports (azure_ai, databricks,
openrouter, and so on) for the same underlying models, since those
flags gate request shapes the provider otherwise rejects with a 400.
TASK-2BK38Y
num_retries_per_request has always capped the retries of one request with its fallback hops included. #40930 started reading the per-hop attempted_retries counter instead, and every fallback hop restarts that counter at zero, so a request could spend a fresh retry budget on each hop and the legacy fallback cap test started seeing the hop run.
Router.log_retry now also keeps request_retry_count on the request metadata, incremented on every retry and fallback hop and never truncated the way previous_models is, and max_retries_per_request_hit reads that count. The flat retry records, the litellm_metadata coverage and caps above four from #40930 stay as they are, and the legacy test goes back to its previous_models == 0 assertion.
