A converted-stream request whose cache entry is a plain (non-stream) object is
replayed as that plain object, so nothing later fires the success callbacks.
Decide deferral from the replayed result's type instead of the request kwargs.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A deployment hook (Headroom, code interpreter, web search) can downgrade
kwargs["stream"] to False while the caller still expects to iterate the
result. The cache handler keyed stream replay and callback deferral off
the raw flag, so a cache hit returned a plain object to a caller that
iterates, and the Responses iterator never persisted the converted
stream in the first place. Key both off the conversion marker as well
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Deployment hooks such as Headroom downgrade stream=True to a non-streaming provider call and the agentic loop then hands back a CustomStreamWrapper (or MockResponsesAPIStreamingIterator for Responses). wrapper_async still saw kwargs["stream"] is False, so it took the non-streaming success path with a lazy stream object: no standard_logging_object was built, the proxy cost callback raised failed_tracking_spend, and the wrapper's own end-of-stream dispatch was deduped away. Treat a lazy stream result as streaming for logging regardless of the downgraded kwarg. Regression in v1.99.0 via #35017
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A guardrail that rewrites text and hands back tool_use arguments that are
not a JSON object used to leave the text rewrite applied when the request
was rejected, so failure logging saw a half-rewritten request. Every
rejection now happens before any write to system or messages.
The router clamps a negative request_retry_count found in request metadata before counting a failure, and the proxy strips a client-supplied request_retry_count with the other router-reserved metadata fields. The rust OCR lifecycle test that trips the per-request cap now plants request_retry_count instead of attempted_retries, which the cap no longer reads since the previous commit
Add 72 model price entries for the aihubmix openai_like provider so
cost tracking and budgets work for aihubmix/* model calls. The
provider is already registered in llms/openai_like/providers.json
but model_prices_and_context_window.json had zero entries for it.
The Anthropic-family entries (claude-fable-5, claude-haiku-4-5,
claude-opus-4-8, claude-opus-5, claude-sonnet-5) carry the same
supports_adaptive_thinking, thinking_always_on,
supports_sampling_params, and prompt_cache_min_tokens flags already
used by this repo's other Anthropic re-exports (azure_ai, databricks,
openrouter, and so on) for the same underlying models, since those
flags gate request shapes the provider otherwise rejects with a 400.
TASK-2BK38Y
num_retries_per_request has always capped the retries of one request with its fallback hops included. #40930 started reading the per-hop attempted_retries counter instead, and every fallback hop restarts that counter at zero, so a request could spend a fresh retry budget on each hop and the legacy fallback cap test started seeing the hop run.
Router.log_retry now also keeps request_retry_count on the request metadata, incremented on every retry and fallback hop and never truncated the way previous_models is, and max_retries_per_request_hit reads that count. The flat retry records, the litellm_metadata coverage and caps above four from #40930 stay as they are, and the legacy test goes back to its previous_models == 0 assertion.
httpx.Headers.items() comma-joins repeated header names, so the rebuilt
response iterates multi_items() and keeps every value, matching what the
raw openai client exposes on e.response.headers
The streaming bridge restored the namespace before deciding whether a tool call was a custom tool, so a namespaced function sharing a short name with a nested custom tool streamed back as a custom_tool_call. Classify on the raw chat tool name first, the way the non-streaming path already does.
The guardrail merge only stripped the namespace prefix and grammar suffix from the ends of the edited description, so a guardrail appending text after the grammar block left the block in the member description and the chat conversion appended it a second time. Strip the first occurrence of each instead.
The proxy edge test file no longer carries any test of this change, and
the remaining diff was the scoped format gate reflowing the whole file to
the 120 limit, so it goes back to the merge base bytes
#40930 (LIT-7505) changed num_retries_per_request from a request-wide
cap to a per-model-group cap that resets on every fallback hop, and its
own comment in litellm/__init__.py names that contract. The legacy
test_async_fallbacks_max_retries_per_request still asserted the old
request-wide reading (previous_models == 0), so the CircleCI router
suite has been red on main since that merge for every run-ci PR.
The test now reads the flat RetryAttemptRecord entries the fallback
call carries and asserts the new contract directly: every record is
from the first group, the retry at attempted_retries 0 is the real
AuthenticationError, and each later attempt was refused with
"Max retries per request hit!".
maturin already packages non-gitignored files under the Python source
directory of a mixed project, and the built wheel contains
litellm/rust_bridge/_native.pyi without the explicit entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The hoisted structured row keeps every text block of the top-level system
prompt, empty ones included, while the scanned texts dropped the empty ones.
Guardrails that count one text per slot then came back with more texts than
the handler could place, so their rewrite was rejected. User text blocks were
already scanned empty or not; the system prompt now matches.
A mapped litellm_proxy exception now attaches an httpx.Response that
carries the proxy's response headers whenever the handler attached a
header-less synthetic one, on every status branch and on the relay
path. BadRequestError keeps its base-class contract: .headers stays the
proxy-supplied channel, so the proxy edge keeps forwarding an upstream
proxy's headers under the llm_provider- prefix and the date and server
edge change is no longer needed.
Only image-only unscoped content stays unrecorded; text or tool content
removed by scoping is recorded as not_run even when an image sits beside it.
Also keeps the type-discipline budget flat by returning the reason from the
helper and annotating the accumulator lists _extract_inputs requires.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
main's test_one_text_per_row_over_a_system_prompt_is_rejected_by_name assumed the
top-level system prompt stays out of the scanned texts. This branch scans it, so one
text per structured row now lines up and the rewrite is applied; the test asserts that,
and a multi-block system prompt keeps the length-guard rejection covered.
The one-hour cache write fallback now takes the applied cache write rate, so an off-peak write price carries into it instead of the input rate
The cost estimate test for a cost-map model without cache prices now expects writes at the input rate, which is what the proxy bills
The recording logger in the deferred guardrail test types its callback parameters
