* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout
A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream.
* test(timeout): cover stream_timeout on live streams only and the router's timeout split
* fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management
* fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface
A faked stream now takes the longer of timeout and stream_timeout, so a caller
who raised stream_timeout to cover a slow fake stream keeps that headroom.
Azure o-series dispatch is mirrored when deciding whether a stream is faked.
text_completion, /v1/messages and /v1/responses through the Router now get
timeout and stream_timeout separately, like chat calls.
* fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages
* fix(router): report and log the timeout each stream was bounded by
Native /v1/messages and /v1/responses now record the timeout they applied, so
x-litellm-timeout and the cooldown guard read the real limit instead of the raw
request value. A prompt-managed model also passes on only the caller's or its
own timeout, so the target deployment's timeout still wins.
* fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit
* fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses
* fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit
* test(integration): cover faked, live and converted stream timeout limits per endpoint
* test(router): regression tests for faked-stream timeouts
* fix(bedrock): propagate timeout to streaming requests
* test(bedrock): prove streaming fails at the request timeout against a slow upstream
* test(bedrock): simulate the slow upstream in process instead of over a local socket
* test(bedrock): audit the Converse and Invoke stream timeout on the proxy
Two integration files drive the per-request timeout on Bedrock streams
through the real proxy against an owned wire peer: the wire file covers
every surface (chat, messages, responses, invoke, pass-through), the
sad, edge and precedence rows, and the chaos file covers bursts, a
dropping upstream, a killed worker and a proxy stopped mid-burst.
The wire peer gains Reply.drop_connection so a cell can close the
socket before any response, and the harness's graceful stop grace is
now INTEGRATION_PROXY_STOP_SECONDS (default unchanged at 30), since a
two-worker supervisor's interpreter finalization takes longer than that
on a loaded box.
* test(bedrock): pin the fallback audit cell to one proxy worker
The fallback cell created both deployments through /model/new on one
worker and sent the chat request to the other, whose registry
read-through loads only the requested model, so the fallback target
was unknown there until the periodic DB poll. The cell now warms the
fallback model and sends the request over one keep-alive client, so
one TCP connection stays with one uvicorn worker, and it expects the
fallback upstream to see both requests.
---------
Co-authored-by: Sainyam Kapoor <hello@sainyam.me>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
