mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-06 02:48:13 +00:00
1857f5d04b
A model with a long time-to-first-token leaves the proxy's response completely idle, so any hop with an idle read timeout (AWS ALB and nginx both default to 60s) drops a connection that is perfectly healthy and would have delivered its tokens shortly after. The keepalive engines LiteLLM already ships wrap the response object, so they fill a gap once the upstream has answered and then gone quiet. They cannot fill the gap before it answers at all, and that is where the whole wait is spent: measured against api.openai.com/v1/chat/completions with gpt-5.6 at reasoning_effort high, the response headers and the first body byte both arrive at 37.90s. Nothing has entered the ASGI response phase by then. The upstream call is now raced against the keepalive interval, and when it loses, the SSE response is opened immediately and ": ping" comments, which every conformant SSE client ignores, fill the wire until the real response is ready to be replayed onto it. One seam per funnel: base_process_llm_request covers every native route, create_pass_through_route covers every passthrough route. Committing the status line that early is the cost. A failure discovered after the first ping reaches the client as an SSE error frame under a 200 rather than as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not yet known. keepalive_ping_has_fired already documents the same trade-off for the existing engines. Both are why this stays off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds. Separately, the passthrough relay reached neither engine even for mid-stream gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same treatment, gated on the upstream declaring text/event-stream and only emitted between complete frames so a binary transport (AWS event streams on /bedrock) and a stall halfway through a frame are both left alone. Fixes #34819 |
||
|---|---|---|
| .. | ||
| llm_provider_handlers | ||
| test_carry_guardrail_logging_info.py | ||
| test_llm_pass_through_endpoints.py | ||
| test_managed_id_rewriter.py | ||
| test_method_specific_routing.py | ||
| test_pass_through_endpoints.py | ||
| test_passthrough_auth_default.py | ||
| test_passthrough_endpoint_router.py | ||
| test_passthrough_endpoints_common_utils.py | ||
| test_passthrough_guardrail_block_otel_span.py | ||
| test_passthrough_guardrails.py | ||
| test_passthrough_guardrails_field_targeting.py | ||
| test_passthrough_post_call_guardrails.py | ||
| test_streaming_handler_interrupt.py | ||
| test_upstream_usage_headers.py | ||
| test_vertex_ai_batch_passthrough.py | ||
| test_vertex_passthrough_load_balancing.py | ||
| test_watsonx_proxy_route.py | ||