mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-13 08:52:02 +00:00
43112547fb
* feat(proxy): total_timeout bounds a whole non-streaming request A `total_timeout` body field or `x-litellm-total-timeout` header (the header wins) gives a non-streaming /v1/chat/completions, /v1/messages or /v1/responses request one budget for preparation, retries, moderation and post-call processing. Past it the proxy answers 408 and cancels the upstream call, and a retry whose backoff cannot fit ends early with the error it would have retried. A body value on a streaming or background request is refused with a 400, a header value leaves it unbounded. A timed out request is billed like a client disconnect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): ignore any total_timeout header on requests it cannot bound, and never interrupt pre-call admission A globally configured x-litellm-total-timeout that is malformed no longer fails the streaming and background requests it is ignored on. The deadline holds while pre-call admission runs, since admission reserves shared quota before it records it, and an expired deadline then ends the request before dispatch, where the failure path hands the reservation back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): a total_timeout cuts slow pre-call hooks short, holding only the rate limit reservation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| core | ||
| __init__.py | ||
| test_enforce_model_rate_limits.py | ||
| test_io_token_rate_limits.py | ||
| test_router.py | ||
| test_router_callback_hook_sequence.py | ||
| test_router_endpoints.py | ||
| test_router_entry_points.py | ||
| test_router_facade.py | ||
| test_router_fallbacks.py | ||
| test_router_fuzz_regressions_rust_only.py | ||
| test_router_get_deployments.py | ||
| test_router_helper_utils.py | ||
| test_router_host.py | ||
| test_router_max_parallel_requests.py | ||
| test_router_provider_endpoints.py | ||
| test_router_retries.py | ||
| test_router_surface_check.py | ||
| test_router_timeout.py | ||
| test_router_vector_store_endpoints.py | ||