mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-26 06:03:19 +00:00
16645cc963
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts |
||
|---|---|---|
| .. | ||
| agent_engine | ||
| audio_transcription | ||
| batches | ||
| context_caching | ||
| count_tokens | ||
| files | ||
| fine_tuning | ||
| fixtures | ||
| gemini | ||
| gemini_embeddings | ||
| image_edit | ||
| image_generation | ||
| interactions | ||
| multimodal_embeddings | ||
| rag_engine | ||
| realtime | ||
| rerank | ||
| text_to_speech | ||
| vertex_ai_partner_models | ||
| vertex_embeddings | ||
| vertex_gemma_models | ||
| videos | ||
| __init__.py | ||
| google_genai_proxy_test_config.yaml | ||
| test_bge_embedding.py | ||
| test_bge_response_transformation.py | ||
| test_common_utils.py | ||
| test_gemini_batch_embeddings.py | ||
| test_gemini_empty_properties.py | ||
| test_gemini_header_forwarding.py | ||
| test_http_status_201.py | ||
| test_vertex.py | ||
| test_vertex_ai_batch_transformation.py | ||
| test_vertex_ai_common_utils.py | ||
| test_vertex_ai_non_gemini.py | ||
| test_vertex_ai_psc_endpoint_support.py | ||
| test_vertex_ai_search_vector_store_transformation.py | ||
| test_vertex_gemini_gcs_uri_mime.py | ||
| test_vertex_global_url_support.py | ||
| test_vertex_image_generation.py | ||
| test_vertex_llm_base.py | ||
| test_vertex_model_garden_openapi.py | ||
| test_vertex_passthrough_logging_handler.py | ||