scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102

fix: use correct list length when averaging TTFT latency for streaming requests (#23100)

Browse source 
In _get_available_deployments, when streaming mode is active the code
sums time-to-first-token values from item_ttft_latency but divides by
len(item_latency) instead of len(item_ttft_latency). These lists can
have different lengths, producing an incorrect average that skews
lowest-latency routing decisions for streaming requests.

Signed-off-by: JiangNan <1394485448@qq.com>
This commit is contained in:
JiangNan 2026-03-10 10:49:31 +08:00 • committed by GitHub
parent 0d3735f9c0
commit 3ce37e6c35
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 5 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
litellm/router_strategy/lowest_latency.py
View file

@ -490,20 +490,22 @@ class LowestLatencyLoggingHandler(CustomLogger):
# get average latency or average ttft (depending on streaming/non-streaming)
total: float = 0.0
if (
use_ttft = (
request_kwargs is not None
and request_kwargs.get("stream", None) is not None
and request_kwargs["stream"] is True
and len(item_ttft_latency) > 0
):
)
if use_ttft:
for _call_latency in item_ttft_latency:
if isinstance(_call_latency, float):
total += _call_latency
item_latency = total / len(item_ttft_latency)
else:
for _call_latency in item_latency:
if isinstance(_call_latency, float):
total += _call_latency
item_latency = total / len(item_latency)
item_latency = total / len(item_latency)
# -------------- #
# Debugging Logic