scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

BUGFIX: Jitter should be added not multiplied (#12877) (#12901)

Browse source 
* Jitter should be added not multiplied

This fixes a bug mentioned in https://github.com/BerriAI/litellm/issues/12877

`JITTER=0.75` is multiplied by `random.random()` so `sleep_seconds*jitter` is a tiny number that is always less than `min_timeout`.

jitter should be added not multiplied

* Add jitter to min_timeout case also

* Cleanup jitter logic

* Always apply jitter
This commit is contained in:
Pathikrit Bhowmick 2025-07-29 18:38:08 -04:00 • committed by GitHub
parent f3b1b416d1
commit 52f097eb9d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 12 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
litellm/utils.py
View file

@ -5639,22 +5639,23 @@ def _calculate_retry_after(
min_timeout: int = 0,
) -> Union[float, int]:
retry_after = _get_retry_after_from_exception_header(response_headers)
# Add some jitter (default JITTER is 0.75 - so upto 0.75s)
jitter = JITTER * random.random()
# If the API asks us to wait a certain amount of time (and it's a reasonable amount), just do what it says.
if retry_after is not None and 0 < retry_after <= 60:
return retry_after
return retry_after + jitter
initial_retry_delay = INITIAL_RETRY_DELAY
max_retry_delay = MAX_RETRY_DELAY
nb_retries = max_retries - remaining_retries
# Calculate exponential backoff
num_retries = max_retries - remaining_retries
sleep_seconds = INITIAL_RETRY_DELAY * pow(2.0, num_retries)
# Apply exponential backoff, but not more than the max.
sleep_seconds = min(initial_retry_delay * pow(2.0, nb_retries), max_retry_delay)
# Apply some jitter, plus-or-minus half a second.
jitter = JITTER * random.random()
timeout = sleep_seconds * jitter
return timeout if timeout >= min_timeout else min_timeout
# Make sure sleep_seconds is boxed between min_timeout and MAX_RETRY_DELAY
sleep_seconds = max(sleep_seconds, min_timeout)
sleep_seconds = min(sleep_seconds, MAX_RETRY_DELAY)
return sleep_seconds + jitter
# custom prompt helper function