diff --git a/docs/my-website/docs/completion/input.md b/docs/my-website/docs/completion/input.md index cc058935221..78bf0b60f8c 100644 --- a/docs/my-website/docs/completion/input.md +++ b/docs/my-website/docs/completion/input.md @@ -264,7 +264,9 @@ messages=[{"role": "user", "content": [ - `api_version`: *string (optional)* - (Azure-specific) the api version for the call -- `num_retries`: *int (optional)* - The number of times to retry the API call if an APIError, TimeoutError or ServiceUnavailableError occurs +- `num_retries`: *int (optional)* - The number of times litellm retries the API call (with backoff) if an APIError, TimeoutError or ServiceUnavailableError occurs. **Note:** This is independent of `max_retries`, which controls the underlying SDK client's own retry loop (default 2 for OpenAI/Azure, i.e. 3 attempts per request). When both are active they **multiply** — e.g. `num_retries=3` with the default `max_retries=2` produces up to 12 HTTP requests. Set `max_retries=0` when using `num_retries` to get predictable retry counts. + +- `max_retries`: *int (optional)* - The number of retries the underlying SDK client (e.g. openai-python) makes per request before returning an error to litellm. Default is **2** (3 total attempts). Set to `0` to disable SDK-level retries and let `num_retries` be the sole retry layer. - `context_window_fallback_dict`: *dict (optional)* - A mapping of model to use if call fails due to context window error diff --git a/docs/my-website/docs/completion/reliable_completions.md b/docs/my-website/docs/completion/reliable_completions.md index f38917fe53d..90502341cf3 100644 --- a/docs/my-website/docs/completion/reliable_completions.md +++ b/docs/my-website/docs/completion/reliable_completions.md @@ -31,6 +31,27 @@ response = completion( ) ``` +### Avoiding retry amplification + +`num_retries` controls litellm's retry loop, but the underlying SDK client (e.g. openai-python) has its **own** retry loop via `max_retries` (default **2**, i.e. 3 total attempts per request). When both are active they multiply: + +| `num_retries` | `max_retries` (SDK) | Total HTTP requests (worst case) | +|---|---|---| +| 3 | 2 (default) | **12** (4 × 3) | +| 3 | 0 | **4** | +| 0 | 2 (default) | **3** | + +To use only litellm-level retries and get predictable retry counts, set `max_retries=0`: + +```python +response = completion( + model="gpt-3.5-turbo", + messages=messages, + num_retries=2, + max_retries=0, # disable SDK-level retries +) +``` + ## Fallbacks (SDK) :::info diff --git a/litellm/main.py b/litellm/main.py index 9de91057300..d794cae5816 100644 --- a/litellm/main.py +++ b/litellm/main.py @@ -1136,8 +1136,8 @@ def completion( # type: ignore # noqa: PLR0915 LITELLM Specific Params mock_response (str, optional): If provided, return a mock completion response for testing or debugging purposes (default is None). custom_llm_provider (str, optional): Used for Non-OpenAI LLMs, Example usage for bedrock, set model="amazon.titan-tg1-large" and custom_llm_provider="bedrock" - max_retries (int, optional): The number of retries the underlying SDK client makes per request (default is 2, i.e. 3 total attempts). This is independent of ``num_retries``, which controls litellm's own retry loop. When both are set they multiply: e.g. ``num_retries=3`` with the default ``max_retries=2`` produces up to 12 HTTP requests (4 litellm attempts × 3 SDK attempts each). Set ``max_retries=0`` when using ``num_retries`` to avoid compounding retries. - num_retries (int, optional): The number of times litellm retries the call on transient errors (429, 5xx, timeout) with backoff. Unlike ``max_retries`` (which controls the SDK client), this wraps the entire completion call. See ``max_retries`` note above about interaction between the two. + max_retries (int, optional): The number of retries the underlying SDK client makes per request (default is 2, i.e. 3 total attempts). In ``completion()``, ``num_retries`` overwrites this value. In ``completion_with_retries()`` and the Router, they are independent and multiply — e.g. ``num_retries=3`` with ``max_retries=2`` produces up to 12 HTTP requests. Set ``max_retries=0`` when using ``num_retries`` via the Router or ``completion_with_retries()`` to avoid compounding. + num_retries (int, optional): In ``completion()``, this is an alias for ``max_retries`` (overwrites it). In ``completion_with_retries()`` and the Router, this drives a separate tenacity retry loop around the full call. See ``max_retries`` note above about the interaction. Returns: ModelResponse: A response object containing the generated completion and associated metadata. @@ -1263,12 +1263,15 @@ def completion( # type: ignore # noqa: PLR0915 verbose_logger.warning(f"Failed to get proxy auth headers: {e}") num_retries = kwargs.get( "num_retries", None - ) ## alt. param for 'max_retries'. Use this to pass retries w/ instructor. - # NOTE: max_retries controls the *SDK-level* retry loop (default 2 in - # openai-python, i.e. 3 total attempts per request). num_retries controls - # litellm's *own* retry loop. When both are active they multiply — e.g. - # num_retries=3 + max_retries=2 → up to 12 HTTP requests. If you rely on - # num_retries for retry logic, pass max_retries=0 to avoid compounding. + ) ## litellm-level retry loop (wraps full completion call); distinct from max_retries (SDK-level). See NOTE below. + # NOTE: In completion(), num_retries is aliased to max_retries (line ~1351) + # and passed to the SDK client. In completion_with_retries(), they are + # independent: num_retries drives a tenacity loop while max_retries is + # reset to 0. When calling completion() directly with num_retries, the + # SDK default of max_retries=2 is overwritten, so there is no + # multiplication. But when using the Router or completion_with_retries(), + # both layers can be active — e.g. num_retries=3 + max_retries=2 → up to + # 12 HTTP requests. Pass max_retries=0 to avoid compounding. max_retries = kwargs.get("max_retries", None) cooldown_time = kwargs.get("cooldown_time", None) context_window_fallback_dict = kwargs.get("context_window_fallback_dict", None)