mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
docs(proxy): describe the health-check routing startup window
Until the first background cycle finishes, every deployment is treated as healthy. A lower model_info.health_check_timeout shortens that window.
This commit is contained in:
parent
f25fff42a6
commit
d61dc49ce3
1 changed files with 9 additions and 0 deletions
9
docs/proxy/health_check_routing.md
Normal file
9
docs/proxy/health_check_routing.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# Health check driven routing
|
||||
|
||||
## Startup behavior
|
||||
|
||||
Until the first background cycle completes, every deployment is treated as healthy. A request in that window can still be sent to a deployment that is already down, and it waits for that deployment's request timeout before falling back.
|
||||
|
||||
The cycle writes those results only after its probes finish. Each probe waits up to `model_info.health_check_timeout` seconds, or `HEALTH_CHECK_TIMEOUT_SECONDS` when that field is unset (60 seconds by default, overridable with the `HEALTH_CHECK_TIMEOUT_SECONDS` environment variable). A lower `model_info.health_check_timeout` shortens this window.
|
||||
|
||||
When `allowed_fails_policy` is set and `background_health_check_model_groups` is not, cooldown is what removes a deployment. That counter is also empty until the first cycle records a failure, so the same window applies.
|
||||
Loading…
Add table
Reference in a new issue