From d61dc49ce37674f61d211ce1d4a7bc8ab4aa71bb Mon Sep 17 00:00:00 2001 From: Quinn Xu Date: Thu, 5 Nov 2026 18:08:15 +0000 Subject: [PATCH] docs(proxy): describe the health-check routing startup window Until the first background cycle finishes, every deployment is treated as healthy. A lower model_info.health_check_timeout shortens that window. --- docs/proxy/health_check_routing.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/proxy/health_check_routing.md diff --git a/docs/proxy/health_check_routing.md b/docs/proxy/health_check_routing.md new file mode 100644 index 00000000000..136274dfea1 --- /dev/null +++ b/docs/proxy/health_check_routing.md @@ -0,0 +1,9 @@ +# Health check driven routing + +## Startup behavior + +Until the first background cycle completes, every deployment is treated as healthy. A request in that window can still be sent to a deployment that is already down, and it waits for that deployment's request timeout before falling back. + +The cycle writes those results only after its probes finish. Each probe waits up to `model_info.health_check_timeout` seconds, or `HEALTH_CHECK_TIMEOUT_SECONDS` when that field is unset (60 seconds by default, overridable with the `HEALTH_CHECK_TIMEOUT_SECONDS` environment variable). A lower `model_info.health_check_timeout` shortens this window. + +When `allowed_fails_policy` is set and `background_health_check_model_groups` is not, cooldown is what removes a deployment. That counter is also empty until the first cycle records a failure, so the same window applies.