diff --git a/docs/my-website/docs/proxy/health.md b/docs/my-website/docs/proxy/health.md index 530bea3d06b..1d893961b62 100644 --- a/docs/my-website/docs/proxy/health.md +++ b/docs/my-website/docs/proxy/health.md @@ -316,86 +316,9 @@ general_settings: ## Health Check Driven Routing -By default, background health checks are observability-only — they populate the `/health` endpoint but don't affect routing. Unhealthy deployments still receive traffic until request failures trigger cooldown. +Route traffic away from unhealthy deployments proactively — before user requests hit them. Supports per-error-type failure thresholds, transient error suppression, and automatic safety nets. -With `enable_health_check_routing: true`, the router **excludes deployments that failed their last background health check** before selecting a candidate. This gives you proactive failover instead of reactive cooldown. - -### How it works - -1. Background health checks run on their configured interval -2. After each cycle, every deployment is marked healthy or unhealthy -3. On each incoming request, the router filters out unhealthy deployments **before** cooldown filtering and load balancing -4. If all deployments are unhealthy, the filter is bypassed (safety net — never causes a total outage) -5. If health state is stale (older than `health_check_staleness_threshold`), it is ignored - -### Quick start - -```yaml -model_list: - - model_name: gpt-4 - litellm_params: - model: openai/gpt-4 - api_key: os.environ/OPENAI_API_KEY - - model_name: gpt-4 - litellm_params: - model: openai/gpt-4 - api_key: os.environ/OPENAI_API_KEY_SECONDARY - -general_settings: - background_health_checks: true - health_check_interval: 60 - enable_health_check_routing: true -``` - -### Configuration - -| Setting | Where | Default | Description | -|---------|-------|---------|-------------| -| `enable_health_check_routing` | `general_settings` | `false` | Enable/disable health-check-driven routing | -| `health_check_staleness_threshold` | `general_settings` | `health_check_interval * 2` | Seconds before health state is considered stale and ignored | -| `background_health_checks` | `general_settings` | `false` | Must be `true` for health check routing to work | -| `health_check_interval` | `general_settings` | `300` | Seconds between health check cycles | - -### Interaction with cooldown - -Health check filtering and cooldown are **additive**. A deployment can be excluded by either mechanism: - -- **Health check filter** — proactive, runs on the configured interval, excludes deployments that failed the last check -- **Cooldown** — reactive, triggered by request failures, excludes deployments for a short TTL - -This means request failures still provide fast detection between health check intervals. - -### Staleness - -If a health check result is older than `health_check_staleness_threshold`, it is ignored and the deployment is treated as eligible. This prevents stale data from permanently excluding a deployment if the health check loop stops or slows down. - -The default staleness threshold is `health_check_interval * 2`. For a 60s interval, health state expires after 120s. - -### Example: custom staleness - -```yaml -general_settings: - background_health_checks: true - health_check_interval: 30 - enable_health_check_routing: true - health_check_staleness_threshold: 90 # ignore health state older than 90s -``` - -### Debugging - -Run the proxy with `--detailed_debug` and look for: - -``` -health_check_routing_state_updated healthy=3 unhealthy=1 -``` - -This is logged after each health check cycle when routing state is written. - -If the safety net triggers (all deployments unhealthy), you'll see: - -``` -All deployments marked unhealthy by health checks, bypassing health filter -``` +See the full guide: [Health Check Driven Routing](./health_check_routing.md) ## Health Check Timeout diff --git a/docs/my-website/docs/proxy/health_check_routing.md b/docs/my-website/docs/proxy/health_check_routing.md new file mode 100644 index 00000000000..daf0b19212c --- /dev/null +++ b/docs/my-website/docs/proxy/health_check_routing.md @@ -0,0 +1,340 @@ +# Health Check Driven Routing + +Route traffic away from unhealthy deployments before users hit errors. Background health checks run on a configurable interval, and any deployment that fails gets removed from the routing pool proactively, not after a user request already failed. + + +## Architecture + + + {/* Background */} + + + {/* LEFT PANEL: Background health check loop */} + + Background Loop + every health_check_interval seconds + + {/* Deployment A */} + + Deployment A + ahealth_check() → 200 ✓ + + {/* Deployment B */} + + Deployment B + ahealth_check() → 401 ✗ + + {/* Deployment C */} + + Deployment C + ahealth_check() → 429 ⚡ + + {/* ignore_transient box */} + + ignore_transient_errors: true + 429 / 408 → ignored + not written to cache + + {/* allowed_fails_policy box */} + + allowed_fails_policy + 401 → increment counter + counter > threshold + → cooldown triggered + + {/* CENTER PANEL: Shared State */} + + Shared State + + {/* Health State Cache */} + + DeploymentHealthCache + A → healthy ✓ + B → unhealthy ✗ + C → not written (ignored) + TTL: staleness_threshold × 1.5 + + {/* Cooldown Cache */} + + Cooldown Cache + B → cooling down + (after policy threshold) + TTL: cooldown_time + + {/* failed_calls counter */} + + failed_calls counter + B: 2 / AuthAllowedFails: 1 + → threshold exceeded + TTL: cooldown_time (must > interval) + + {/* RIGHT PANEL: Request path */} + + Request Path + + {/* Incoming request */} + + Incoming request + + {/* All deployments */} + + All deployments [A, B, C] + + + + {/* Health check filter */} + + ① Health Check Filter + if policy set → bypass + else → remove unhealthy + + + + {/* Cooldown filter */} + + ② Cooldown Filter + remove deployments in cooldown + + + + {/* Safety net */} + + Safety Net + if all removed → return all + + + + {/* Load balancer */} + + ③ Load Balancer + + + + {/* Selected deployment */} + + Selected: Deployment A ✓ + + + + {/* ARROWS: left → center */} + + + + + + {/* ARROWS: center → right */} + + + + {/* Arrow markers */} + + + + + + + + + + + + + + + + + + + + + + + +## What problem does this solve? + +By default, LiteLLM routes traffic to all deployments and only stops sending to a broken one after it has already failed a user request. The cooldown system is reactive. + +Health check driven routing makes this **proactive**: a background loop pings every deployment on a configurable interval. If a deployment fails its health check, it gets removed from the routing pool immediately, before a user request lands on it. + +When you also set `allowed_fails_policy`, you control exactly how many health check failures of each error type (auth errors, rate limits, timeouts) are needed before a deployment enters cooldown. This avoids false positives from transient noise. + + +## Setup + +### Step 1: Enable background health checks + +Background health checks are off by default. Turn them on in `general_settings`: + +```yaml +general_settings: + background_health_checks: true + health_check_interval: 60 # seconds between each full check cycle +``` + +### Step 2: Enable health check routing + +```yaml +general_settings: + background_health_checks: true + health_check_interval: 60 + enable_health_check_routing: true # ← route away from unhealthy deployments +``` + +At this point, any deployment that fails its health check is immediately excluded from routing until the next check cycle clears it. + +### Step 3: Add a policy to control how many failures trigger cooldown + +Without a policy, the first health check failure marks a deployment as unhealthy. If you want more tolerance (e.g., only act after 2 consecutive auth failures), use `allowed_fails_policy`: + +```yaml +model_list: + - model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-5 + api_key: os.environ/ANTHROPIC_API_KEY + + - model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-5 + api_key: os.environ/ANTHROPIC_API_KEY_SECONDARY + +general_settings: + background_health_checks: true + health_check_interval: 30 + enable_health_check_routing: true + +router_settings: + cooldown_time: 60 # how long a deployment stays in cooldown + allowed_fails_policy: + AuthenticationErrorAllowedFails: 1 # cooldown after 2nd auth failure + TimeoutErrorAllowedFails: 3 # cooldown after 4th timeout +``` + +When `allowed_fails_policy` is set, the binary health check filter is bypassed. Only the cooldown system controls routing exclusion, and it only fires after your configured threshold is crossed. + +### Step 4 (optional): Ignore transient errors + +429 (rate limit) and 408 (timeout) from a health check usually mean the deployment is temporarily overloaded, not broken. To prevent these from affecting routing at all: + +```yaml +general_settings: + background_health_checks: true + health_check_interval: 30 + enable_health_check_routing: true + health_check_ignore_transient_errors: true # 429 and 408 never affect routing +``` + +With this on, only hard failures (401, 404, 5xx) from health checks contribute to cooldown. + + +## Full example + +```yaml +model_list: + - model_name: gpt-4o + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY + + - model_name: gpt-4o + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY_SECONDARY + + - model_name: gpt-4o + litellm_params: + model: azure/gpt-4o + api_base: os.environ/AZURE_API_BASE + api_key: os.environ/AZURE_API_KEY + +general_settings: + background_health_checks: true + health_check_interval: 30 + enable_health_check_routing: true + health_check_ignore_transient_errors: true + +router_settings: + cooldown_time: 60 + allowed_fails_policy: + AuthenticationErrorAllowedFails: 0 # cooldown immediately on auth failure + TimeoutErrorAllowedFails: 2 # cooldown after 3 timeouts + RateLimitErrorAllowedFails: 5 # cooldown after 6 rate limits (if not ignoring transients) +``` + + +## Configuration reference + +| Setting | Where | Default | Description | +|---|---|---|---| +| `enable_health_check_routing` | `general_settings` | `false` | Route away from deployments that fail health checks | +| `background_health_checks` | `general_settings` | `false` | Must be `true` for health check routing to work | +| `health_check_interval` | `general_settings` | `300` | Seconds between full health check cycles | +| `health_check_staleness_threshold` | `general_settings` | `interval x 2` | Seconds before cached health state is ignored | +| `health_check_ignore_transient_errors` | `general_settings` | `false` | Ignore 429 and 408 from health checks; these never affect routing | +| `cooldown_time` | `router_settings` | `5` | Seconds a deployment stays in cooldown after threshold is crossed | +| `allowed_fails_policy` | `router_settings` | `null` | Per-error-type failure thresholds before cooldown (see below) | + +### `allowed_fails_policy` fields + +| Field | Error type | HTTP status | +|---|---|---| +| `AuthenticationErrorAllowedFails` | Bad API key | 401 | +| `TimeoutErrorAllowedFails` | Request timeout | 408 | +| `RateLimitErrorAllowedFails` | Rate limit exceeded | 429 | +| `BadRequestErrorAllowedFails` | Malformed request | 400 | +| `ContentPolicyViolationErrorAllowedFails` | Content filtered | 400 | + +The value is the number of failures **tolerated** before cooldown. `0` means cooldown on the first failure. `2` means cooldown on the third. + + +## Things to keep in mind + +- **Counter TTL must be longer than the health check interval.** `allowed_fails_policy` works by incrementing a `failed_calls` counter per deployment. That counter expires after `cooldown_time` seconds. If `cooldown_time` is shorter than `health_check_interval`, the counter resets between every check cycle and failures never accumulate. Set `cooldown_time` greater than `health_check_interval` when using `allowed_fails_policy`. + + ```yaml + router_settings: + cooldown_time: 60 # must be > health_check_interval (30s here) + + general_settings: + health_check_interval: 30 + ``` + +- **`AllowedFails: N` means cooldown on the (N+1)th failure.** The counter check is `updated_fails > allowed_fails`, so `0` triggers on the 1st failure, `1` on the 2nd, `2` on the 3rd. + + | `AllowedFails` | Cooldown triggers after | + |---|---| + | `0` | 1st failure | + | `1` | 2nd failure | + | `2` | 3rd failure | + +- **Without `allowed_fails_policy`, the first failure is enough.** The first failed health check immediately excludes the deployment from routing. Use `allowed_fails_policy` when you want tolerance for flaky checks. + +- **If all deployments are unhealthy, the filter is bypassed.** Traffic keeps flowing rather than returning no deployment at all. Requests will fail, but the router keeps trying. + +- **Health check failures and request failures share the same counters.** When `allowed_fails_policy` is set, both sources increment the same `failed_calls` counter. A deployment at 1 health check failure that then receives 1 failing request will hit the threshold for `AllowedFails: 1` and enter cooldown. + + +## Debugging + +Run the proxy with `--detailed_debug` and look for these log lines: + +After each health check cycle (written at DEBUG level): +``` +health_check_routing_state_updated healthy=2 unhealthy=1 +``` + +When a health check failure increments the counter and triggers cooldown (DEBUG level): +``` +checks 'should_run_cooldown_logic' +Attempting to add to cooldown list +``` + +When safety net fires because all deployments are in cooldown: +``` +All deployments in cooldown via health-check routing, bypassing cooldown filter +``` + +When safety net fires because all deployments are unhealthy (binary filter, no `allowed_fails_policy`): +``` +All deployments marked unhealthy by health checks, bypassing health filter +``` diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 6446e227d99..3681d90fd5f 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -1049,7 +1049,8 @@ const sidebars = { "proxy/fallback_management", "proxy/tag_routing", "proxy/timeout", - "wildcard_routing" + "wildcard_routing", + "proxy/health_check_routing" ], }, {