mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-17 23:51:30 +00:00
f769aa4675
Cooldown entries rode the router-wide DualCache, which re-reads a key that is missing from memory at most once every 10s. A deployment benched on one replica therefore kept taking traffic on its siblings for up to 10 seconds, and the same shared in-memory tier could evict a live cooldown once 200 unrelated router keys crowded it out, which sent even the benching replica back to the dead deployment. CooldownCache now owns a DualCache over the router's Redis with a 1s read interval and an in-memory tier that only holds cooldown keys. Redis is attached lazily because the router builds the cooldown cache before it wires Redis up. |
||
|---|---|---|
| .. | ||
| pre_call_checks | ||
| test_add_retry_fallback_headers.py | ||
| test_auto_router_model_naming.py | ||
| test_auto_router_tuning_baseline.py | ||
| test_cooldown_cache.py | ||
| test_cooldown_handlers.py | ||
| test_fallback_event_handlers.py | ||
| test_get_retry_from_policy.py | ||
| test_health_check_allowed_fails_integration.py | ||
| test_health_state_cache.py | ||
| test_pattern_match_deployments.py | ||
| test_reasoning_effort_capability.py | ||
| test_router_health_check_routing.py | ||
| test_router_interactions_endpoints.py | ||
| test_router_utils_common_utils.py | ||