mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
f769aa4675
Cooldown entries rode the router-wide DualCache, which re-reads a key that is missing from memory at most once every 10s. A deployment benched on one replica therefore kept taking traffic on its siblings for up to 10 seconds, and the same shared in-memory tier could evict a live cooldown once 200 unrelated router keys crowded it out, which sent even the benching replica back to the dead deployment. CooldownCache now owns a DualCache over the router's Redis with a 1s read interval and an in-memory tier that only holds cooldown keys. Redis is attached lazily because the router builds the cooldown cache before it wires Redis up. |
||
|---|---|---|
| .. | ||
| conftest.py | ||
| create_mock_standard_logging_payload.py | ||
| gettysburg.wav | ||
| README.md | ||
| test_completion_no_copy.py | ||
| test_default_deployment_copy.py | ||
| test_get_model_list_alias_optimization.py | ||
| test_pre_call_checks_optimization.py | ||
| test_prompt_management_check.py | ||
| test_router_acancel_batch.py | ||
| test_router_adding_deployments.py | ||
| test_router_anthropic_messages_fallback.py | ||
| test_router_aresponses_streaming_fallback.py | ||
| test_router_batch_utils.py | ||
| test_router_cooldown_per_deployment.py | ||
| test_router_cooldown_utils.py | ||
| test_router_embedding_headers.py | ||
| test_router_embedding_integration.py | ||
| test_router_endpoints.py | ||
| test_router_handle_error.py | ||
| test_router_helper_utils.py | ||
| test_router_index_management.py | ||
| test_router_prompt_caching.py | ||
Router component unit tests.
Please name all files with the word 'router' in them.
This is used to ensure all functions in the router are tested.