mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
e79ea57fb4
The capture plane (parent commit) records each warming-enabled session's latest payload in Redis. This adds the consume side so every tier model's provider prompt cache stays warm while a session is active and tier switches bill as cache reads: - CacheWarmingRefresher.run_tick: single-pod via the Redis cron lock; when the injected PodLockManager has no Redis attached the refresher elects through a fallback PodLockManager bound to the warming store's own Redis, so multi-pod duplicate replays cannot happen whenever warming itself is active. Enumerates session records, drops idle sessions, skips keys that are deleted, blocked, expired, or at 95% of max_budget (fails open on lookup errors; the config-defined master key has no token row and stays eligible), and replays each due payload against every warm-set model with max_tokens=1, stream=False and response-cache bypass, bounded by a 10-replay semaphore. The warm set is warm_models or the first member of each tier pool, filtered to anthropic/bedrock deployments that support prompt caching; a deployment's declared custom_llm_provider wins over inference. Chat payloads replay via router.acompletion with internal metadata in the metadata slot; anthropic_messages payloads replay via router.aanthropic_messages with system and tool_choice forwarded and internal metadata in litellm_metadata, since metadata is the provider body field on that surface. Replays carry the loop-guard marker, the originating key's attribution, and the litellm_cache_warming tag unless tag filtering is enabled. Warmth stamps land even for failed attempts so a dead deployment is retried once per interval, not every tick. - proxy_server._complexity_cache_warming_loop: 30s tick modeled on the adaptive flusher (sleep-first, resolves llm_router fresh each tick, re-raises CancelledError, survives tick exceptions), started unconditionally at startup so hot-reloaded routers are covered. - ComplexityRouter._warm_aware_pick: consulted by _pick_model_for_tier on both the plain and plugin-narrowed branches. Prefers pool members whose warmth stamp is fresh within refresh_interval_seconds plus 60s slack, unions the served model while the session is active, and falls back to the existing pick when there is no session, record, store, or intersection. - README: Cache Warming section covering the YAML shape, the Redis and store_prompts_in_spend_logs prerequisites, the Redis Cluster limitation, spend attribution and tagging, max_sessions semantics, and the session_affinity interplay. |
||
|---|---|---|
| .. | ||
| .coverage_baseline | ||
| __init__.py | ||
| _coverage_check.py | ||
| _pin_check.py | ||
| conftest.py | ||
| test_background_health.py | ||
| test_exception_handlers.py | ||
| test_harness_smoke.py | ||
| test_lifecycle.py | ||
| test_openapi_customization.py | ||
| test_proxy_config.py | ||
| test_routes_anthropic_beta.py | ||
| test_routes_assistants.py | ||
| test_routes_audio.py | ||
| test_routes_chat_completions.py | ||
| test_routes_completions.py | ||
| test_routes_config.py | ||
| test_routes_embeddings.py | ||
| test_routes_invitation.py | ||
| test_routes_login_sso.py | ||
| test_routes_misc.py | ||
| test_routes_model_cost_map.py | ||
| test_routes_model_info.py | ||
| test_routes_model_metrics.py | ||
| test_routes_models.py | ||
| test_routes_moderations.py | ||
| test_routes_onboarding.py | ||
| test_routes_queue.py | ||
| test_routes_threads.py | ||
| test_routes_utils.py | ||
| test_spend_counters.py | ||
| test_streaming_helpers.py | ||
| test_team_model_name_translation.py | ||