The capture plane (parent commit) records each warming-enabled session's
latest payload in Redis. This adds the consume side so every tier model's
provider prompt cache stays warm while a session is active and tier
switches bill as cache reads:
- CacheWarmingRefresher.run_tick: single-pod via the Redis cron lock;
when the injected PodLockManager has no Redis attached the refresher
elects through a fallback PodLockManager bound to the warming store's
own Redis, so multi-pod duplicate replays cannot happen whenever
warming itself is active. Enumerates session records, drops idle sessions, skips keys that are deleted, blocked, expired, or at 95% of
max_budget (fails open on lookup errors; the config-defined master key
has no token row and stays eligible), and replays each due payload against every
warm-set model with max_tokens=1, stream=False and response-cache
bypass, bounded by a 10-replay semaphore. The warm set is warm_models
or the first member of each tier pool, filtered to anthropic/bedrock
deployments that support prompt caching; a deployment's declared
custom_llm_provider wins over inference. Chat payloads replay via
router.acompletion with internal metadata in the metadata slot;
anthropic_messages payloads replay via router.aanthropic_messages with
system and tool_choice forwarded and internal metadata in
litellm_metadata, since metadata is the provider body field on that
surface. Replays carry the loop-guard marker, the originating key's
attribution, and the litellm_cache_warming tag unless tag filtering is
enabled. Warmth stamps land even for failed attempts so a dead
deployment is retried once per interval, not every tick.
- proxy_server._complexity_cache_warming_loop: 30s tick modeled on the
adaptive flusher (sleep-first, resolves llm_router fresh each tick,
re-raises CancelledError, survives tick exceptions), started
unconditionally at startup so hot-reloaded routers are covered.
- ComplexityRouter._warm_aware_pick: consulted by _pick_model_for_tier on
both the plain and plugin-narrowed branches. Prefers pool members whose
warmth stamp is fresh within refresh_interval_seconds plus 60s slack,
unions the served model while the session is active, and falls back to
the existing pick when there is no session, record, store, or
intersection.
- README: Cache Warming section covering the YAML shape, the Redis and
store_prompts_in_spend_logs prerequisites, the Redis Cluster
limitation, spend attribution and tagging, max_sessions semantics, and
the session_affinity interplay.
e79ea57fb4