Provider prompt caches are per-model, so every mid-session tier switch the
complexity auto-router makes lands on a cold cache and pays the full cache write
again. Opt-in cache_warming captures each session's latest payload at the routing
decision and a leader-elected background refresher replays it with max_tokens=1
against every cacheable tier model just under the provider cache TTL, so the
switch is a pure cache read.
A replay is a request, so it is admitted through the request path's own entry
points rather than beside them. For each replay the refresher assembles a request
body, reserves budget through the same wrapper auth calls right after
common_checks, stamps identity with the proxy's own stamper, applies every
key-level, team-level and project-level control, applies the key and team scoped
dynamic logging settings, runs ProxyLogging.pre_call_hook, applies the
fully-blocked-model check, and hands the dict that hook returns to
Router.acompletion or Router.aanthropic_messages. post_call_failure_hook runs on
every rejection and every dispatch failure, so the parallel request slot, the
reserved TPM tokens and the budget reservation all come back. Warming therefore
inherits both halves of every contract it touches (the limiter's descriptors
across every scope with its own configured window, its RPM and max-parallel
check, its upfront reservation and the stash its success callback reconciles
from; the key, team, user, end-user, organization and tag budget counters; every
configured guardrail and pipeline, including the ones defined on the deployment)
instead of reimplementing them. That deletes nine functions and the admission
block they served.
Warming writes no spend logs of its own, so the replay rows are the only record
of warming cost that will exist. They carry the customer's own tags and
spend_logs_metadata with the litellm_cache_warming tag alongside rather than
instead, and they fan out to the key and team scoped loggers, so warming is
included in per-tag chargeback and filterable out of it. Two Request-free blocks
of add_litellm_data_to_request are extracted verbatim as
LiteLLMProxyRequestSetup.add_key_team_project_metadata and
apply_dynamic_logging_settings so both callers share them; the move is
statement-for-statement identical, with no behavior change on the request path.
Ordering there is load-bearing: add_key_level_controls resets data["cache"] and
refills it from key metadata, so it runs after the body is built and a key's own
cache controls override warming's response-cache bypass exactly as they override
a caller's.
Blocked and expired keys are still checked locally because common_checks
dereferences the FastAPI Request; extracting a Request-free core so its other
gates bind on a replay too is a follow-up. Sessions on a key that declares
max_iterations are skipped entirely, because that limiter counts every request on
a session_id and cannot be consulted without incrementing it.
Metadata precedence (litellm_metadata before metadata, stringified) had three
implementations; core_helpers.iter_request_metadata_dicts and
get_request_metadata_field are now the single owner and DeploymentAffinityCheck
deletes its four private copies to delegate to them.
Resolves LIT-4865
00e1fe8822