mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
3d2e18f41d
Provider prompt caches are per-model, so every mid-session tier switch the complexity auto-router makes lands on a cold cache and pays the full cache write again. Opt-in cache_warming captures each session's latest payload at the routing decision and a leader-elected background refresher replays it with max_tokens=1 against every cacheable tier model just under the provider cache TTL, so the switch is a pure cache read. A replay is a request, so it is admitted through the request path's own entry points rather than beside them. For each replay the refresher assembles a request body, reserves budget through the same wrapper auth calls right after common_checks, stamps identity with the proxy's own stamper, applies every key-level, team-level and project-level control, applies the key and team scoped dynamic logging settings, runs ProxyLogging.pre_call_hook, applies the fully-blocked-model check, and hands the dict that hook returns to Router.acompletion or Router.aanthropic_messages. post_call_failure_hook runs on every rejection and every dispatch failure, so the parallel request slot, the reserved TPM tokens and the budget reservation all come back. Warming therefore inherits both halves of every contract it touches (the limiter's descriptors across every scope with its own configured window, its RPM and max-parallel check, its upfront reservation and the stash its success callback reconciles from; the key, team, user, end-user, organization and tag budget counters; every configured guardrail and pipeline, including the ones defined on the deployment) instead of reimplementing them. That deletes nine functions and the admission block they served. Warming writes no spend logs of its own, so the replay rows are the only record of warming cost that will exist. They carry the customer's own tags and spend_logs_metadata with the litellm_cache_warming tag alongside rather than instead, and they fan out to the key and team scoped loggers, so warming is included in per-tag chargeback and filterable out of it. Two Request-free blocks of add_litellm_data_to_request are extracted verbatim as LiteLLMProxyRequestSetup.add_key_team_project_metadata and apply_dynamic_logging_settings so both callers share them; the move is statement-for-statement identical, with no behavior change on the request path. Ordering there is load-bearing: add_key_level_controls resets data["cache"] and refills it from key metadata, so it runs after the body is built and a key's own cache controls override warming's response-cache bypass exactly as they override a caller's. Blocked and expired keys are still checked locally because common_checks dereferences the FastAPI Request; extracting a Request-free core so its other gates bind on a replay too is a follow-up. Sessions on a key that declares max_iterations are skipped entirely, because that limiter counts every request on a session_id and cannot be consulted without incrementing it. Metadata precedence (litellm_metadata before metadata, stringified) had three implementations; core_helpers.iter_request_metadata_dicts and get_request_metadata_field are now the single owner and DeploymentAffinityCheck deletes its four private copies to delegate to them. Resolves LIT-4865 |
||
|---|---|---|
| .. | ||
| .coverage_baseline | ||
| __init__.py | ||
| _coverage_check.py | ||
| _pin_check.py | ||
| conftest.py | ||
| test_background_health.py | ||
| test_exception_handlers.py | ||
| test_harness_smoke.py | ||
| test_lifecycle.py | ||
| test_openapi_customization.py | ||
| test_proxy_config.py | ||
| test_routes_anthropic_beta.py | ||
| test_routes_assistants.py | ||
| test_routes_audio.py | ||
| test_routes_chat_completions.py | ||
| test_routes_completions.py | ||
| test_routes_config.py | ||
| test_routes_embeddings.py | ||
| test_routes_invitation.py | ||
| test_routes_login_sso.py | ||
| test_routes_misc.py | ||
| test_routes_model_cost_map.py | ||
| test_routes_model_info.py | ||
| test_routes_model_metrics.py | ||
| test_routes_models.py | ||
| test_routes_moderations.py | ||
| test_routes_onboarding.py | ||
| test_routes_queue.py | ||
| test_routes_threads.py | ||
| test_routes_utils.py | ||
| test_spend_counters.py | ||
| test_streaming_helpers.py | ||
| test_team_model_name_translation.py | ||