mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
a8b94b9a87
Some checks failed
Agent Shin — rollout heads-up (one-shot) / heads-up (push) Has been cancelled
Budget enforcement reads spend from the cross-pod Redis counter via get_current_spend, which trusted the counter whenever Redis returned a value. A Redis instance that restarts and reloads an older RDB snapshot (the customer's logs repeat "Redis is loading the dataset in memory") comes back with a stale-low counter; that read is a hit, not a clean miss, so the existing DB reseed never ran and a key kept getting admitted even though its recorded spend was already over max_budget. The symptom was recorded spend sitting above the limit while requests kept succeeding. Read-time enforcement: get_current_spend takes an optional max_budget and, when the counter would admit the request but reads below this caller's last-known recorded spend, re-reads the authoritative spend and enforces against the higher value. The authoritative source depends on the counter: key/team/user/org/team-member read the DB row, per-window budgets aggregate spend logs, and end-user/tag have no DB row so the caller's freshly-loaded recorded spend is used. Healthy primary counters and freshly reset keys stay off the DB path, and the value is cached in-process for a few seconds, so a persistently stale counter drives at most one read per counter per window. When the DB value is higher, the counter is repaired with a monotonic, atomic set-max (RedisCache.async_set_max) so every worker reads the corrected total and a concurrent increment is never clobbered. Reconcile no longer fails open: when the post-call reservation reconcile found the counter missing or an adjustment that would drive it negative, it deleted the counter and continued (the deletion is what left counters nil/unenforced after a Redis reload). It now reseeds from the DB's lagging authoritative floor instead of deleting; the monotonic set-max can only raise a stale-low counter, and the read-time floor converges to the true total as the spend buffer flushes. The pre-call admission resize path keeps its original fail-closed behavior. Opt-in strict enforcement: general_settings.fail_closed_budget_enforcement (default False) makes the authoritative re-check run for every budgeted entity (closing the gap where a stale-low counter and a stale-low cached fallback would otherwise both pass the cheap guard), and rejects a request with 503 when the spend backing an admit decision can be verified against neither Redis nor the database. Default behavior is unchanged; the re-check stays bounded by the in-process cache. Resolves LIT-3772 |
||
|---|---|---|
| .. | ||
| .coverage_baseline | ||
| __init__.py | ||
| _coverage_check.py | ||
| _pin_check.py | ||
| conftest.py | ||
| test_background_health.py | ||
| test_exception_handlers.py | ||
| test_harness_smoke.py | ||
| test_lifecycle.py | ||
| test_openapi_customization.py | ||
| test_proxy_config.py | ||
| test_routes_anthropic_beta.py | ||
| test_routes_assistants.py | ||
| test_routes_audio.py | ||
| test_routes_chat_completions.py | ||
| test_routes_completions.py | ||
| test_routes_config.py | ||
| test_routes_embeddings.py | ||
| test_routes_invitation.py | ||
| test_routes_login_sso.py | ||
| test_routes_misc.py | ||
| test_routes_model_cost_map.py | ||
| test_routes_model_info.py | ||
| test_routes_model_metrics.py | ||
| test_routes_models.py | ||
| test_routes_moderations.py | ||
| test_routes_onboarding.py | ||
| test_routes_queue.py | ||
| test_routes_threads.py | ||
| test_routes_utils.py | ||
| test_spend_counters.py | ||
| test_streaming_helpers.py | ||
| test_team_model_name_translation.py | ||