mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
7ba47a5b6e
The budget-tier reset read every customer linked to an expiring tier into one result set before the write, then invalidated their caches one key at a time. Both of those scale with the customer count, so a large enough deployment can OOM the proxy pod on the read, and the tail of the population sits on a stale spend counter while the per-key invalidations drain PR #40639 moved the reset write itself to a link-based UPDATE, so that pre-commit read no longer feeds the write. It only fed cache invalidation and the service-logging counts, which means it can move after the commit. This replaces it with a keyset walk over litellm_endusertable ordered by user_id, taking RESET_BUDGET_JOB_BATCH_SIZE rows per page, the same shape _reset_windows_for_source already uses, with no per-run page cap for the same reason that walk has none: the cursor cannot survive the run, so a cap would restart at the first customer on every tick and never reach the tail Each page's counter and cache keys now go out as one batched delete through a new DualCache.async_delete_cache_keys, which drops the in-memory entries and chunks the Redis DELETE at DEFAULT_MAX_REDIS_BATCH_CACHE_SIZE num_endusers_found and num_endusers_updated now report the customers whose caches were invalidated after the commit rather than the rows read before it, so both read 0 when the cascade write fails |
||
|---|---|---|
| .. | ||
| base_token_counter_test.py | ||
| conftest.py | ||
| log.txt | ||
| test_aiohttp_handler.py | ||
| test_anthropic_token_counter.py | ||
| test_aws_secret_manager.py | ||
| test_azure_ai_anthropic_token_counter.py | ||
| test_bedrock_token_counter.py | ||
| test_cyberark.py | ||
| test_get_secret.py | ||
| test_hashicorp.py | ||
| test_health_check.py | ||
| test_litellm_overhead.py | ||
| test_logging_callback_manager.py | ||
| test_proxy_budget_reset.py | ||
| test_secret_manager.py | ||
| test_utils.py | ||
| test_validate_tool_choice.py | ||
| vertex_key.json | ||