* fix(proxy): charge budget windows when the cached key or team expired before the request ended
The post-call spend update reads a key's or team's budget_limits from
user_api_key_cache and returned early on a miss, skipping both the window
counter increment and the LiteLLM_BudgetWindowSpend row. The cached object
expires after user_api_key_cache_ttl (60s by default), so any request that ran
longer than that without another request refreshing the entry was counted in
the key's lifetime spend but never in its budget windows. With budget
reservation disabled nothing else charges the windows, so keys kept running
past their window caps
On a miss the update now reads budget_limits straight from the key or team row.
It does not re-cache the object, since a write from the spend path could race a
/key/delete and put a just-deleted key back into the auth cache
* test(proxy): assert the budget window fallback never re-caches the key or team
* fix(proxy): read a missing key or team row once per cache TTL in the budget window fallback
The master key and UI session tokens are logged under names no key row
carries, so every one of their requests missed the auth cache and read the
database. A missing row is now remembered for the cache TTL in this process
only. A failed read is not remembered, and a row that exists is read fresh on
every miss so a key update is never served stale.
* fix(proxy): skip the budget window row read for a key or team with no windows
The fallback remembered only a missing row, so a key or team that exists but
holds no budget_limits was read from the database on every request that
outlived the key cache. It now remembers both cases under one cache key built
in user_api_key_cache.py, for the same TTL as the auth cache
* fix(proxy): keep no-window markers in their own cache and clear them when an update adds windows
Markers for keys and teams with no budget windows lived in the shared 200-entry auth cache, so a burst of late costs could evict cached teams and users. They now live in a dedicated in-memory cache on UserApiKeyCache, reached by local and broadcast evictions. /key/update and /team/update now evict the marker on every worker, so a late cost after an update charges the windows the update added.
* test(proxy): give the spend counter tests the proxy's real key cache and type the window spend helper
* fix(proxy): keep a row read that an update outran from restoring the no-window marker
* fix(proxy): clear finished no-window reads and evict a marker-shaped user id from both caches
* fix(proxy): apply other spend counters before a window row read and let no-window notes follow the key cache settings
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
523b87ec71