scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-15 23:31:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 165

docs(spend): say why the staging cap is not BaseUpdateQueue's

Browse source 
SpendUpdateQueue and DailySpendUpdateQueue both bound their in-memory buffers by
subclassing BaseUpdateQueue, so counting a cap by hand here reads like not
knowing the base class exists. It is deliberate: BaseUpdateQueue bounds on an
asyncio.Queue maxsize, and a full queue blocks whoever is putting to it. That
backpressure would land on the spend logging path in service of a dashboard
rollup, trading a wrong number on one tab for a spend write that never happened
This commit is contained in:
Tin Chi Lo 2026-08-04 00:18:08 -07:00
parent a5e6720613
commit 4a47416c25
1 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
litellm/proxy/spend_tracking/auto_router_session_queue.py
View file

@ -148,6 +148,13 @@ class AutoRouterSessionQueue:
held rather than on sessions seen, which is the quantity that actually
bounds the memory. Past the cap a turn is dropped and logged, because
benchmark rows are not worth an out-of-memory kill.
That cap is counted here rather than inherited from ``BaseUpdateQueue``,
which ``SpendUpdateQueue`` and ``DailySpendUpdateQueue`` both use, on
purpose: its bound is an ``asyncio.Queue`` maxsize, and a full queue
blocks the producer. Applying that backpressure here would stall spend
logging behind a dashboard rollup. An undercounted benchmark is a wrong
number on one tab; a stalled spend write is money that went unbilled.
"""
async with self._lock:
if self._staged_turns >= self._max_staged_turns: