From 4a47416c259c0bdff83b438ecb146a8ec0d4e308 Mon Sep 17 00:00:00 2001 From: Tin Chi Lo Date: Tue, 4 Aug 2026 00:18:08 -0700 Subject: [PATCH] docs(spend): say why the staging cap is not BaseUpdateQueue's SpendUpdateQueue and DailySpendUpdateQueue both bound their in-memory buffers by subclassing BaseUpdateQueue, so counting a cap by hand here reads like not knowing the base class exists. It is deliberate: BaseUpdateQueue bounds on an asyncio.Queue maxsize, and a full queue blocks whoever is putting to it. That backpressure would land on the spend logging path in service of a dashboard rollup, trading a wrong number on one tab for a spend write that never happened --- litellm/proxy/spend_tracking/auto_router_session_queue.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/litellm/proxy/spend_tracking/auto_router_session_queue.py b/litellm/proxy/spend_tracking/auto_router_session_queue.py index a0c94d2000c..b705c044ca2 100644 --- a/litellm/proxy/spend_tracking/auto_router_session_queue.py +++ b/litellm/proxy/spend_tracking/auto_router_session_queue.py @@ -148,6 +148,13 @@ class AutoRouterSessionQueue: held rather than on sessions seen, which is the quantity that actually bounds the memory. Past the cap a turn is dropped and logged, because benchmark rows are not worth an out-of-memory kill. + + That cap is counted here rather than inherited from ``BaseUpdateQueue``, + which ``SpendUpdateQueue`` and ``DailySpendUpdateQueue`` both use, on + purpose: its bound is an ``asyncio.Queue`` maxsize, and a full queue + blocks the producer. Applying that backpressure here would stall spend + logging behind a dashboard rollup. An undercounted benchmark is a wrong + number on one tab; a stalled spend write is money that went unbilled. """ async with self._lock: if self._staged_turns >= self._max_staged_turns: