The benchmarks dashboard answered every question by scanning LiteLLM_SpendLogs at
read time: four aggregate queries per auto-router, two of them window functions
over the response JSONB, re-deriving on every page load which model the previous
turn used, how long a tier had been idle, and how big the prefix was last time.
Those are sequential facts and the request that produces them already knows all
of them, so they are now computed once, when the turn happens.
A new LiteLLM_AutoRouterSession row per (session, auto-router) carries both the
counters and the state that classifies the next turn. fold_turn is pure, so every
rate and dollar formula is unit-testable without a database, and the in-memory
queue plus background flusher follow AdaptiveRouterUpdateQueue: atomic increment
upserts, so two pods writing one session compose instead of overwriting. A pod
that has never seen a session loads its row once and classifies from memory
after, which is what keeps a session correct across a restart or a pod move.
The counters are declared once, on TurnDelta. COUNTER_FIELDS derives from that
declaration and the merge, the flush payload and the read query all build off it,
so a metric added there reaches the database and the dashboard without a second
edit. A test asserts the read query aggregates every declared counter; it caught
two that were being written on every request and read by nothing.
The read path is a single aggregate over pre-folded rows covering every
auto-router at once, and touches no per-request table at all. Rollup rows expire
on the existing spend-log retention cutoff, keyed on last activity so a live
conversation is not pruned out from under itself.
Two behaviour fixes came with the move. The turn buckets are now exhaustive: a
session's opening turn used to land in the headline turn count and in none of the
three buckets, so the bucket totals silently disagreed with the headline. And a
turn with no ephemeral cache-creation evidence now reads as the five minute tier
rather than the one hour tier, which had been the default purely because zero is
not less than zero.
Savings come from compute_savings_spend, the same primitive the usage tab uses,
so the two surfaces cannot report different numbers for the same traffic. The
baseline recorded on each row is the one that priced its turns, so the tab names
what the numbers were computed against rather than whatever the config says by
the time someone opens it.
d0c24f2a29