mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
c5d56e4e86
The benchmarks dashboard answered every question by scanning LiteLLM_SpendLogs at read time: four aggregate queries per auto-router, two of them window functions over the response JSONB, re-deriving on every page load which model the previous turn used, how long a tier had been idle, and how big the prefix was last time. Those are sequential facts and the request that produces them already knows all of them, so they are now computed once, when the turn happens. A new LiteLLM_AutoRouterSession row per (session, auto-router) carries both the counters and the state that classifies the next turn. fold_turn is pure, so every rate and dollar formula is unit-testable without a database, and the in-memory queue plus background flusher follow AdaptiveRouterUpdateQueue: atomic increment upserts, so two pods writing one session compose instead of overwriting. A pod that has never seen a session loads its row once and classifies from memory after, which is what keeps a session correct across a restart or a pod move. The counters are declared once, on TurnDelta. COUNTER_FIELDS derives from that declaration and the merge, the flush payload and the read query all build off it, so a metric added there reaches the database and the dashboard without a second edit. A test asserts the read query aggregates every declared counter; it caught two that were being written on every request and read by nothing. The read path is a single aggregate over pre-folded rows covering every auto-router at once, and touches no per-request table at all. Rollup rows expire on the existing spend-log retention cutoff, keyed on last activity so a live conversation is not pruned out from under itself. Two behaviour fixes came with the move. The turn buckets are now exhaustive: a session's opening turn used to land in the headline turn count and in none of the three buckets, so the bucket totals silently disagreed with the headline. And a turn with no ephemeral cache-creation evidence now reads as the five minute tier rather than the one hour tier, which had been the default purely because zero is not less than zero. Savings come from compute_savings_spend, the same primitive the usage tab uses, so the two surfaces cannot report different numbers for the same traffic. The baseline recorded on each row is the one that priced its turns, so the tab names what the numbers were computed against rather than whatever the config says by the time someone opens it. |
||
|---|---|---|
| .. | ||
| dist | ||
| litellm_proxy_extras | ||
| tests | ||
| build_and_publish.md | ||
| LICENSE | ||
| migration_runbook.md | ||
| pyproject.toml | ||
| README.md | ||
Additional files for the proxy. Reduces the size of the main litellm package.
Currently, only stores the migration.sql files for litellm-proxy.
To install, run:
uv add litellm-proxy-extras
OR
uv tool install 'litellm[proxy]' # installs litellm-proxy-extras and other proxy dependencies
To use the migrations, run:
litellm --use_prisma_migrate