mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
2e6d196684
Cost tells an operator the auto-router is cheaper, not that it is good. This adds the two quality signals that are already recorded on every request, so no new instrumentation is needed: escalation: a turn moved to a costlier model than the turn before it, inside one session. High precision, low recall. Only upward moves count, since a move down to a cheaper model is a deliberate price or latency trade, not evidence the router picked wrong. abandonment: the caller hung up mid-stream. Low precision, high recall, so it catches the giving-up that escalation structurally misses. They render side by side rather than blended, because they fail in opposite directions and one number would hide which of them fired. Both are computed for auto-routed sessions and for the same keys' directly addressed sessions, with identical definitions over LiteLLM_SpendLogs, so the two are comparable. The baseline is restricted to keys that actually used the router, since keys carry different model lists and pooling all non-router traffic would compare unlike populations. It is withheld entirely when the cohort is too small or when its traffic is not carrying caller-supplied session IDs, with the reason shown in place of a misleading rate. Neither signal can come from LiteLLM_AutoRouterSession: that rollup folds a session to per-model last-touch facts and discards turn order, and escalation is a question about order. Also moves avg saved per session up under the headline cost figure. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||