litellm/ui
joshua-berri 076ded32be
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session

Claude Code sends its background calls straight to Haiku under the same session id, so those requests never
reached the auto-router session rollup and the status line under-reported the session. Every billed spend
row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write
resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend
row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total
without a shared Redis

* fix(auto_router): name the unsupported provider when a turn routes off Anthropic

A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model,
failed strict native validation and was logged as observation_unavailable, which reads like a broken
capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their
existing reasons, and no estimate is added, so savings for these turns stay unknown

* docs(auto_router): explain when session savings stay unestimated

saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's
first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected
for that window

* fix(auto_router): say why a system-only cache prefix has no savings estimate

Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a
body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails
only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the
empty body still count it. A live check against claude-opus-5-5 showed a differential count with a
placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated
and the savings gap for Claude Code is not fixed here

* fix(auto_router): declare the session total's Redis key family and test its expiry

The session total read and wrote Redis without a service target, so its calls showed up as bare redis
spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a
hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an
active session keeps its total and an idle one restarts

* docs(auto_router): note which turns are still estimated inside the history window

Turns that took the identical baseline path, and turns whose whole cached prefix the session had already
written, are estimated even inside the first cache TTL window, so saved_spend now says so

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 17:07:11 -08:00
..
docker-entrypoint.d feat(helm): support a separate hostname per component in the componentized deploy (#46360) 2026-10-14 14:56:36 -07:00
litellm-dashboard feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066) 2026-11-10 17:07:11 -08:00
Dockerfile build(docker): refresh base images and ship only runtime files in component images (#48279) 2026-10-24 22:33:45 -07:00
nginx.conf feat(helm): support a separate hostname per component in the componentized deploy (#46360) 2026-10-14 14:56:36 -07:00