Every complexity router now derives and records its savings baseline from
its hardest configured tier, and the spend writer always prices against the
decision-recorded baseline model and deployment id. A leftover
litellm_settings.autorouter_savings_baseline_model key is inert
* feat(spend): report prompt caching savings as total and gateway-attributed
`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.
Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.
Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.
The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.
Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.
* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)
The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.
Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
* feat(ui): session-level cache observability in request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: guard cache_hit filter against non-string defaults in direct calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): drop redundant cache_hit field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): stop billing and logging response reads as LLM calls
Retrieving, deleting or cancelling a stored response, and vector store management calls, run through the same logging lifecycle as inference. A retrieved response replays the usage of the call that created it, so every read priced it again and wrote a second spend log row for the same tokens. Non-inference calls now cost 0, report no usage, log no placeholder chat message, and get a litellm.responses_management operation name instead of reading as chat.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep billing background response jobs after the poll
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): use an empty list for read-call messages
A tuple matches no branch in the loggers that walk this value, so lunary's
parse_messages falls through to clean_message and raises AttributeError on the
success hook. An empty list reads as no messages everywhere: it satisfies the
isinstance(list) checks in newrelic, mlflow and datadog, iterates zero times in
traceloop and helicone, and is what StandardLoggingPayload.messages is typed to
hold. None would be type-legal too but is not iterable, so it trades one crash
for another in mlflow and traceloop.
* fix(otel): stop the legacy emitter reporting replayed tokens on response reads
The zeroing so far lands in the standard logging payload, which the legacy
OpenTelemetry emitter does not read for usage: it takes prompt, completion and
total tokens straight off the response object, so a retrieval span still carried
the token counts of the call that produced the response, and the token usage
histogram still recorded them. That emitter is the default, so the spend row said
zero while the trace said otherwise. The background cost poller keeps its counts,
the same exemption the pricing path already makes.
* fix(logging): keep billing a background response when its retrieval is read
A response created with background=true comes back queued and carries no usage, so
its create bills nothing. The retrieval that first sees the finished job is the only
place that job's tokens are ever visible, and pricing every read at zero therefore
loses the spend outright rather than deduplicating it. On a proxy without the
enterprise cost poller a background job ended up costing $0 end to end.
is_unbilled_non_inference_call now takes the response it is deciding about and treats
a background response the same way it already treats the poller's own read, which is
the same exemption seen from the other side. The legacy OpenTelemetry emitter's time
per output token metric picks up the read gate it was missing, so it stops dividing a
read's latency by the replayed completion token count.
* test(proxy): pass the read response to the non-inference predicate
The poller test called is_unbilled_non_inference_call with the pre-background signature, so it broke when the predicate gained the response it classifies. It now hands the predicate a foreground read, and asserts that the same read is free without the origin stamp, so the stamp is what the test proves.
* fix(otel): stop the v2 metrics recorder reporting replayed tokens on response reads
The v2 span builder sources usage from the standard logging payload, so the
earlier fix already zeroes it there. The metrics recorder reads response_obj
directly, so a responses-management read still recorded the original
generation's tokens into gen_ai.client.token.usage and divided generation time
by them for gen_ai.server.time_per_output_token.
The read still records operation and response duration, under the
litellm.responses_management operation, so it stays observable.
* fix(proxy): keep the response-cost headers on calls priced at zero
Pricing responses reads and vector-store management routes at zero dropped the whole
x-litellm-response-cost family off those replies. The header build reads a falsy zero as
a cost this response never recorded and filters it out, and a call that returns before
pricing stores no cost breakdown for the component headers to read, so a client parsing
the cost off a read got a KeyError where it had previously been handed a number.
Those calls now advertise the family at zero. Retrieving a background response, and the
cost poller's read of one, still report their real cost.
The params-taking form of the predicate moves from opentelemetry into
internal_call_metadata so the proxy header build and the OTEL recorders share one copy.
* fix(proxy): report a zero cost split only under a zero cost total
The component headers were filled from call-type membership alone, while the
total they sit beside keeps its real value when the read priced normally, so a
breakdown that had not landed by the time headers were built could advertise a
real total next to an all-zero split. The split is now reported as zero only
when the total agrees with it, and is otherwise left absent.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
The auto-router savings figure was computed only inside the spend-update
writer, downstream of where logging callbacks consume the standard logging
payload, so Datadog-style callbacks never received it. Compute it once in
the payload builder, stamp it as a top-level payload field beside
cost_breakdown, thread it into the spend log metadata, and have both
spend-writer call sites read the recorded value with recomputation as the
fallback for rows written before the field shipped. Internal sub-calls
(classifier, shadow eval) are never stamped, and a caller-forged metadata
value is discarded by the unconditional overwrite.
Resolves LIT-5973
The sweep ran unbounded whenever no config.yaml deployment was present,
deleting the day's sentinel rows for deployments absent from the run's own
view. A written charge records capacity that was reserved, so the only rows
a run may retract are the ones it can reassess: a deployment it scanned and
then declined to charge, because the window closed or the PTU config was
removed. It is now always bounded to the ids it scanned
The rollup read litellm.proxy.proxy_server.llm_router out of sys.modules, so a run
priced and swept whatever deployments anything else in the process had left on that
module. Under xdist the shard's module-to-worker assignment varies per run, which made
three rollup tests fail or pass on the same commit depending on ordering.
Callers now hand the router in, and the proxy's scheduled job passes its own.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Master-key auth stamps the stable alias litellm_proxy_master_key instead of the
raw key, so spend logs carry a readable, non-secret identifier for those rows.
The new redaction path only recognized sha256 and hashed-jwt shapes, so it
hashed that alias and broke continuity with every master-key row written
before this change. The alias joins the recognized non-secret values, still
behind the same provenance gate, so a caller who sends the alias string as
their own bearer token still gets it hashed.
The hashed-jwt branch trusted the value's shape alone, so a caller-supplied key in that shape was stored unhashed. Both pass-throughs now require the value to match the auth-time user_api_key_hash, and the shape check is a full match.
The spend-log helper no longer treats a 64-hex shape as proof a value was already hashed, so this case has to say where the hash came from. Reconciles the test that came in with #31799 against that change.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
* fix(ptu): hand the prune a plain delete filter the query builder can serialise
The bounded sweep built its predicate as a read-only mapping view, which the query
builder refuses to serialise, so the nightly job raised as soon as a config-declared
deployment was priced. The charges were already written by then, which is why the run
looked like it had produced its rows.
The in-memory table these tests run against accepts any mapping, so only a live run
caught it. A predicate builder now returns a plain dict and is asserted as one, and the
catch-up pass has a test covering a config-declared reservation.
* refactor(ptu): build the prune predicate in one shot
Both filter shapes are known upfront, so the bounded one is constructed
directly rather than by mutating a value already declared Final.
The catch-up test took two independent clock reads, which disagree across
UTC midnight; it now derives both the reservation start and the expected
last charged day from a single read, matching the three sibling tests.
* feat(ptu): accrue flat cost for PTU deployments declared in config.yaml
The flat-cost rollup reads deployments from LiteLLM_ProxyModelTable, and config.yaml
models never reach that table by design, so a PTU deployment declared there accrued no
flat cost at all while still billing its traffic per token. The provider bills the
reservation whichever file declared it.
The rollup now also reads the deployments the router holds that no database row owns,
identified by db_model, skipping the per-request credential clones that carry
original_model_id and reuse their source's PTU config under a fresh id. Registering such
a deployment zeroes its pricing, since reserved capacity already pays for the traffic it
serves, and leaving a rate unset falls back to the public cost map, which makes the double
charge the default rather than an opt-in.
The rules both halves apply now live in one module. The rollup's test for what it will
charge and the router's test for what to zero have to agree, or a deployment one accepts
and the other declines serves its traffic for free. That module also owns the fields the
write endpoints already zero, so the two paths cannot drift: tiered_pricing is emptied
rather than zeroed because its tiers outrank the rates beside them, the search context
table is written zeroed because an absent one means the provider default, and any further
rate the deployment itself declares is zeroed alongside the standing set.
The prune is bounded to the deployments a run scanned, but only for a run that priced a
config-declared deployment. Deciding a row is garbage on staleness alone stays correct
while every run derives its charges from the same table, so a database-only run sweeps
exactly as it did before; once one host's charges come from a file the others cannot read,
a row it never considered is not evidence of anything.
Behaviour change worth calling out: a zeroed deployment sorts ahead of an unpriced sibling
in QualityRouter's cost tiebreak, where an unset rate previously sorted last. Reserved
capacity really is the cheaper choice, but the ordering moves.
* refactor(ptu): drop a Final rebind and two redundant isinstance guards
The basedpyright budget rejected reassigning a Final in the datetime coercion and
two isinstance calls the router entry's own type already guarantees. Filtering the
built records rather than the raw entries removes both guards and leaves
_router_deployment as the single validator.
The flat-cost rollup reads deployments only from LiteLLM_ProxyModelTable, so a PTU
deployment declared in config.yaml never accrues flat cost. Those deployments live in
llm_router.model_list as plain dicts whose id sits in model_info rather than on the entry,
so they do not satisfy the shape _parse_ptu_model reads.
Adds a frozen record in that shape and a factory that maps a router entry onto it, leaving
_parse_ptu_model byte-identical so the existing cases stand as evidence of no behaviour
change. Nothing calls the factory yet; the caller lands with the loader union.
_decode_model_info also stops handing back valid JSON that is not an object. It decoded
a list or a scalar and returned it as a mapping, so the caller read fields off it and
raised, losing the whole run rather than the one bad deployment.
request_id is the primary key of LiteLLM_SpendLogs and the flush inserts with
skip_duplicates, so a spend log whose id already exists is dropped with no error
raised and a "processed 1 spend log" line still logged. Batch cost accounting
produced exactly such an id twice over, and on a proxy with message redaction
enabled no batch cost row could be written at all.
get_spend_logs_id derived the id by md5-hashing the response for two call types,
aretrieve_batch and acreate_file. Redaction makes that hash a constant:
perform_redaction returns the fixed {"text": "redacted-by-litellm"} placeholder
for any shape it cannot redact, which is what a batch object and a file body both
become, so every such row hashed to md5('{"text": "redacted-by-litellm"}') =
00fcbef15a3b0097e14b0ca016ed30a0 regardless of provider, user, or amount. The
first row to claim that id owned it and every later row was discarded. Verified
against a live proxy: four payloads spanning two providers and three distinct
spend values all computed that id, and the table held one acreate_file row dating
to 2025-05-25, the row that had claimed it.
Keying off the batch's own identity instead is necessary but not sufficient,
because creating a batch already writes an acreate_batch row under exactly that
id, so the cost row becomes a duplicate of the batch's own creation row. Also
verified live: after the hash was removed the poller computed and flushed a
batch's cost, and the only row carrying that id was the acreate_batch row from
when the batch was submitted.
The id now comes from the response's own id, then the standard logging payload's
id, then litellm_call_id, and a batch cost row is namespaced with a _batch_cost
suffix so it cannot collide with the creation row. The middle term is what keeps
this correct under redaction: that payload is built from the unredacted response,
so it still carries the batch id after redaction has flattened the body. Keying
the cost row to the batch rather than to the call also keeps accounting the same
batch twice collapsing to one row instead of billing it twice. Every other call
type still derives its key exactly as before.
Cost and usage themselves are unaffected by redaction: the token columns fall back
to the standard logging payload and spend comes from its response_cost, neither of
which redaction touches. generate_hash_from_response had no other caller and is
removed with it.
A deployment with PTU flat-cost attribution also billed every request per
token, so a team paid for reserved capacity and again for the traffic that
capacity serves. Nothing set the per-token price and an unset price falls
back to the public cost map, which made the double charge the default.
/model/new and /model/{id}/update now store zero for every pricing field the
cost map could otherwise fill, refuse a price the caller supplies alongside
PTU config with a 400 naming the field, zero a price already on the row
rather than rejecting later edits of unrelated fields, and drop the zeros
again when the PTU config goes.
A PTU deployment is no longer read as a free model by the budget checks,
which would have waived every budget for it.
Resolve the authorized own-user and permitted-team predicates once and add
regression coverage for explicit-user intersection, unfiltered team scope,
and team lookup failure fallback.
Co-Authored-By: Codex
Add a bounded spend-log user facet for the Request Logs picker and
intersect explicit user filters with the caller's own and permitted-team
scope.
Co-Authored-By: Codex
* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
LITELLM_ENABLE_PTU_COST_ATTRIBUTION, read through get_secret_bool and defaulting to
false, makes the whole PTU flat-cost feature inert unless an operator opts in. The
daily rollup cron is not registered at all, so no sentinel row is ever written;
/model/new and /model/{id}/update reject a request that carries any PTU model_info
field with a 400 naming the fields and the env var rather than dropping them; the
daily activity read path reports zero flat cost; and the model add and edit forms
hide the four PTU inputs.
The read gate lives where flat cost enters SpendMetrics rather than in the aggregated
SQL select. /team/daily/activity, the endpoint the Usage page reads, is served by the
paginated find_many path and never runs that query, so forcing the select to a
constant zero would have left the reporting surface that matters still showing flat
cost.
Sentinel row filtering is deliberately not gated. An operator can enable the flag,
accrue rows under the __ptu_flat_cost__ api_key, then disable it, and those rows stay
in LiteLLM_DailyTeamSpend; gating the filter too would surface the sentinel as a bogus
api_key and mint a provider bucket for its empty provider. Response fields keep their
shape and report 0.0, so typed clients are unaffected, and the migration and the
ModelInfo field declarations are untouched.
The write gate reads the incoming request rather than the merged deployment, so a
model configured during an earlier opt-in stays editable, and the edit form drops the
PTU keys from the payload instead of sending nulls that would clear stored config.
The dashboard reads the flag from a read-only enable_ptu_cost_attribution key on
/get/ui_settings, computed from the environment on every read. It is deliberately not
an allowlisted persisted setting, and PATCH /update/ui_settings rejects it with a 400,
so an admin cannot flip an env-gated feature from the UI.
Two review findings on the gate itself. The PTU clear loop now runs only when the
feature is enabled: the write gate rejects a value but lets an explicit null through,
and a client round-tripping a model_info blob sends the PTU keys as nulls, so a
disabled proxy would have quietly erased a billing configuration set up during an
earlier opt-in. Disabling pauses PTU rather than discarding its setup. And the
dashboard flag is re-read every thirty seconds instead of the hour the other UI settings
use, since those are persisted records while this one tracks the proxy process; a
restart that flips the variable would otherwise leave the model form offering inputs
the backend now rejects. The flag is polled rather than only marked stale, since a form
that stays mounted and focused never refetches on its own.
The read gate checks the row before the flag. It runs once per metric accumulation and a
record fans out across roughly a dozen breakdowns, while the flag reads through the secret
manager uncached, so consulting it for every accumulation put thousands of lookups on a
shared endpoint that made none before. Only a row actually carrying flat cost reaches it.
Add the daily rollup that reads PTU config off model deployments and writes flat
cost to LiteLLM_DailyTeamSpend. For each UTC day a deployment carrying ptu_count
and cost_per_ptu_per_hour accrues ptu_count * cost_per_ptu_per_hour * active_hours,
where active_hours is the overlap between the day and the optional
[ptu_effective_from, ptu_effective_to) window clamped to 24; a window opening at
23:00 charges one hour that day. Rows use a sentinel api_key so they stay
distinguishable from per-request rows and share the existing unique constraint,
and the write is idempotent so re-runs never double count.
The cron is registered at proxy startup and runs at 00:15 UTC. Pricing one day per
fire leaves two ways for a day to end up unpriced and stay that way: a window
backdated at configuration time, which no fire ever revisits, and a fire that is
missed or lands late, which the next one does not replay because the billed day
comes from the wall clock rather than the scheduled time. Both are silent, since
the failure alert only fires for a charge that was attempted. Each scheduled run
therefore follows the day's reconcile with a catch-up pass that prices the
(team, model, date) charges inside every declared window that carry no row yet,
bounded at the earliest ptu_effective_from and floored at
PTU_ROLLUP_MAX_BACKFILL_DAYS. It writes only what is missing: a day already priced
keeps the amount it was billed whatever the config says now, and it runs no prune,
so deciding a row is stale stays the single-day path's job. Zero-cost days write
nothing, which leaves an out-of-window day reconsidered each run rather than
recorded as done. A catch-up pass that fails cannot take the day's own result with
it, and an explicit target_date still means reconcile exactly that day.
The sentinel row keys on the deployment id, with the operator-facing name alongside it
in model_group, which sits outside the table's unique key. The name is what a usage view
displays, but a deployment can be renamed, and two runs holding config views from either
side of a rename then wrote the same day under two different keys, so nothing collided
and both charges survived. A multi-pod rig reproduced that as a permanent double charge
that no later run repaired. Keyed on the id both writes land on one key and the upsert
collapses them; when the rate changed too, last writer wins on the amount rather than
adding a row. Deployments sharing a public name inside a team therefore no longer need
collapsing into a single charge: each keys its own row, and the read path merges them
back under the shared name.
The read path that surfaces the amount lands in a follow-up PR.
The prune is the one destructive step, so it only runs when the pod took the cross-pod
lock. The upserts stay unguarded, since they are idempotent and no lock problem may cost
a day, but the delete compares a cutoff and an updated_at stamped on different hosts, and
a live rig showed a pod whose clock ran ten minutes ahead sweeping the charge a concurrent
pod had just written, leaving the day at zero. Its cutoff also allows
PTU_PRUNE_SKEW_GRACE_SECONDS of slack, which separates the two populations without
requiring clocks to agree: a stale row is hours old and a concurrently written one is
seconds old.
The catch-up deletes nothing. Removing a deployment or narrowing its window stops it
accruing new charges and leaves the days it was already billed for standing, since those
days were incurred and a usage view has to keep reporting them.
A deployment carrying no ptu_effective_from is skipped rather than treated as open ended.
The endpoints require a start, and substituting the cap floor for a missing one meant a
windowless deployment accrued the whole ninety day window on its first run, billing days
it did not exist while the result still reported a single row written.
Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one
conditional upsert at spend-write time, classifying each turn (same model, first
visit, return to tier, out of order) against the row's own columns so nothing is
read before the write. The upsert's placeholders and argument tuple both derive
from the transaction dataclass's own field order, so the SQL and the call site
cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped
by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's
cache interaction is derived once from its usage record (savings.py owns the
extraction; compute_savings_spend derives cache reads from usage_object itself),
hits are counted order-independently so the overall hit rate matches its covered
denominator, caller-chosen session ids are bounded before entering the primary
key, and a poisoned statement drops only its own session's remaining turns.
Return misses inside the recorded TTL are named for what the telemetry shows
(within_ttl) rather than a presumed cause, since a provider can evict early.
Savings ride each router's derived baseline by default, so the response carries
no deployment-wide baseline label. Rollup retention has its own
maximum_autorouter_session_retention_period setting, pattern-identical to the
spend-logs knob and running in the same cleanup job on its own cutoff. Every
drain trigger sizes the queues through one owner and the enqueue honors
disable_spend_logs beside the tool-usage queue it mirrors.
* feat(spend): derive a default auto-router savings baseline from the hardest tier
The savings driver shipped off by default: unless an operator names
litellm_settings.autorouter_savings_baseline_model, every auto-routed request
records $0.00 and the dashboard card never populates. Nobody discovers a knob
whose feature they have never seen work, so the default has to come from
somewhere the proxy already knows.
The router's own tier ladder is that place. Without a router a deployment runs
one model that can carry the hardest request it will see, so the derived
baseline is the priciest model in the hardest configured tier, REASONING when
present, otherwise the most severe tier the router actually defines. A cheap
tier is a choice the router made, not a ceiling it was bounded by.
An earlier draft of #35521 derived this per request and was deleted for it:
ranking candidates against the request that ran meant reading the request, and
every input shape it could take produced its own review finding. This
derivation is ranked against one fixed reference request instead, a cache-heavy
shape matching real auto-routed traffic, so it never reads the request at all.
Candidates still resolve through the router's deployments, so Azure base_model
and per-deployment pricing overrides rank correctly.
The deciding router records the result on its routing_decision, because one
model name can carry several tag-scoped routers with different tier ladders and
only the deciding instance knows which of them routed the request. The spend
writer's precedence is: configured baseline, then the recorded one, then off.
When the setting is present the router skips deriving entirely rather than
pricing candidates per decision only to be ignored.
Resolution never raises; an unresolvable baseline zeroes the driver instead of
failing a live request. Rows queued by a pod on the previous release carry no
recorded baseline and fall back to the configured setting, exactly as today.
The schema.d.ts regeneration also picks up the reminder_markers field that
UI-19232 (#35874) added without regenerating, so one hunk there is inherited
staleness rather than part of this change.
* fix(spend): cache the derived baseline, price it by deployment, keep it out of the routing preview
Three review findings on the derived baseline, addressed together because they
all sit on the same value's path from derivation to consumer.
Derivation walked and priced the hardest tier's whole pool inside a property
read on every routing decision, unbounded by pool size. The router now caches
the result per instance with a 30 second TTL, None results included, so the
hot path is a clock compare and a deployment edit still lands within a window
no operator watches closer than.
Ranking used each deployment's effective pricing but recorded only the model
name, so the spend writer priced the winning baseline at its public rate: a
hardest tier whose deployment carries a negotiated rate produced materially
wrong savings. The decision now also records savings_baseline_deployment_id
and the writer resolves it through Router.get_deployment_model_info, exactly
as the selected arm already does. The id is ignored whenever the configured
setting overrides the recorded baseline, since the setting names a model, not
a deployment.
/auto_router/test_routing returns the routing decision verbatim to team admins
while only authorizing the classifier and embedding models, so a derived
baseline would resolve another team's model-group alias into its backend
provider/model mapping and hand it to a caller never authorized for it. The
preview's throwaway router is built with derive_savings_baseline=False; its
decisions are never spend-tracked, so nothing is lost, and a source-pinning
test keeps the flag on the endpoint.
Also strips the explanatory comments this PR had added.
* refactor(spend): pin the derived baseline per router instance instead of a TTL
Creating or editing a router already rebuilds its ComplexityRouter instance,
through unregister and re-add on upsert and through the registry reset on a
full model_list load, so a value derived once per instance refreshes on
exactly the flows that can change it. That makes the TTL a solution to a
problem the rebuild lifecycle already solves, and it goes.
Derivation stays deferred to first use rather than running in __init__: during
a config load this router can be constructed before the deployments its tiers
name, and a baseline pinned at that moment would be empty for the process
lifetime.
The one behavior the TTL had that the pin does not: editing a tier deployment
without touching the router itself refreshed the baseline within a window.
That edit path rebuilds only the edited deployment's own strategies, so the
pin holds the old answer until the router is next saved or the config next
loads. A stale deployment id degrades to public-rate pricing rather than
failing, which is where every other unresolvable baseline already lands.
