Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one
conditional upsert at spend-write time, classifying each turn (same model, first
visit, return to tier, out of order) against the row's own columns so nothing is
read before the write. The upsert's placeholders and argument tuple both derive
from the transaction dataclass's own field order, so the SQL and the call site
cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped
by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's
cache interaction is derived once from its usage record (savings.py owns the
extraction; compute_savings_spend derives cache reads from usage_object itself),
hits are counted order-independently so the overall hit rate matches its covered
denominator, caller-chosen session ids are bounded before entering the primary
key, and a poisoned statement drops only its own session's remaining turns.
Return misses inside the recorded TTL are named for what the telemetry shows
(within_ttl) rather than a presumed cause, since a provider can evict early.
Savings ride each router's derived baseline by default, so the response carries
no deployment-wide baseline label. Rollup retention has its own
maximum_autorouter_session_retention_period setting, pattern-identical to the
spend-logs knob and running in the same cleanup job on its own cutoff. Every
drain trigger sizes the queues through one owner and the enqueue honors
disable_spend_logs beside the tool-usage queue it mirrors.
GET /v1/tool/spend served the Cost Optimization card with two raw queries
over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load;
the totals query's driving scan was all of SpendLogs in the window. Both
per-request tables reach 1M+ rows at customer scale, so the card cost
O(traffic) per view and had to be capped at 30 days.
The index writer also mined proxy_server_request.tools, i.e. tools DECLARED
in the request body, attributing each request's full spend to tools that
never ran; and all non-MCP mining ran against payload fields that are '{}'
unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently
depended on a privacy setting.
Now the spend writer builds a ToolUsageTransaction at request time from
invoked tools only, resolved by the shared get_tool_calls_from_response
normalizer so every response surface (chat completions, Responses API,
Anthropic Messages) is covered; the tool registry's response arm delegates
to the same owner. Transactions queue beside the spend-log queue and the
flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup
(date, tool_name PK) in one transaction, retrying connection errors with
backoff (a failed batch commits nothing, so the retry cannot double-count)
and dropping the batch with an error log on anything else.
The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS
tools by spend via group_by and daily covers only those tools, so the
response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or
tool-name cardinality; the 30-day clamp is gone. total_spend is dropped
from the response; it was never rendered and its deduplicated semantics
are not computable from a rollup. Spend-log retention deliberately does
not touch the rollup, so tool spend history outlives per-request rows.
* catch DB_CONNECTION_ERROR_TYPES
* fix DB retry mechanism for SpendLog updates
* use DB_CONNECTION_ERROR_TYPES in auth checks
* fix exp back off for writing SpendLogs
* use _raise_failed_update_spend_exception to ensure errors print as NON blocking
* test_update_spend_logs_multiple_batches_with_failure
