Cherry-pick of merge commit b3882d8e43 (PRs #39321, #39562, #40107), which landed on litellm_internal_staging instead of main.
Co-authored-by: ojensen-berri <ojensen@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A policy attachment with default: true applies only when no non-default
attachment matches the request, so an opt-in guardrail policy replaces the
fallback one instead of running alongside it. Supported in config.yaml,
/policies/attachments, the Admin UI Attachments tab and the resolver
(matched_via is prefixed with default:).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
vLLM serves no /v1/files or /v1/batches, so a hosted_vllm deployment can never
host a batch. Batch inputs for such a deployment now land in a LiteLLM-owned
storage backend, the batch is executed line by line through the deployment's
own chat, completion, embedding, or responses route, and the batch plus its
output and error files are served back from the database under the creating key
Adds a persistent total_spend column to LiteLLM_VerificationToken and LiteLLM_DeletedVerificationToken, incremented in the same write as spend and left alone by budget resets. Surfaces it on /key/info, /key/list and the Admin UI Virtual Keys table and key detail view
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds a daily spend table without api_key or user_id, written atomically alongside
LiteLLM_DailyUserSpend from the batched writer, reconciled from history by a
scheduled job that advances a marker in LiteLLM_Config, and read by the key-free
arm of the aggregated usage query once the marker covers the requested range.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Roll request_duration_ms of successful, non-internal requests into the
daily spend tables as total_response_time_ms plus timed_requests, expose
both through the daily activity endpoints, and derive the average in the
Usage -> Model Activity view of the Admin UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds a nullable tpd_limit column and field to keys, teams, budgets and end users. The batch submission limiter swaps the per-minute RPM/TPM descriptor of any scope that has a tpd_limit for a token-only 24h descriptor, so batch traffic is budgeted per day while online traffic keeps the existing per-minute limits. The Admin UI exposes the field on key, team and budget create/edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keeps the base's rule that a non-admin id lookup matching no spend-log row answers 403, so the detail route never consults cold storage without an owner row
Adds object_permission.skills to keys and teams, enforces it on
/claude-code/marketplace.json?key=, /claude-code/plugins and
/claude-code/plugins/{name}, and exposes an Allowed Skills selector in
the key and team create/edit forms of the Admin UI
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LiteLLM_JWTKeyMapping_token_fkey was created ON DELETE RESTRICT, so deleting
a virtual key that a JWT mapping pointed at failed with a foreign key
violation on every deletion path (/key/delete, Admin UI, alias delete,
team and user cascades). Declaring onDelete: Cascade on the relation lets
the database clean the mapping up uniformly, so the next JWT call from that
identity re-registers against the newly created key.
Rebase of #33703 onto current staging.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The daily guardrail usage rollup stored billable units per counter but no
cost, so the usage endpoints could only report units. The Bedrock hook now
stamps guardrail_cost_by_unit next to guardrail_usage, the spend-log
aggregator sums it into a new nullable cost column on
LiteLLM_DailyGuardrailUsageUnits, and /guardrails/usage/overview and
/guardrails/usage/detail/{id} return cost, totalCost and cost_by_unit /
cost_by_team / cost_by_key alongside the existing unit breakdowns.
Cost is nullable on purpose. Rows written before this migration, and rows
whose hook had no pricing entry, read as null rather than $0, and a single
unpriced increment keeps that row's cost unknown instead of partial.
guardrail_cost and the spend/budget path are untouched.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
Success spend rows are keyed by the upstream provider response id, so the
x-litellm-call-id response header value never found them. Add a nullable
indexed litellm_call_id column to LiteLLM_SpendLogs, populate it at write
time, and widen every request_id lookup surface (/spend/logs,
/spend/logs/ui, request details, ownership check) to match either id.
Shadow eval jobs previously targeted only virtual keys, so deployments on
pure JWT auth (which present no key at all) could never sample their
traffic. Jobs now carry a typed (target_type, target_id) pair covering
keys, teams, and users; sampling matches the identity every request
resolves to at auth time, so team and user jobs cover JWT traffic with
no client changes.
Resolves LIT-6578
Every per-key spend read filters WHERE api_key = ... AND startTime in a
range (/spend/logs?api_key=..., the key-filtered UI logs page, external
billing readers), but LiteLLM_SpendLogs carries no api_key index, so each
such query scans every logged request on the instance. Composite with
startTime to match the query shape, mirroring the existing
(startTime, request_id) composite.
Measured on Postgres 18 with 1.6M spend rows: one key's 7-day SUM goes
from a 93.9ms parallel seq scan to a 0.7ms bitmap index scan (~140x);
a batch job reading per-key spend for 10k keys went from 20.5s to 1.7s.
