The per-guardrail status a scoped-out evaluation records is now skipped, matching the
skip_*_in_guardrail settings that cause it. Request-level rollup still maps it to not_run
so the StandardLoggingPayload status contract is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Custom code guardrails could only allow(), block(reason) or modify(). This adds flag(reason, metadata={}) which lets the request or response through unchanged and records a guardrail_flagged entry carrying the guardrail name, configured mode, evaluated input_type (request or response), reason and structured metadata. The new status is threaded through the request-level guardrail_status aggregation, the Guardrails Monitor rollup (flagged_count), Request Logs (action=flagged, most severe phase wins when a guardrail runs pre and post call) and the Request Logs detail view in the dashboard, which now renders FLAGGED with warning styling instead of falling into FAILED.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The detail endpoint now returns untracked_usage_units_by_team and
untracked_usage_units_by_key next to the cost breakdowns, and the By team and
By key tables show them in an Unpriced Units column, so a row that pairs its
total units with a partial cost says how many units that cost leaves out.
The overview comparator no longer treats a missing cost as zero: guardrails
with no known cost sort last in both directions instead of mixing in with
genuinely free ones.
Refs LIT-5652
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
A row's cost sums only the daily rows that carry a tracked cost, so it
silently under-reports whenever some rows are NULL (pre-migration days,
old pods mid-rollout, an unpriced counter). Both usage endpoints now
return the per-counter units behind those NULL rows next to the cost
(untrackedUsageUnits / totalUntrackedUsageUnits on the overview,
untracked_usage_units on the detail), so a partial cost is never mistaken
for a complete one and the reader can see exactly what it excludes
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The daily guardrail usage rollup stored billable units per counter but no
cost, so the usage endpoints could only report units. The Bedrock hook now
stamps guardrail_cost_by_unit next to guardrail_usage, the spend-log
aggregator sums it into a new nullable cost column on
LiteLLM_DailyGuardrailUsageUnits, and /guardrails/usage/overview and
/guardrails/usage/detail/{id} return cost, totalCost and cost_by_unit /
cost_by_team / cost_by_key alongside the existing unit breakdowns.
Cost is nullable on purpose. Rows written before this migration, and rows
whose hook had no pricing entry, read as null rather than $0, and a single
unpriced increment keeps that row's cost unknown instead of partial.
guardrail_cost and the spend/budget path are untouched.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
GET /guardrails/usage/overview and GET /guardrails/usage/detail/{id} 500ed on a
database that has not applied 20260817143646_add_daily_guardrail_usage_units yet
(pip installs on litellm-proxy-extras 0.4.86 with DISABLE_SCHEMA_UPDATE=true).
Both endpoints now return their metrics with empty units and log one warning
until the migration lands.
The flush and the usage endpoints summed units with a scan per distinct key,
quadratic in rows times keys; group sorted rows instead. Skip payloads without
a request_id like the metrics path, type the flush key as a NamedTuple, and drop
the (guardrail_id, date) index that the primary key already covers
The /guardrails/usage/{overview,detail,logs} endpoints resolved guardrails only
from the litellm_guardrailstable Prisma table, so guardrails defined in
config.yaml (which live only in IN_MEMORY_GUARDRAIL_HANDLER) were invisible:
detail 404'd, overview omitted them or rendered them as Custom/Guardrail
orphans, and logs missed their logical-name alias.
Add config-owned accessors (list_config_guardrails, get_config_guardrail_by_id)
to the in-memory handler and use them in the usage endpoints, mirroring the
union/fallback already used by list_guardrails_v2 and get_guardrail_info. Also
preserve guardrail_info when storing a config guardrail (type/description were
dropped at initialize time) and read the Prisma-row / dict / LitellmParams
shapes uniformly.
Resolves LIT-2529
