feat: group Performance alerts by resource and day

This commit is contained in:
moe-berri 2026-11-05 13:20:47 +08:00 • committed by ishaan-j-berri
parent 549e263984
commit 949cc1e2ee
9 changed files with 1023 additions and 20 deletions

View file

@ -1,19 +1,40 @@
# Daily usage alerts
# Daily Performance alerts
Daily usage alerts compare a completed UTC day with earlier recorded days. They cover spend, request count, average latency, and failed-request count for the gateway, a key, a user, a team, or a model. Open an incident to inspect its saved comparison and follow its Activity link to the corresponding usage view.
Performance alerts compare a completed UTC day with earlier recorded days. They cover spend, request count, average latency, and failed-request count for the gateway, a key, a user, a team, or a model. The Performance tab groups alerts for the same resource and original UTC day. Open a resource to inspect its saved comparison and Activity. The API paths remain under `/usage`.
This feature requires a database. Apply the normal LiteLLM database migrations when deploying a version that includes usage alerts. The API and dashboard use the same persisted rules and incidents; no enterprise license is required.
## Read incidents
## Read resource groups
Set `LITELLM_API_BASE` to your gateway's base URL and `LITELLM_MANAGEMENT_KEY` to a management-capable credential. Keep the key in your environment.
```bash
curl "$LITELLM_API_BASE/usage/alerts?status=open&limit=50" \
curl "$LITELLM_API_BASE/usage/alert-groups?status=open&limit=50" \
-H "Authorization: Bearer $LITELLM_MANAGEMENT_KEY"
```
The response has `data`, `total`, `limit`, and `offset`. Each incident includes its scope, metric, direction, status, UTC day, observed value, baseline value, threshold, baseline date range, and `activity_path`. The Activity path is relative to the dashboard mount: for example, `/api-keys?...` is opened below the dashboard's `/ui` mount.
The response has `data`, `total`, `limit`, and `offset`. `total` counts resource/day groups, including when the requested offset is beyond the last page. Filters are `scope_type`, `scope_id`, `status`, `start_date`, and `end_date`. Status filters apply to individual incidents before grouping, so a group with open and resolved incidents can appear under either filter with the corresponding members. No default date cutoff hides older open incidents.
Each group includes its stable `id`, `scope_type`, exact `scope_id`, current `scope_label` when available, original UTC `day`, `incident_count`, and `activity_path`. Its `metrics` list contains up to four entries with a `metric`, `incident_count`, and the most recently detected matching `representative` incident. These representatives are summaries, not the entire group. Missing or deleted resources keep their original IDs. Names are current display labels, not historical evidence.
To inspect every matching rule, use `GET /usage/alerts` with the group's scope and `start_date` and `end_date` both set to its `day`. Add `metric` and `status` as needed and paginate that response. Rules can have different baseline windows and thresholds, so their comparisons must not be combined into an error rate or a cost-per-request estimate.
The group Activity link uses a representative's saved date range and alert context. The path is relative to the dashboard mount: for example, `/api-keys?...` is opened below the dashboard's `/ui` mount. A continuing episode retains its original day and evidence; the grouped row does not describe today's traffic.
Resolve or dismiss all matching incidents in one resource/day group with an expected current status:
```bash
curl -X PATCH "$LITELLM_API_BASE/usage/alert-groups" \
-H "Authorization: Bearer $LITELLM_MANAGEMENT_KEY" \
-H "Content-Type: application/json" \
-d '{"scope_type":"team","scope_id":"team-example","day":"2026-11-04","expected_status":"open","status":"resolved"}'
```
The response contains `updated_count`. Only incidents in that exact group that still have `expected_status` are updated atomically. Other statuses remain unchanged; repeating the same request returns zero after all matching incidents have changed. Use `scope_type: "deployment"` with `scope_id: null` for the gateway.
## Read individual incidents
`GET /usage/alerts` retains the individual-incident view. Each incident includes its scope, metric, direction, status, UTC day, observed value, baseline value, threshold, baseline date range, and `activity_path`.
`GET /usage/alerts/{alert_id}` returns one incident. List filters are `scope_type`, `scope_id`, `metric`, `status`, `start_date`, and `end_date`. Dates use `YYYY-MM-DD`; date filters include both endpoints. The maximum page size is 100.
@ -55,10 +76,12 @@ The evaluator runs against daily aggregates. An hourly scheduler does not make t
Use `scope_type: "deployment"` with no `scope_id` for the gateway. For `model`, use the underlying model identity recorded in Activity. For `key`, use its reporting ID, never the secret API key. Omitting `scope_id` for key, user, team, or model evaluates all entities of that type and requires a gateway administrator.
`GET /usage/alert-models?limit=20&search=example` returns paginated model identities recorded in the most recent 30 UTC days. Its `data` entries contain `scope_id` and `label`; `start_date` and `end_date` state the searched window. Pass `next_cursor` as `cursor` to read the next page. A null cursor ends the result. Search matches literal text, not SQL wildcards. This endpoint requires gateway admin or admin-viewer access. Configured aliases and deployment IDs are not reporting-model identities. An existing rule's saved model remains valid even when it falls outside the options window.
## Manage rules and incidents
- `GET /usage/alert-rules` lists rules; `GET /usage/alert-rules/{rule_id}` reads one.
- `PUT /usage/alert-rules/{rule_id}` replaces the rule configuration. Send the configuration fields from the create example, including any notification settings you want to retain. Set `enabled` to `false` to pause it.
- `GET /usage/alert-rules` lists rules; `GET /usage/alert-rules/{rule_id}` reads one. Use `kind=custom`, `kind=default`, or `kind=all` to choose which rules to list. Each rule exposes `is_default`. The list's `default_summary` counts total and enabled defaults within the requested scope, independent of `kind`, and reports evaluation-status counts among enabled defaults.
- `PUT /usage/alert-rules/{rule_id}` replaces the rule configuration. Send the configuration fields from the create example, including any notification settings you want to retain. Set `enabled` to `false` to pause it. Built-in rules keep their scope, metric, and direction; create a custom rule to change those fields. Their names, thresholds, notification settings, and enabled state remain editable.
- `DELETE /usage/alert-rules/{rule_id}` removes a rule from evaluation and preserves existing incidents. Deleted defaults stay deleted.
- `PATCH /usage/alerts/{alert_id}` with `{"status":"dismissed"}` dismisses an incident. `open` and `resolved` are also accepted.

View file

@ -3,10 +3,15 @@ from typing import Protocol
from litellm.types.proxy.usage_alerts import (
UsageAlert,
UsageAlertGroupList,
UsageAlertGroupUpdate,
UsageAlertGroupUpdateResult,
UsageAlertList,
UsageAlertMetric,
UsageAlertModelList,
UsageAlertRule,
UsageAlertRuleInput,
UsageAlertRuleKind,
UsageAlertRuleList,
UsageAlertScope,
UsageAlertStatus,
@ -23,7 +28,12 @@ class UsageAlertStore(Protocol):
async def delete_rule(self, rule_id: str) -> None: ...
async def list_rules(
self, scope_type: UsageAlertScope | None = None, scope_id: str | None = None, limit: int = 50, offset: int = 0
self,
scope_type: UsageAlertScope | None = None,
scope_id: str | None = None,
limit: int = 50,
offset: int = 0,
kind: UsageAlertRuleKind = "all",
) -> UsageAlertRuleList: ...
async def get_alert(self, alert_id: str) -> UsageAlert | None: ...
@ -42,6 +52,21 @@ class UsageAlertStore(Protocol):
async def set_status(self, alert_id: str, status: UsageAlertStatus) -> UsageAlert | None: ...
async def list_groups(
self,
scope_type: UsageAlertScope | None = None,
scope_id: str | None = None,
status: UsageAlertStatus | None = None,
start_date: date | None = None,
end_date: date | None = None,
limit: int = 50,
offset: int = 0,
) -> UsageAlertGroupList: ...
async def set_group_status(self, data: UsageAlertGroupUpdate) -> UsageAlertGroupUpdateResult: ...
async def list_reported_models(self, search: str | None, cursor: str | None, limit: int) -> UsageAlertModelList: ...
async def key_ownership(self, key_id: str) -> tuple[str | None, str | None] | None: ...
async def team_role(self, team_id: str, user_id: str) -> str | None: ...

View file

@ -579,6 +579,8 @@ class LiteLLMRoutes(enum.Enum):
)
llm_api_routes_without_mcp = sorted(set(llm_api_routes) - set(mcp_inference_routes))
info_routes = [
"/usage/alert-groups",
"/usage/alert-models",
"/usage/alert-rules",
"/usage/alert-rules/{rule_id}",
"/usage/alerts",
@ -650,6 +652,8 @@ class LiteLLMRoutes(enum.Enum):
management_routes = (
[
"/usage/alert-groups",
"/usage/alert-models",
"/usage/alert-rules",
"/usage/alert-rules/{rule_id}",
"/usage/alerts",
@ -864,6 +868,8 @@ class LiteLLMRoutes(enum.Enum):
)
usage_alert_routes = (
"/usage/alert-groups",
"/usage/alert-models",
"/usage/alert-rules",
"/usage/alert-rules/{rule_id}",
"/usage/alerts",

View file

@ -13,10 +13,15 @@ from litellm.repositories.daily_activity_buckets import api_key_from_bucket
from litellm.types.proxy.auth.user_roles import LitellmUserRoles
from litellm.types.proxy.usage_alerts import (
UsageAlert,
UsageAlertGroupList,
UsageAlertGroupUpdate,
UsageAlertGroupUpdateResult,
UsageAlertList,
UsageAlertMetric,
UsageAlertModelList,
UsageAlertRule,
UsageAlertRuleInput,
UsageAlertRuleKind,
UsageAlertRuleList,
UsageAlertScope,
UsageAlertStatus,
@ -93,10 +98,52 @@ async def list_usage_alert_rules(
scope_id: ScopeId = None,
limit: Limit = 50,
offset: Offset = 0,
kind: UsageAlertRuleKind = "all",
) -> UsageAlertRuleList:
store: Final = require_repository(repository)
await require_scope(user, store, scope_type, scope_id, write=False)
return await store.list_rules(scope_type, scope_id, limit, offset)
return await store.list_rules(scope_type, scope_id, limit, offset, kind)
@router.get("/alert-groups", response_model=UsageAlertGroupList)
async def list_usage_alert_groups(
user: User,
repository: Repository,
scope_type: UsageAlertScope | None = None,
scope_id: ScopeId = None,
status: UsageAlertStatus | None = None,
start_date: date | None = None,
end_date: date | None = None,
limit: Limit = 50,
offset: Offset = 0,
) -> UsageAlertGroupList:
store: Final = require_repository(repository)
await require_scope(user, store, scope_type, scope_id, write=False)
if start_date is not None and end_date is not None and start_date > end_date:
raise HTTPException(status_code=422, detail="start_date must not be after end_date")
return await store.list_groups(scope_type, scope_id, status, start_date, end_date, limit, offset)
@router.patch("/alert-groups", response_model=UsageAlertGroupUpdateResult)
async def update_usage_alert_group(
data: UsageAlertGroupUpdate, user: User, repository: Repository
) -> UsageAlertGroupUpdateResult:
store: Final = require_repository(repository)
await require_scope(user, store, data.scope_type, data.scope_id, write=True)
return await store.set_group_status(data)
@router.get("/alert-models", response_model=UsageAlertModelList)
async def list_usage_alert_models(
user: User,
repository: Repository,
search: ScopeId = None,
cursor: ScopeId = None,
limit: Limit = 20,
) -> UsageAlertModelList:
store: Final = require_repository(repository)
await require_scope(user, store, "model", None, write=False)
return await store.list_reported_models(search, cursor, limit)
@router.post("/alert-rules", response_model=UsageAlertRule, status_code=201)
@ -129,6 +176,15 @@ async def update_usage_alert_rule(
if rule is None:
raise HTTPException(status_code=404, detail="Alert rule not found")
await require_scope(user, store, rule.scope_type, rule.scope_id, write=True)
if rule.is_default and (rule.scope_type, rule.scope_id, rule.metric, rule.direction) != (
data.scope_type,
data.scope_id,
data.metric,
data.direction,
):
raise HTTPException(
status_code=422, detail="Create a custom rule to change a built-in rule's scope, metric, or direction"
)
await require_scope(user, store, data.scope_type, data.scope_id, write=True)
saved: Final = await store.save_rule(data, user.user_id, existing=rule)
if saved is None:

View file

@ -3,6 +3,8 @@
import json
from collections.abc import Mapping
from datetime import date, datetime, timedelta, timezone
from itertools import product
from types import MappingProxyType
from typing import (
TYPE_CHECKING,
Final,
@ -14,16 +16,26 @@ from uuid import NAMESPACE_URL, uuid4, uuid5
from pydantic import BaseModel, TypeAdapter
from litellm.repositories.daily_activity_buckets import api_key_from_bucket
from litellm.types.proxy.usage_alerts import (
UsageAlert,
UsageAlertChannel,
UsageAlertDefaultSummary,
UsageAlertEvaluationStatus,
UsageAlertGroup,
UsageAlertGroupList,
UsageAlertGroupMetric,
UsageAlertGroupUpdate,
UsageAlertGroupUpdateResult,
UsageAlertList,
UsageAlertMetric,
UsageAlertModelList,
UsageAlertModelOption,
UsageAlertNotification,
UsageAlertObservation,
UsageAlertRule,
UsageAlertRuleInput,
UsageAlertRuleKind,
UsageAlertRuleList,
UsageAlertScope,
UsageAlertStatus,
@ -45,6 +57,37 @@ class _Count(BaseModel, frozen=True):
count: int
class _RuleCounts(_Count, frozen=True):
default_summary: UsageAlertDefaultSummary
class _GroupRow(BaseModel, frozen=True):
scope_type: UsageAlertScope
scope_id: str | None
day: date
incident_count: int
metrics: tuple[UsageAlertGroupMetric, ...]
class _GroupPage(BaseModel, frozen=True):
total: int
data: tuple[_GroupRow, ...]
class _ResourceLabel(BaseModel, frozen=True):
scope_type: UsageAlertScope
scope_id: str | None
label: str | None
class _ModelIdentity(BaseModel, frozen=True):
model: str
class _Identifier(BaseModel, frozen=True):
id: str
class _Pending(_Record, frozen=True):
rule: Mapping[str, object]
notified_channels: tuple[UsageAlertChannel, ...]
@ -55,6 +98,20 @@ class _Ownership(BaseModel, frozen=True):
team_id: str | None
_DEFAULT_RULE_IDS: Final = frozenset(
str(uuid5(NAMESPACE_URL, f"litellm:usage-alert:{scope}:{metric}:{direction}"))
for scope, metric, direction in product(
("deployment", "key", "user", "team", "model"),
("spend", "requests", "latency", "errors"),
("increase", "decrease"),
)
)
def _rule(record: Mapping[str, object]) -> UsageAlertRule:
return UsageAlertRule.model_validate({**record, "is_default": str(record.get("id")) in _DEFAULT_RULE_IDS})
def activity_path(
scope: UsageAlertScope, entity_id: str | None, day: date, alert_id: str, baseline_days: int = 7
) -> str:
@ -151,7 +208,7 @@ class UsageAlertsRepository:
'SELECT record FROM "LiteLLM_UsageAlertRule" WHERE id=$1 AND NOT deleted', rule_id
)
)
return UsageAlertRule.model_validate(rows[0].record) if rows else None
return _rule(rows[0].record) if rows else None
async def save_rule(
self, data: UsageAlertRuleInput, created_by: str | None, existing: UsageAlertRule | None = None
@ -177,7 +234,7 @@ class UsageAlertsRepository:
rule.model_dump_json(),
)
saved: Final = TypeAdapter(tuple[_Record, ...]).validate_python(rows)
return UsageAlertRule.model_validate(saved[0].record) if saved else None
return _rule(saved[0].record) if saved else None
async def delete_rule(self, rule_id: str) -> None:
await self.database.query_raw(
@ -185,28 +242,46 @@ class UsageAlertsRepository:
)
async def list_rules(
self, scope_type: UsageAlertScope | None = None, scope_id: str | None = None, limit: int = 50, offset: int = 0
self,
scope_type: UsageAlertScope | None = None,
scope_id: str | None = None,
limit: int = 50,
offset: int = 0,
kind: UsageAlertRuleKind = "all",
) -> UsageAlertRuleList:
args: Final = (scope_type, scope_id)
args: Final = (scope_type, scope_id, json.dumps(tuple(sorted(_DEFAULT_RULE_IDS))), kind)
where: Final = "NOT deleted AND ($1::text IS NULL OR scope_type=$1) AND ($2::text IS NULL OR scope_id=$2)"
count: Final = TypeAdapter(tuple[_Count, ...]).validate_python(
builtin: Final = "id IN (SELECT jsonb_array_elements_text($3::jsonb))"
count: Final = TypeAdapter(tuple[_RuleCounts, ...]).validate_python(
await self.database.query_raw(
f'SELECT count(*)::int AS count FROM "LiteLLM_UsageAlertRule" WHERE {where}', *args
f"""WITH scoped AS (
SELECT enabled,record,({builtin}) AS builtin FROM "LiteLLM_UsageAlertRule" WHERE {where}
) SELECT count(*) FILTER (WHERE $4='all' OR builtin=($4='default'))::int AS count,
jsonb_build_object('total',count(*) FILTER (WHERE builtin),
'enabled',count(*) FILTER (WHERE builtin AND enabled),
'evaluation_status_counts',jsonb_build_object(
'pending',count(*) FILTER (WHERE builtin AND enabled AND COALESCE(record->>'evaluation_status','pending')='pending'),
'complete',count(*) FILTER (WHERE builtin AND enabled AND record->>'evaluation_status'='complete'),
'insufficient_coverage',count(*) FILTER (WHERE builtin AND enabled AND record->>'evaluation_status'='insufficient_coverage'),
'insufficient_history',count(*) FILTER (WHERE builtin AND enabled AND record->>'evaluation_status'='insufficient_history')
)) AS default_summary FROM scoped""",
*args,
)
)
rows: Final = TypeAdapter(tuple[_Record, ...]).validate_python(
await self.database.query_raw(
f'SELECT record FROM "LiteLLM_UsageAlertRule" WHERE {where} ORDER BY id LIMIT $3 OFFSET $4',
f"SELECT record FROM \"LiteLLM_UsageAlertRule\" WHERE {where} AND ($4='all' OR ({builtin})=($4='default')) ORDER BY id LIMIT $5 OFFSET $6",
*args,
limit,
offset,
)
)
return UsageAlertRuleList(
data=tuple(UsageAlertRule.model_validate(row.record) for row in rows),
data=tuple(_rule(row.record) for row in rows),
total=count[0].count,
limit=limit,
offset=offset,
default_summary=count[0].default_summary,
)
async def list_enabled_rules(self, limit: int = 100, offset: int = 0) -> tuple[UsageAlertRule, ...]:
@ -275,6 +350,165 @@ class UsageAlertsRepository:
)
return UsageAlert.model_validate(rows[0].record) if rows else None
async def list_groups(
self,
scope_type: UsageAlertScope | None = None,
scope_id: str | None = None,
status: UsageAlertStatus | None = None,
start_date: date | None = None,
end_date: date | None = None,
limit: int = 50,
offset: int = 0,
) -> UsageAlertGroupList:
rows: Final = TypeAdapter(tuple[_GroupPage, ...]).validate_python(
await self.database.query_raw(
"""WITH groups AS MATERIALIZED (
SELECT scope_type,scope_id,day,count(*)::int AS incident_count FROM "LiteLLM_UsageAlertIncident"
WHERE ($1::text IS NULL OR scope_type=$1) AND ($2::text IS NULL OR scope_id=$2)
AND ($3::text IS NULL OR status=$3) AND ($4::text IS NULL OR day >= $4)
AND ($5::text IS NULL OR day <= $5)
GROUP BY scope_type,scope_id,day
), page AS MATERIALIZED (
SELECT * FROM groups ORDER BY day DESC,scope_type,scope_id NULLS FIRST LIMIT $6 OFFSET $7
), selected AS (
SELECT i.* FROM page p JOIN "LiteLLM_UsageAlertIncident" i
ON i.scope_type=p.scope_type AND i.scope_id=p.scope_id AND i.day=p.day
WHERE ($3::text IS NULL OR i.status=$3)
UNION ALL
SELECT i.* FROM page p JOIN "LiteLLM_UsageAlertIncident" i
ON i.scope_type=p.scope_type AND i.scope_id IS NULL AND i.day=p.day
WHERE p.scope_id IS NULL AND ($3::text IS NULL OR i.status=$3)
), ranked AS (
SELECT scope_type,scope_id,day,metric,record,
count(*) OVER (PARTITION BY scope_type,scope_id,day,metric)::int AS incident_count,
row_number() OVER (PARTITION BY scope_type,scope_id,day,metric
ORDER BY record->>'first_seen' DESC,id) AS rank
FROM selected
), packed AS (
SELECT p.scope_type,p.scope_id,p.day,p.incident_count,
jsonb_agg(jsonb_build_object('metric',r.metric,'incident_count',r.incident_count,
'representative',r.record) ORDER BY CASE r.metric
WHEN 'requests' THEN 0 WHEN 'spend' THEN 1 WHEN 'errors' THEN 2 ELSE 3 END) AS metrics
FROM page p JOIN ranked r ON p.scope_type=r.scope_type
AND p.scope_id IS NOT DISTINCT FROM r.scope_id AND p.day=r.day AND r.rank=1
GROUP BY p.scope_type,p.scope_id,p.day,p.incident_count
) SELECT (SELECT count(*)::int FROM groups) AS total,
COALESCE((SELECT jsonb_agg(to_jsonb(packed) ORDER BY day DESC,scope_type,scope_id NULLS FIRST)
FROM packed),'[]'::jsonb) AS data""",
scope_type,
scope_id,
status,
start_date.isoformat() if start_date else None,
end_date.isoformat() if end_date else None,
limit,
offset,
)
)
page: Final = rows[0]
labels: Final[Mapping[tuple[UsageAlertScope, str | None], str | None]] = (
await self._group_labels(page.data) if page.data else {}
)
return UsageAlertGroupList(
data=tuple(
UsageAlertGroup(
id=str(
uuid5(
NAMESPACE_URL,
json.dumps(
["litellm:performance-group", row.scope_type, row.scope_id, row.day.isoformat()]
),
)
),
scope_type=row.scope_type,
scope_id=row.scope_id,
scope_label=labels.get((row.scope_type, row.scope_id)),
day=row.day,
incident_count=row.incident_count,
metrics=row.metrics,
activity_path=row.metrics[0].representative.activity_path,
)
for row in page.data
),
total=page.total,
limit=limit,
offset=offset,
)
async def _group_labels(
self, groups: tuple[_GroupRow, ...]
) -> Mapping[tuple[UsageAlertScope, str | None], str | None]:
rows: Final = TypeAdapter(tuple[_ResourceLabel, ...]).validate_python(
await self.database.query_raw(
"""WITH scopes AS (
SELECT DISTINCT scope_type,scope_id,key_id FROM jsonb_to_recordset($1::jsonb)
AS x(scope_type text,scope_id text,key_id text)
) SELECT s.scope_type,s.scope_id,CASE s.scope_type
WHEN 'key' THEN COALESCE(NULLIF(k.key_alias,''),NULLIF(k.key_name,''))
WHEN 'team' THEN NULLIF(t.team_alias,'')
WHEN 'user' THEN COALESCE(NULLIF(u.user_alias,''),NULLIF(u.user_email,''))
WHEN 'model' THEN s.scope_id WHEN 'deployment' THEN 'Gateway' END AS label
FROM scopes s LEFT JOIN "LiteLLM_VerificationToken" k ON s.scope_type='key' AND k.token=s.key_id
LEFT JOIN "LiteLLM_TeamTable" t ON s.scope_type='team' AND t.team_id=s.scope_id
LEFT JOIN "LiteLLM_UserTable" u ON s.scope_type='user' AND u.user_id=s.scope_id""",
json.dumps(
tuple(
{
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"key_id": api_key_from_bucket(row.scope_id)
if row.scope_type == "key" and row.scope_id
else None,
}
for row in groups
)
),
)
)
return MappingProxyType({(row.scope_type, row.scope_id): row.label for row in rows})
async def set_group_status(self, data: UsageAlertGroupUpdate) -> UsageAlertGroupUpdateResult:
rows: Final = TypeAdapter(tuple[_Count, ...]).validate_python(
await self.database.query_raw(
"""WITH changed AS (
UPDATE "LiteLLM_UsageAlertIncident" SET status=$5,
record=jsonb_set(record,'{status}',to_jsonb($5::text)),updated_at=CURRENT_TIMESTAMP
WHERE scope_type=$1 AND scope_id IS NOT DISTINCT FROM $2::text AND day=$3 AND status=$4
RETURNING id
) SELECT count(*)::int AS count FROM changed""",
data.scope_type,
data.scope_id,
data.day.isoformat(),
data.expected_status,
data.status,
)
)
return UsageAlertGroupUpdateResult(updated_count=rows[0].count)
async def list_reported_models(self, search: str | None, cursor: str | None, limit: int) -> UsageAlertModelList:
today: Final = datetime.now(timezone.utc).date()
rows: Final = TypeAdapter(tuple[_ModelIdentity, ...]).validate_python(
await self.database.query_raw(
"""SELECT DISTINCT model FROM "LiteLLM_DailyUserSpend"
WHERE date >= $1 AND date <= $2 AND model IS NOT NULL AND model <> ''
AND ($3::text IS NULL OR strpos(lower(model),lower($3))>0)
AND ($4::text IS NULL OR model>$4)
ORDER BY model LIMIT $5""",
(today - timedelta(days=29)).isoformat(),
today.isoformat(),
search,
cursor,
limit + 1,
)
)
selected: Final = rows[:limit]
return UsageAlertModelList(
data=tuple(UsageAlertModelOption(scope_id=row.model, label=row.model) for row in selected),
limit=limit,
next_cursor=selected[-1].model if len(rows) > limit else None,
start_date=today - timedelta(days=29),
end_date=today,
)
async def key_ownership(self, key_id: str) -> tuple[str | None, str | None] | None:
rows: Final = TypeAdapter(tuple[_Ownership, ...]).validate_python(
await self.database.query_raw(

View file

@ -11,6 +11,7 @@ UsageAlertDirection: TypeAlias = Literal["increase", "decrease"]
UsageAlertStatus: TypeAlias = Literal["open", "resolved", "dismissed"]
UsageAlertChannel: TypeAlias = Literal["email", "slack"]
UsageAlertEvaluationStatus: TypeAlias = Literal["pending", "complete", "insufficient_coverage", "insufficient_history"]
UsageAlertRuleKind: TypeAlias = Literal["all", "default", "custom"]
class UsageAlertRuleInput(BaseModel):
@ -60,6 +61,7 @@ class UsageAlertRule(UsageAlertRuleInput):
created_by: str | None
last_evaluated_day: date | None = None
evaluation_status: UsageAlertEvaluationStatus = "pending"
is_default: bool = False
class UsageAlert(BaseModel):
@ -90,11 +92,25 @@ class UsageAlertUpdate(BaseModel):
status: UsageAlertStatus
class UsageAlertEvaluationCounts(BaseModel, frozen=True):
pending: int = 0
complete: int = 0
insufficient_coverage: int = 0
insufficient_history: int = 0
class UsageAlertDefaultSummary(BaseModel, frozen=True):
total: int = 0
enabled: int = 0
evaluation_status_counts: UsageAlertEvaluationCounts = Field(default_factory=UsageAlertEvaluationCounts)
class UsageAlertRuleList(BaseModel, frozen=True):
data: tuple[UsageAlertRule, ...]
total: int
limit: int
offset: int
default_summary: UsageAlertDefaultSummary = Field(default_factory=UsageAlertDefaultSummary)
class UsageAlertList(BaseModel, frozen=True):
@ -104,6 +120,64 @@ class UsageAlertList(BaseModel, frozen=True):
offset: int
class UsageAlertGroupMetric(BaseModel, frozen=True):
metric: UsageAlertMetric
incident_count: int
representative: UsageAlert
class UsageAlertGroup(BaseModel, frozen=True):
id: str
scope_type: UsageAlertScope
scope_id: str | None
scope_label: str | None
day: date
incident_count: int
metrics: tuple[UsageAlertGroupMetric, ...] = Field(max_length=4)
activity_path: str
class UsageAlertGroupList(BaseModel, frozen=True):
data: tuple[UsageAlertGroup, ...]
total: int
limit: int
offset: int
class UsageAlertGroupUpdate(BaseModel, frozen=True):
model_config = ConfigDict(extra="forbid")
scope_type: UsageAlertScope
scope_id: str | None = Field(default=None, min_length=1, max_length=1024)
day: date
expected_status: UsageAlertStatus
status: UsageAlertStatus
@model_validator(mode="after")
def validate_group(self) -> Self:
if (self.scope_type == "deployment") != (self.scope_id is None):
raise ValueError("Specify a resource ID except for deployment alerts")
if self.expected_status == self.status:
raise ValueError("The new status must differ from the expected status")
return self
class UsageAlertGroupUpdateResult(BaseModel, frozen=True):
updated_count: int
class UsageAlertModelOption(BaseModel, frozen=True):
scope_id: str
label: str
class UsageAlertModelList(BaseModel, frozen=True):
data: tuple[UsageAlertModelOption, ...]
limit: int
next_cursor: str | None
start_date: date
end_date: date
class UsageAlertNotification(BaseModel):
model_config = ConfigDict(frozen=True)
incident: UsageAlert

View file

@ -1,13 +1,21 @@
from datetime import date
from datetime import date, timedelta
from hashlib import sha256
from typing import Final
from uuid import uuid4
import pytest
from integration._support.actors import ManagementDatabase
from integration._support.client import Gateway, string_value
from litellm.repositories.usage_alerts import UsageAlertsRepository
from litellm.types.proxy.usage_alerts import UsageAlertObservation, UsageAlertRule, UsageAlertRuleInput
from litellm.types.proxy.usage_alerts import (
UsageAlertGroupList,
UsageAlertMetric,
UsageAlertModelList,
UsageAlertObservation,
UsageAlertRule,
UsageAlertRuleInput,
)
def test_scoped_rules_use_real_management_auth_and_preserve_read_only_roles(gateway: Gateway) -> None:
@ -252,3 +260,277 @@ async def test_rule_identity_edits_preserve_old_incident_and_filter_consistency(
assert resolved is not None and resolved.status == "resolved"
finally:
await repository.delete_rule(rule.id)
@pytest.mark.asyncio(loop_scope="session")
async def test_grouped_http_pages_preserve_all_metrics_history_and_scoped_actions(
gateway: Gateway, db: ManagementDatabase
) -> None:
repository: Final = UsageAlertsRepository.from_prisma(db.prisma)
day: Final = date(2026, 11, 4)
with gateway.scenario() as scenario:
owner: Final = scenario.user(user_role="internal_user", user_alias="Checkout operator")
outsider: Final = scenario.user(user_role="internal_user")
viewer: Final = scenario.user(user_role="proxy_admin_viewer")
owner_key: Final = scenario.key(user_id=owner, key_type="management")
outsider_key: Final = scenario.key(user_id=outsider, key_type="management")
viewer_key: Final = scenario.key(user_id=viewer, key_type="read_only")
inference_key: Final = scenario.key(user_id=owner, key_type="llm_api")
metrics: Final[tuple[UsageAlertMetric, ...]] = (*(("requests",) * 27), "spend", "errors", "latency")
rules: Final = tuple(
[
await repository.save_rule(
UsageAlertRuleInput(
name=f"Group rule {index}",
scope_type="user",
scope_id=owner,
metric=metric,
direction="increase",
baseline_days=7 if index % 2 else 14,
),
owner,
)
for index, metric in enumerate(metrics)
]
)
assert all(rule is not None for rule in rules)
incidents: Final = tuple(
[
(
await repository.record_observations(
rule,
(
UsageAlertObservation(
entity_id=owner,
day=day,
observed=420,
baseline=0 if index == 0 else 120,
threshold=100 if index == 0 else 360,
breached=True,
),
),
)
)[0]
for index, rule in enumerate(rules)
if rule is not None
]
)
old_rule: Final = await repository.save_rule(
UsageAlertRuleInput(
name="Earlier day", scope_type="user", scope_id=owner, metric="requests", direction="increase"
),
owner,
)
assert old_rule is not None
old: Final = (
await repository.record_observations(
old_rule,
(
UsageAlertObservation(
entity_id=owner,
day=day - timedelta(days=1),
observed=420,
baseline=120,
threshold=360,
breached=True,
),
),
)
)[0]
other_rule: Final = await repository.save_rule(
UsageAlertRuleInput(
name="Other resource", scope_type="user", scope_id=outsider, metric="requests", direction="increase"
),
outsider,
)
assert other_rule is not None
other: Final = (
await repository.record_observations(
other_rule,
(
UsageAlertObservation(
entity_id=outsider,
day=day,
observed=420,
baseline=120,
threshold=360,
breached=True,
),
),
)
)[0]
params: Final = {"scope_type": "user", "scope_id": owner, "status": "open", "limit": "1"}
try:
response: Final = gateway.request("GET", "/usage/alert-groups", key=owner_key, params=params)
assert response.status_code == 200, response.text
page: Final = UsageAlertGroupList.model_validate(response.json())
assert page.total == 2 and len(page.data) == 1
group: Final = page.data[0]
assert group.day == day and group.scope_label == "Checkout operator"
assert group.incident_count == 30 and len(group.metrics) == 4
assert {metric.metric: metric.incident_count for metric in group.metrics} == {
"requests": 27,
"spend": 1,
"errors": 1,
"latency": 1,
}
assert all(metric.representative.id in {incident.id for incident in incidents} for metric in group.metrics)
assert group.activity_path in {incident.activity_path for incident in incidents}
second: Final = UsageAlertGroupList.model_validate(
gateway.request("GET", "/usage/alert-groups", key=owner_key, params={**params, "offset": "1"}).json()
)
assert second.total == 2 and second.data[0].day == old.day
empty: Final = UsageAlertGroupList.model_validate(
gateway.request("GET", "/usage/alert-groups", key=owner_key, params={**params, "offset": "2"}).json()
)
assert empty.total == 2 and empty.data == ()
history: Final = gateway.request(
"GET",
"/usage/alerts",
key=owner_key,
params={**params, "start_date": day.isoformat(), "end_date": day.isoformat(), "metric": "requests"},
)
assert history.status_code == 200 and history.json()["total"] == 27
await repository.set_status(incidents[0].id, "dismissed")
body: Final = {
"scope_type": "user",
"scope_id": owner,
"day": day.isoformat(),
"expected_status": "open",
"status": "resolved",
}
for forbidden in (outsider_key, inference_key):
assert gateway.request("GET", "/usage/alert-groups", key=forbidden, params=params).status_code == 403
for forbidden in (outsider_key, inference_key, viewer_key):
assert gateway.request("PATCH", "/usage/alert-groups", body, key=forbidden).status_code == 403
assert gateway.request("GET", "/usage/alert-groups", key=viewer_key, params=params).status_code == 200
updated: Final = gateway.request("PATCH", "/usage/alert-groups", body, key=owner_key)
assert updated.status_code == 200 and updated.json()["updated_count"] == 29
assert gateway.request("PATCH", "/usage/alert-groups", body, key=owner_key).json()["updated_count"] == 0
assert await repository.get_alert(old.id) == old
assert await repository.get_alert(other.id) == other
dismissed: Final = await repository.get_alert(incidents[0].id)
assert dismissed is not None and dismissed.status == "dismissed" and dismissed.baseline_value == 0
resolved: Final = await repository.get_alert(incidents[-1].id)
assert resolved == incidents[-1].model_copy(update={"status": "resolved"})
assert (
gateway.request(
"GET", "/usage/alert-groups", params={"scope_type": "user", "scope_id": "x' OR 1=1 --"}
).json()["total"]
== 0
)
assert (
gateway.request(
"GET", "/usage/alert-groups", params={"start_date": "2026-11-05", "end_date": "2026-11-04"}
).status_code
== 422
)
finally:
for rule in (*rules, old_rule, other_rule):
if rule is not None:
await repository.delete_rule(rule.id)
deleted: Final = await repository.list_groups(scope_type="user", scope_id=owner)
assert deleted.total == 2 and all(group.scope_label is None and group.scope_id == owner for group in deleted.data)
@pytest.mark.asyncio(loop_scope="session")
async def test_default_rule_summary_and_kind_filters_do_not_require_loading_every_rule(
gateway: Gateway, db: ManagementDatabase
) -> None:
repository: Final = UsageAlertsRepository.from_prisma(db.prisma)
default: Final = UsageAlertRuleInput(
name="Built-in daily model requests", scope_type="model", metric="requests", direction="increase"
)
await repository.ensure_default_rules((default,))
custom: Final = await repository.save_rule(
default.model_copy(update={"scope_id": f"model-{uuid4().hex}"}), "qa-admin"
)
assert custom is not None
try:
all_rules: Final = await repository.list_rules(scope_type="model", limit=1)
builtins: Final = await repository.list_rules(scope_type="model", limit=1, kind="default")
customs: Final = await repository.list_rules(scope_type="model", limit=1, kind="custom")
assert builtins.total >= 1 and all(rule.is_default for rule in builtins.data)
assert customs.total >= 1 and all(not rule.is_default for rule in customs.data)
assert all_rules.total == builtins.total + customs.total
assert all_rules.default_summary == builtins.default_summary == customs.default_summary
assert builtins.default_summary.total == builtins.total
assert (
sum(builtins.default_summary.evaluation_status_counts.model_dump().values())
== builtins.default_summary.enabled
)
assert (await repository.list_rules(scope_type="model", kind="default", offset=builtins.total)).data == ()
stored: Final = await repository.get_rule(builtins.data[0].id)
assert stored is not None and stored.is_default
filtered: Final = gateway.request(
"GET", "/usage/alert-rules", params={"scope_type": "model", "kind": "custom", "limit": "1"}
)
assert filtered.status_code == 200 and filtered.json()["total"] == customs.total
assert all(not rule["is_default"] for rule in filtered.json()["data"])
configuration: Final = stored.model_dump(mode="json", include=frozenset(UsageAlertRuleInput.model_fields))
path: Final = f"/usage/alert-rules/{stored.id}"
for change in (
{"scope_type": "key"},
{"scope_id": "model-example"},
{"metric": "requests" if stored.metric != "requests" else "spend"},
{"direction": "increase" if stored.direction == "decrease" else "decrease"},
):
assert gateway.request("PUT", path, {**configuration, **change}).status_code == 422
assert await repository.get_rule(stored.id) == stored
try:
updated: Final = gateway.request("PUT", path, {**configuration, "min_value": stored.min_value + 1})
assert updated.status_code == 200 and updated.json()["is_default"] is True
assert updated.json()["min_value"] == stored.min_value + 1
finally:
assert gateway.request("PUT", path, configuration).status_code == 200
custom_input: Final = custom.model_dump(mode="json", include=frozenset(UsageAlertRuleInput.model_fields))
assert (
gateway.request("PUT", f"/usage/alert-rules/{custom.id}", {**custom_input, "metric": "latency"}).status_code
== 200
)
finally:
await repository.delete_rule(custom.id)
@pytest.mark.asyncio(loop_scope="session")
async def test_reported_model_search_preserves_exact_identity_and_cursor(
gateway: Gateway, db: ManagementDatabase
) -> None:
repository: Final = UsageAlertsRepository.from_prisma(db.prisma)
prefix: Final = f"reported-{uuid4().hex}"
initial: Final = await repository.list_reported_models(prefix, None, 1)
assert initial.data == ()
models: Final = (f"{prefix}/first", f"{prefix}/literal%quote'", f"{prefix}/second")
try:
for index, model in enumerate((*models, models[0])):
await db.prisma.writer_db.execute_raw(
'INSERT INTO "LiteLLM_DailyUserSpend" (id,date,api_key,model,updated_at) VALUES ($1,$2,$3,$4,CURRENT_TIMESTAMP)',
f"{prefix}-{index}",
initial.end_date.isoformat(),
prefix,
model,
)
await db.prisma.writer_db.execute_raw(
'INSERT INTO "LiteLLM_DailyUserSpend" (id,date,api_key,model,updated_at) VALUES ($1,$2,$3,$4,CURRENT_TIMESTAMP)',
f"{prefix}-old",
(initial.start_date - timedelta(days=1)).isoformat(),
prefix,
f"{prefix}/old",
)
response: Final = gateway.request("GET", "/usage/alert-models", params={"search": prefix, "limit": "1"})
assert response.status_code == 200, response.text
first: Final = UsageAlertModelList.model_validate(response.json())
assert first.data[0].scope_id == models[0] and first.next_cursor == models[0]
second: Final = await repository.list_reported_models(prefix, first.next_cursor, 1)
third: Final = await repository.list_reported_models(prefix, second.next_cursor, 1)
assert second.data[0].scope_id == models[1] and third.data[0].scope_id == models[2]
assert third.next_cursor is None
literal: Final = await repository.list_reported_models(f"{prefix}/literal%quote'", None, 20)
assert tuple(option.scope_id for option in literal.data) == (models[1],)
assert literal.data[0].label == models[1]
with gateway.scenario() as scenario:
user: Final = scenario.user(user_role="internal_user")
key: Final = scenario.key(user_id=user, key_type="management")
assert gateway.request("GET", "/usage/alert-models", key=key).status_code == 403
finally:
await db.prisma.writer_db.execute_raw('DELETE FROM "LiteLLM_DailyUserSpend" WHERE api_key=$1', prefix)

View file

@ -3,7 +3,7 @@ from typing import Final
import pytest
from pydantic import ValidationError
from litellm.types.proxy.usage_alerts import UsageAlertRuleInput
from litellm.types.proxy.usage_alerts import UsageAlertGroupUpdate, UsageAlertRuleInput
RULE: Final = {
"name": "Daily spend",
@ -41,3 +41,21 @@ def test_zero_request_floor_and_daily_drop_multiplier_are_supported() -> None:
assert rule.direction == "decrease"
assert rule.multiplier == 2
assert rule.min_requests == 0
@pytest.mark.parametrize(
"changes",
[{"scope_id": None}, {"scope_type": "deployment"}, {"status": "open"}, {"status": "unknown"}],
)
def test_group_updates_require_exact_resource_and_a_different_valid_status(changes: dict[str, object]) -> None:
with pytest.raises(ValidationError):
UsageAlertGroupUpdate.model_validate(
{
"scope_type": "team",
"scope_id": "team-example",
"day": "2026-11-04",
"expected_status": "open",
"status": "resolved",
**changes,
}
)

View file

@ -18834,6 +18834,41 @@ export interface paths {
patch?: never;
trace?: never;
};
"/usage/alert-groups": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** List Usage Alert Groups */
get: operations["list_usage_alert_groups_usage_alert_groups_get"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
/** Update Usage Alert Group */
patch: operations["update_usage_alert_group_usage_alert_groups_patch"];
trace?: never;
};
"/usage/alert-models": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** List Usage Alert Models */
get: operations["list_usage_alert_models_usage_alert_models_get"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/usage/alert-rules": {
parameters: {
query?: never;
@ -50472,6 +50507,120 @@ export interface components {
/** Threshold */
threshold: number;
};
/** UsageAlertDefaultSummary */
UsageAlertDefaultSummary: {
/**
* Enabled
* @default 0
*/
enabled: number;
evaluation_status_counts?: components["schemas"]["UsageAlertEvaluationCounts"];
/**
* Total
* @default 0
*/
total: number;
};
/** UsageAlertEvaluationCounts */
UsageAlertEvaluationCounts: {
/**
* Complete
* @default 0
*/
complete: number;
/**
* Insufficient Coverage
* @default 0
*/
insufficient_coverage: number;
/**
* Insufficient History
* @default 0
*/
insufficient_history: number;
/**
* Pending
* @default 0
*/
pending: number;
};
/** UsageAlertGroup */
UsageAlertGroup: {
/** Activity Path */
activity_path: string;
/**
* Day
* Format: date
*/
day: string;
/** Id */
id: string;
/** Incident Count */
incident_count: number;
/** Metrics */
metrics: components["schemas"]["UsageAlertGroupMetric"][];
/** Scope Id */
scope_id: string | null;
/** Scope Label */
scope_label: string | null;
/**
* Scope Type
* @enum {string}
*/
scope_type: "deployment" | "key" | "user" | "team" | "model";
};
/** UsageAlertGroupList */
UsageAlertGroupList: {
/** Data */
data: components["schemas"]["UsageAlertGroup"][];
/** Limit */
limit: number;
/** Offset */
offset: number;
/** Total */
total: number;
};
/** UsageAlertGroupMetric */
UsageAlertGroupMetric: {
/** Incident Count */
incident_count: number;
/**
* Metric
* @enum {string}
*/
metric: "spend" | "requests" | "latency" | "errors";
representative: components["schemas"]["UsageAlert"];
};
/** UsageAlertGroupUpdate */
UsageAlertGroupUpdate: {
/**
* Day
* Format: date
*/
day: string;
/**
* Expected Status
* @enum {string}
*/
expected_status: "open" | "resolved" | "dismissed";
/** Scope Id */
scope_id?: string | null;
/**
* Scope Type
* @enum {string}
*/
scope_type: "deployment" | "key" | "user" | "team" | "model";
/**
* Status
* @enum {string}
*/
status: "open" | "resolved" | "dismissed";
};
/** UsageAlertGroupUpdateResult */
UsageAlertGroupUpdateResult: {
/** Updated Count */
updated_count: number;
};
/** UsageAlertList */
UsageAlertList: {
/** Data */
@ -50483,6 +50632,32 @@ export interface components {
/** Total */
total: number;
};
/** UsageAlertModelList */
UsageAlertModelList: {
/** Data */
data: components["schemas"]["UsageAlertModelOption"][];
/**
* End Date
* Format: date
*/
end_date: string;
/** Limit */
limit: number;
/** Next Cursor */
next_cursor: string | null;
/**
* Start Date
* Format: date
*/
start_date: string;
};
/** UsageAlertModelOption */
UsageAlertModelOption: {
/** Label */
label: string;
/** Scope Id */
scope_id: string;
};
/** UsageAlertRule */
UsageAlertRule: {
/**
@ -50520,6 +50695,11 @@ export interface components {
evaluation_status: "pending" | "complete" | "insufficient_coverage" | "insufficient_history";
/** Id */
id: string;
/**
* Is Default
* @default false
*/
is_default: boolean;
/** Last Evaluated Day */
last_evaluated_day?: string | null;
/**
@ -50635,6 +50815,7 @@ export interface components {
UsageAlertRuleList: {
/** Data */
data: components["schemas"]["UsageAlertRule"][];
default_summary?: components["schemas"]["UsageAlertDefaultSummary"];
/** Limit */
limit: number;
/** Offset */
@ -78561,6 +78742,109 @@ export interface operations {
};
};
};
list_usage_alert_groups_usage_alert_groups_get: {
parameters: {
query?: {
scope_type?: ("deployment" | "key" | "user" | "team" | "model") | null;
scope_id?: string | null;
status?: ("open" | "resolved" | "dismissed") | null;
start_date?: string | null;
end_date?: string | null;
limit?: number;
offset?: number;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["UsageAlertGroupList"];
};
};
/** @description Validation Error */
422: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HTTPValidationError"];
};
};
};
};
update_usage_alert_group_usage_alert_groups_patch: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["UsageAlertGroupUpdate"];
};
};
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["UsageAlertGroupUpdateResult"];
};
};
/** @description Validation Error */
422: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HTTPValidationError"];
};
};
};
};
list_usage_alert_models_usage_alert_models_get: {
parameters: {
query?: {
search?: string | null;
cursor?: string | null;
limit?: number;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["UsageAlertModelList"];
};
};
/** @description Validation Error */
422: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HTTPValidationError"];
};
};
};
};
list_usage_alert_rules_usage_alert_rules_get: {
parameters: {
query?: {
@ -78568,6 +78852,7 @@ export interface operations {
scope_id?: string | null;
limit?: number;
offset?: number;
kind?: "all" | "default" | "custom";
};
header?: never;
path?: never;