mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
feat: group Performance alerts by resource and day
This commit is contained in:
parent
549e263984
commit
949cc1e2ee
9 changed files with 1023 additions and 20 deletions
|
|
@ -1,19 +1,40 @@
|
|||
# Daily usage alerts
|
||||
# Daily Performance alerts
|
||||
|
||||
Daily usage alerts compare a completed UTC day with earlier recorded days. They cover spend, request count, average latency, and failed-request count for the gateway, a key, a user, a team, or a model. Open an incident to inspect its saved comparison and follow its Activity link to the corresponding usage view.
|
||||
Performance alerts compare a completed UTC day with earlier recorded days. They cover spend, request count, average latency, and failed-request count for the gateway, a key, a user, a team, or a model. The Performance tab groups alerts for the same resource and original UTC day. Open a resource to inspect its saved comparison and Activity. The API paths remain under `/usage`.
|
||||
|
||||
This feature requires a database. Apply the normal LiteLLM database migrations when deploying a version that includes usage alerts. The API and dashboard use the same persisted rules and incidents; no enterprise license is required.
|
||||
|
||||
## Read incidents
|
||||
## Read resource groups
|
||||
|
||||
Set `LITELLM_API_BASE` to your gateway's base URL and `LITELLM_MANAGEMENT_KEY` to a management-capable credential. Keep the key in your environment.
|
||||
|
||||
```bash
|
||||
curl "$LITELLM_API_BASE/usage/alerts?status=open&limit=50" \
|
||||
curl "$LITELLM_API_BASE/usage/alert-groups?status=open&limit=50" \
|
||||
-H "Authorization: Bearer $LITELLM_MANAGEMENT_KEY"
|
||||
```
|
||||
|
||||
The response has `data`, `total`, `limit`, and `offset`. Each incident includes its scope, metric, direction, status, UTC day, observed value, baseline value, threshold, baseline date range, and `activity_path`. The Activity path is relative to the dashboard mount: for example, `/api-keys?...` is opened below the dashboard's `/ui` mount.
|
||||
The response has `data`, `total`, `limit`, and `offset`. `total` counts resource/day groups, including when the requested offset is beyond the last page. Filters are `scope_type`, `scope_id`, `status`, `start_date`, and `end_date`. Status filters apply to individual incidents before grouping, so a group with open and resolved incidents can appear under either filter with the corresponding members. No default date cutoff hides older open incidents.
|
||||
|
||||
Each group includes its stable `id`, `scope_type`, exact `scope_id`, current `scope_label` when available, original UTC `day`, `incident_count`, and `activity_path`. Its `metrics` list contains up to four entries with a `metric`, `incident_count`, and the most recently detected matching `representative` incident. These representatives are summaries, not the entire group. Missing or deleted resources keep their original IDs. Names are current display labels, not historical evidence.
|
||||
|
||||
To inspect every matching rule, use `GET /usage/alerts` with the group's scope and `start_date` and `end_date` both set to its `day`. Add `metric` and `status` as needed and paginate that response. Rules can have different baseline windows and thresholds, so their comparisons must not be combined into an error rate or a cost-per-request estimate.
|
||||
|
||||
The group Activity link uses a representative's saved date range and alert context. The path is relative to the dashboard mount: for example, `/api-keys?...` is opened below the dashboard's `/ui` mount. A continuing episode retains its original day and evidence; the grouped row does not describe today's traffic.
|
||||
|
||||
Resolve or dismiss all matching incidents in one resource/day group with an expected current status:
|
||||
|
||||
```bash
|
||||
curl -X PATCH "$LITELLM_API_BASE/usage/alert-groups" \
|
||||
-H "Authorization: Bearer $LITELLM_MANAGEMENT_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"scope_type":"team","scope_id":"team-example","day":"2026-11-04","expected_status":"open","status":"resolved"}'
|
||||
```
|
||||
|
||||
The response contains `updated_count`. Only incidents in that exact group that still have `expected_status` are updated atomically. Other statuses remain unchanged; repeating the same request returns zero after all matching incidents have changed. Use `scope_type: "deployment"` with `scope_id: null` for the gateway.
|
||||
|
||||
## Read individual incidents
|
||||
|
||||
`GET /usage/alerts` retains the individual-incident view. Each incident includes its scope, metric, direction, status, UTC day, observed value, baseline value, threshold, baseline date range, and `activity_path`.
|
||||
|
||||
`GET /usage/alerts/{alert_id}` returns one incident. List filters are `scope_type`, `scope_id`, `metric`, `status`, `start_date`, and `end_date`. Dates use `YYYY-MM-DD`; date filters include both endpoints. The maximum page size is 100.
|
||||
|
||||
|
|
@ -55,10 +76,12 @@ The evaluator runs against daily aggregates. An hourly scheduler does not make t
|
|||
|
||||
Use `scope_type: "deployment"` with no `scope_id` for the gateway. For `model`, use the underlying model identity recorded in Activity. For `key`, use its reporting ID, never the secret API key. Omitting `scope_id` for key, user, team, or model evaluates all entities of that type and requires a gateway administrator.
|
||||
|
||||
`GET /usage/alert-models?limit=20&search=example` returns paginated model identities recorded in the most recent 30 UTC days. Its `data` entries contain `scope_id` and `label`; `start_date` and `end_date` state the searched window. Pass `next_cursor` as `cursor` to read the next page. A null cursor ends the result. Search matches literal text, not SQL wildcards. This endpoint requires gateway admin or admin-viewer access. Configured aliases and deployment IDs are not reporting-model identities. An existing rule's saved model remains valid even when it falls outside the options window.
|
||||
|
||||
## Manage rules and incidents
|
||||
|
||||
- `GET /usage/alert-rules` lists rules; `GET /usage/alert-rules/{rule_id}` reads one.
|
||||
- `PUT /usage/alert-rules/{rule_id}` replaces the rule configuration. Send the configuration fields from the create example, including any notification settings you want to retain. Set `enabled` to `false` to pause it.
|
||||
- `GET /usage/alert-rules` lists rules; `GET /usage/alert-rules/{rule_id}` reads one. Use `kind=custom`, `kind=default`, or `kind=all` to choose which rules to list. Each rule exposes `is_default`. The list's `default_summary` counts total and enabled defaults within the requested scope, independent of `kind`, and reports evaluation-status counts among enabled defaults.
|
||||
- `PUT /usage/alert-rules/{rule_id}` replaces the rule configuration. Send the configuration fields from the create example, including any notification settings you want to retain. Set `enabled` to `false` to pause it. Built-in rules keep their scope, metric, and direction; create a custom rule to change those fields. Their names, thresholds, notification settings, and enabled state remain editable.
|
||||
- `DELETE /usage/alert-rules/{rule_id}` removes a rule from evaluation and preserves existing incidents. Deleted defaults stay deleted.
|
||||
- `PATCH /usage/alerts/{alert_id}` with `{"status":"dismissed"}` dismisses an incident. `open` and `resolved` are also accepted.
|
||||
|
||||
|
|
|
|||
|
|
@ -3,10 +3,15 @@ from typing import Protocol
|
|||
|
||||
from litellm.types.proxy.usage_alerts import (
|
||||
UsageAlert,
|
||||
UsageAlertGroupList,
|
||||
UsageAlertGroupUpdate,
|
||||
UsageAlertGroupUpdateResult,
|
||||
UsageAlertList,
|
||||
UsageAlertMetric,
|
||||
UsageAlertModelList,
|
||||
UsageAlertRule,
|
||||
UsageAlertRuleInput,
|
||||
UsageAlertRuleKind,
|
||||
UsageAlertRuleList,
|
||||
UsageAlertScope,
|
||||
UsageAlertStatus,
|
||||
|
|
@ -23,7 +28,12 @@ class UsageAlertStore(Protocol):
|
|||
async def delete_rule(self, rule_id: str) -> None: ...
|
||||
|
||||
async def list_rules(
|
||||
self, scope_type: UsageAlertScope | None = None, scope_id: str | None = None, limit: int = 50, offset: int = 0
|
||||
self,
|
||||
scope_type: UsageAlertScope | None = None,
|
||||
scope_id: str | None = None,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
kind: UsageAlertRuleKind = "all",
|
||||
) -> UsageAlertRuleList: ...
|
||||
|
||||
async def get_alert(self, alert_id: str) -> UsageAlert | None: ...
|
||||
|
|
@ -42,6 +52,21 @@ class UsageAlertStore(Protocol):
|
|||
|
||||
async def set_status(self, alert_id: str, status: UsageAlertStatus) -> UsageAlert | None: ...
|
||||
|
||||
async def list_groups(
|
||||
self,
|
||||
scope_type: UsageAlertScope | None = None,
|
||||
scope_id: str | None = None,
|
||||
status: UsageAlertStatus | None = None,
|
||||
start_date: date | None = None,
|
||||
end_date: date | None = None,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
) -> UsageAlertGroupList: ...
|
||||
|
||||
async def set_group_status(self, data: UsageAlertGroupUpdate) -> UsageAlertGroupUpdateResult: ...
|
||||
|
||||
async def list_reported_models(self, search: str | None, cursor: str | None, limit: int) -> UsageAlertModelList: ...
|
||||
|
||||
async def key_ownership(self, key_id: str) -> tuple[str | None, str | None] | None: ...
|
||||
|
||||
async def team_role(self, team_id: str, user_id: str) -> str | None: ...
|
||||
|
|
|
|||
|
|
@ -579,6 +579,8 @@ class LiteLLMRoutes(enum.Enum):
|
|||
)
|
||||
llm_api_routes_without_mcp = sorted(set(llm_api_routes) - set(mcp_inference_routes))
|
||||
info_routes = [
|
||||
"/usage/alert-groups",
|
||||
"/usage/alert-models",
|
||||
"/usage/alert-rules",
|
||||
"/usage/alert-rules/{rule_id}",
|
||||
"/usage/alerts",
|
||||
|
|
@ -650,6 +652,8 @@ class LiteLLMRoutes(enum.Enum):
|
|||
|
||||
management_routes = (
|
||||
[
|
||||
"/usage/alert-groups",
|
||||
"/usage/alert-models",
|
||||
"/usage/alert-rules",
|
||||
"/usage/alert-rules/{rule_id}",
|
||||
"/usage/alerts",
|
||||
|
|
@ -864,6 +868,8 @@ class LiteLLMRoutes(enum.Enum):
|
|||
)
|
||||
|
||||
usage_alert_routes = (
|
||||
"/usage/alert-groups",
|
||||
"/usage/alert-models",
|
||||
"/usage/alert-rules",
|
||||
"/usage/alert-rules/{rule_id}",
|
||||
"/usage/alerts",
|
||||
|
|
|
|||
|
|
@ -13,10 +13,15 @@ from litellm.repositories.daily_activity_buckets import api_key_from_bucket
|
|||
from litellm.types.proxy.auth.user_roles import LitellmUserRoles
|
||||
from litellm.types.proxy.usage_alerts import (
|
||||
UsageAlert,
|
||||
UsageAlertGroupList,
|
||||
UsageAlertGroupUpdate,
|
||||
UsageAlertGroupUpdateResult,
|
||||
UsageAlertList,
|
||||
UsageAlertMetric,
|
||||
UsageAlertModelList,
|
||||
UsageAlertRule,
|
||||
UsageAlertRuleInput,
|
||||
UsageAlertRuleKind,
|
||||
UsageAlertRuleList,
|
||||
UsageAlertScope,
|
||||
UsageAlertStatus,
|
||||
|
|
@ -93,10 +98,52 @@ async def list_usage_alert_rules(
|
|||
scope_id: ScopeId = None,
|
||||
limit: Limit = 50,
|
||||
offset: Offset = 0,
|
||||
kind: UsageAlertRuleKind = "all",
|
||||
) -> UsageAlertRuleList:
|
||||
store: Final = require_repository(repository)
|
||||
await require_scope(user, store, scope_type, scope_id, write=False)
|
||||
return await store.list_rules(scope_type, scope_id, limit, offset)
|
||||
return await store.list_rules(scope_type, scope_id, limit, offset, kind)
|
||||
|
||||
|
||||
@router.get("/alert-groups", response_model=UsageAlertGroupList)
|
||||
async def list_usage_alert_groups(
|
||||
user: User,
|
||||
repository: Repository,
|
||||
scope_type: UsageAlertScope | None = None,
|
||||
scope_id: ScopeId = None,
|
||||
status: UsageAlertStatus | None = None,
|
||||
start_date: date | None = None,
|
||||
end_date: date | None = None,
|
||||
limit: Limit = 50,
|
||||
offset: Offset = 0,
|
||||
) -> UsageAlertGroupList:
|
||||
store: Final = require_repository(repository)
|
||||
await require_scope(user, store, scope_type, scope_id, write=False)
|
||||
if start_date is not None and end_date is not None and start_date > end_date:
|
||||
raise HTTPException(status_code=422, detail="start_date must not be after end_date")
|
||||
return await store.list_groups(scope_type, scope_id, status, start_date, end_date, limit, offset)
|
||||
|
||||
|
||||
@router.patch("/alert-groups", response_model=UsageAlertGroupUpdateResult)
|
||||
async def update_usage_alert_group(
|
||||
data: UsageAlertGroupUpdate, user: User, repository: Repository
|
||||
) -> UsageAlertGroupUpdateResult:
|
||||
store: Final = require_repository(repository)
|
||||
await require_scope(user, store, data.scope_type, data.scope_id, write=True)
|
||||
return await store.set_group_status(data)
|
||||
|
||||
|
||||
@router.get("/alert-models", response_model=UsageAlertModelList)
|
||||
async def list_usage_alert_models(
|
||||
user: User,
|
||||
repository: Repository,
|
||||
search: ScopeId = None,
|
||||
cursor: ScopeId = None,
|
||||
limit: Limit = 20,
|
||||
) -> UsageAlertModelList:
|
||||
store: Final = require_repository(repository)
|
||||
await require_scope(user, store, "model", None, write=False)
|
||||
return await store.list_reported_models(search, cursor, limit)
|
||||
|
||||
|
||||
@router.post("/alert-rules", response_model=UsageAlertRule, status_code=201)
|
||||
|
|
@ -129,6 +176,15 @@ async def update_usage_alert_rule(
|
|||
if rule is None:
|
||||
raise HTTPException(status_code=404, detail="Alert rule not found")
|
||||
await require_scope(user, store, rule.scope_type, rule.scope_id, write=True)
|
||||
if rule.is_default and (rule.scope_type, rule.scope_id, rule.metric, rule.direction) != (
|
||||
data.scope_type,
|
||||
data.scope_id,
|
||||
data.metric,
|
||||
data.direction,
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=422, detail="Create a custom rule to change a built-in rule's scope, metric, or direction"
|
||||
)
|
||||
await require_scope(user, store, data.scope_type, data.scope_id, write=True)
|
||||
saved: Final = await store.save_rule(data, user.user_id, existing=rule)
|
||||
if saved is None:
|
||||
|
|
|
|||
|
|
@ -3,6 +3,8 @@
|
|||
import json
|
||||
from collections.abc import Mapping
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from itertools import product
|
||||
from types import MappingProxyType
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
Final,
|
||||
|
|
@ -14,16 +16,26 @@ from uuid import NAMESPACE_URL, uuid4, uuid5
|
|||
|
||||
from pydantic import BaseModel, TypeAdapter
|
||||
|
||||
from litellm.repositories.daily_activity_buckets import api_key_from_bucket
|
||||
from litellm.types.proxy.usage_alerts import (
|
||||
UsageAlert,
|
||||
UsageAlertChannel,
|
||||
UsageAlertDefaultSummary,
|
||||
UsageAlertEvaluationStatus,
|
||||
UsageAlertGroup,
|
||||
UsageAlertGroupList,
|
||||
UsageAlertGroupMetric,
|
||||
UsageAlertGroupUpdate,
|
||||
UsageAlertGroupUpdateResult,
|
||||
UsageAlertList,
|
||||
UsageAlertMetric,
|
||||
UsageAlertModelList,
|
||||
UsageAlertModelOption,
|
||||
UsageAlertNotification,
|
||||
UsageAlertObservation,
|
||||
UsageAlertRule,
|
||||
UsageAlertRuleInput,
|
||||
UsageAlertRuleKind,
|
||||
UsageAlertRuleList,
|
||||
UsageAlertScope,
|
||||
UsageAlertStatus,
|
||||
|
|
@ -45,6 +57,37 @@ class _Count(BaseModel, frozen=True):
|
|||
count: int
|
||||
|
||||
|
||||
class _RuleCounts(_Count, frozen=True):
|
||||
default_summary: UsageAlertDefaultSummary
|
||||
|
||||
|
||||
class _GroupRow(BaseModel, frozen=True):
|
||||
scope_type: UsageAlertScope
|
||||
scope_id: str | None
|
||||
day: date
|
||||
incident_count: int
|
||||
metrics: tuple[UsageAlertGroupMetric, ...]
|
||||
|
||||
|
||||
class _GroupPage(BaseModel, frozen=True):
|
||||
total: int
|
||||
data: tuple[_GroupRow, ...]
|
||||
|
||||
|
||||
class _ResourceLabel(BaseModel, frozen=True):
|
||||
scope_type: UsageAlertScope
|
||||
scope_id: str | None
|
||||
label: str | None
|
||||
|
||||
|
||||
class _ModelIdentity(BaseModel, frozen=True):
|
||||
model: str
|
||||
|
||||
|
||||
class _Identifier(BaseModel, frozen=True):
|
||||
id: str
|
||||
|
||||
|
||||
class _Pending(_Record, frozen=True):
|
||||
rule: Mapping[str, object]
|
||||
notified_channels: tuple[UsageAlertChannel, ...]
|
||||
|
|
@ -55,6 +98,20 @@ class _Ownership(BaseModel, frozen=True):
|
|||
team_id: str | None
|
||||
|
||||
|
||||
_DEFAULT_RULE_IDS: Final = frozenset(
|
||||
str(uuid5(NAMESPACE_URL, f"litellm:usage-alert:{scope}:{metric}:{direction}"))
|
||||
for scope, metric, direction in product(
|
||||
("deployment", "key", "user", "team", "model"),
|
||||
("spend", "requests", "latency", "errors"),
|
||||
("increase", "decrease"),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _rule(record: Mapping[str, object]) -> UsageAlertRule:
|
||||
return UsageAlertRule.model_validate({**record, "is_default": str(record.get("id")) in _DEFAULT_RULE_IDS})
|
||||
|
||||
|
||||
def activity_path(
|
||||
scope: UsageAlertScope, entity_id: str | None, day: date, alert_id: str, baseline_days: int = 7
|
||||
) -> str:
|
||||
|
|
@ -151,7 +208,7 @@ class UsageAlertsRepository:
|
|||
'SELECT record FROM "LiteLLM_UsageAlertRule" WHERE id=$1 AND NOT deleted', rule_id
|
||||
)
|
||||
)
|
||||
return UsageAlertRule.model_validate(rows[0].record) if rows else None
|
||||
return _rule(rows[0].record) if rows else None
|
||||
|
||||
async def save_rule(
|
||||
self, data: UsageAlertRuleInput, created_by: str | None, existing: UsageAlertRule | None = None
|
||||
|
|
@ -177,7 +234,7 @@ class UsageAlertsRepository:
|
|||
rule.model_dump_json(),
|
||||
)
|
||||
saved: Final = TypeAdapter(tuple[_Record, ...]).validate_python(rows)
|
||||
return UsageAlertRule.model_validate(saved[0].record) if saved else None
|
||||
return _rule(saved[0].record) if saved else None
|
||||
|
||||
async def delete_rule(self, rule_id: str) -> None:
|
||||
await self.database.query_raw(
|
||||
|
|
@ -185,28 +242,46 @@ class UsageAlertsRepository:
|
|||
)
|
||||
|
||||
async def list_rules(
|
||||
self, scope_type: UsageAlertScope | None = None, scope_id: str | None = None, limit: int = 50, offset: int = 0
|
||||
self,
|
||||
scope_type: UsageAlertScope | None = None,
|
||||
scope_id: str | None = None,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
kind: UsageAlertRuleKind = "all",
|
||||
) -> UsageAlertRuleList:
|
||||
args: Final = (scope_type, scope_id)
|
||||
args: Final = (scope_type, scope_id, json.dumps(tuple(sorted(_DEFAULT_RULE_IDS))), kind)
|
||||
where: Final = "NOT deleted AND ($1::text IS NULL OR scope_type=$1) AND ($2::text IS NULL OR scope_id=$2)"
|
||||
count: Final = TypeAdapter(tuple[_Count, ...]).validate_python(
|
||||
builtin: Final = "id IN (SELECT jsonb_array_elements_text($3::jsonb))"
|
||||
count: Final = TypeAdapter(tuple[_RuleCounts, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
f'SELECT count(*)::int AS count FROM "LiteLLM_UsageAlertRule" WHERE {where}', *args
|
||||
f"""WITH scoped AS (
|
||||
SELECT enabled,record,({builtin}) AS builtin FROM "LiteLLM_UsageAlertRule" WHERE {where}
|
||||
) SELECT count(*) FILTER (WHERE $4='all' OR builtin=($4='default'))::int AS count,
|
||||
jsonb_build_object('total',count(*) FILTER (WHERE builtin),
|
||||
'enabled',count(*) FILTER (WHERE builtin AND enabled),
|
||||
'evaluation_status_counts',jsonb_build_object(
|
||||
'pending',count(*) FILTER (WHERE builtin AND enabled AND COALESCE(record->>'evaluation_status','pending')='pending'),
|
||||
'complete',count(*) FILTER (WHERE builtin AND enabled AND record->>'evaluation_status'='complete'),
|
||||
'insufficient_coverage',count(*) FILTER (WHERE builtin AND enabled AND record->>'evaluation_status'='insufficient_coverage'),
|
||||
'insufficient_history',count(*) FILTER (WHERE builtin AND enabled AND record->>'evaluation_status'='insufficient_history')
|
||||
)) AS default_summary FROM scoped""",
|
||||
*args,
|
||||
)
|
||||
)
|
||||
rows: Final = TypeAdapter(tuple[_Record, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
f'SELECT record FROM "LiteLLM_UsageAlertRule" WHERE {where} ORDER BY id LIMIT $3 OFFSET $4',
|
||||
f"SELECT record FROM \"LiteLLM_UsageAlertRule\" WHERE {where} AND ($4='all' OR ({builtin})=($4='default')) ORDER BY id LIMIT $5 OFFSET $6",
|
||||
*args,
|
||||
limit,
|
||||
offset,
|
||||
)
|
||||
)
|
||||
return UsageAlertRuleList(
|
||||
data=tuple(UsageAlertRule.model_validate(row.record) for row in rows),
|
||||
data=tuple(_rule(row.record) for row in rows),
|
||||
total=count[0].count,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
default_summary=count[0].default_summary,
|
||||
)
|
||||
|
||||
async def list_enabled_rules(self, limit: int = 100, offset: int = 0) -> tuple[UsageAlertRule, ...]:
|
||||
|
|
@ -275,6 +350,165 @@ class UsageAlertsRepository:
|
|||
)
|
||||
return UsageAlert.model_validate(rows[0].record) if rows else None
|
||||
|
||||
async def list_groups(
|
||||
self,
|
||||
scope_type: UsageAlertScope | None = None,
|
||||
scope_id: str | None = None,
|
||||
status: UsageAlertStatus | None = None,
|
||||
start_date: date | None = None,
|
||||
end_date: date | None = None,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
) -> UsageAlertGroupList:
|
||||
rows: Final = TypeAdapter(tuple[_GroupPage, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
"""WITH groups AS MATERIALIZED (
|
||||
SELECT scope_type,scope_id,day,count(*)::int AS incident_count FROM "LiteLLM_UsageAlertIncident"
|
||||
WHERE ($1::text IS NULL OR scope_type=$1) AND ($2::text IS NULL OR scope_id=$2)
|
||||
AND ($3::text IS NULL OR status=$3) AND ($4::text IS NULL OR day >= $4)
|
||||
AND ($5::text IS NULL OR day <= $5)
|
||||
GROUP BY scope_type,scope_id,day
|
||||
), page AS MATERIALIZED (
|
||||
SELECT * FROM groups ORDER BY day DESC,scope_type,scope_id NULLS FIRST LIMIT $6 OFFSET $7
|
||||
), selected AS (
|
||||
SELECT i.* FROM page p JOIN "LiteLLM_UsageAlertIncident" i
|
||||
ON i.scope_type=p.scope_type AND i.scope_id=p.scope_id AND i.day=p.day
|
||||
WHERE ($3::text IS NULL OR i.status=$3)
|
||||
UNION ALL
|
||||
SELECT i.* FROM page p JOIN "LiteLLM_UsageAlertIncident" i
|
||||
ON i.scope_type=p.scope_type AND i.scope_id IS NULL AND i.day=p.day
|
||||
WHERE p.scope_id IS NULL AND ($3::text IS NULL OR i.status=$3)
|
||||
), ranked AS (
|
||||
SELECT scope_type,scope_id,day,metric,record,
|
||||
count(*) OVER (PARTITION BY scope_type,scope_id,day,metric)::int AS incident_count,
|
||||
row_number() OVER (PARTITION BY scope_type,scope_id,day,metric
|
||||
ORDER BY record->>'first_seen' DESC,id) AS rank
|
||||
FROM selected
|
||||
), packed AS (
|
||||
SELECT p.scope_type,p.scope_id,p.day,p.incident_count,
|
||||
jsonb_agg(jsonb_build_object('metric',r.metric,'incident_count',r.incident_count,
|
||||
'representative',r.record) ORDER BY CASE r.metric
|
||||
WHEN 'requests' THEN 0 WHEN 'spend' THEN 1 WHEN 'errors' THEN 2 ELSE 3 END) AS metrics
|
||||
FROM page p JOIN ranked r ON p.scope_type=r.scope_type
|
||||
AND p.scope_id IS NOT DISTINCT FROM r.scope_id AND p.day=r.day AND r.rank=1
|
||||
GROUP BY p.scope_type,p.scope_id,p.day,p.incident_count
|
||||
) SELECT (SELECT count(*)::int FROM groups) AS total,
|
||||
COALESCE((SELECT jsonb_agg(to_jsonb(packed) ORDER BY day DESC,scope_type,scope_id NULLS FIRST)
|
||||
FROM packed),'[]'::jsonb) AS data""",
|
||||
scope_type,
|
||||
scope_id,
|
||||
status,
|
||||
start_date.isoformat() if start_date else None,
|
||||
end_date.isoformat() if end_date else None,
|
||||
limit,
|
||||
offset,
|
||||
)
|
||||
)
|
||||
page: Final = rows[0]
|
||||
labels: Final[Mapping[tuple[UsageAlertScope, str | None], str | None]] = (
|
||||
await self._group_labels(page.data) if page.data else {}
|
||||
)
|
||||
return UsageAlertGroupList(
|
||||
data=tuple(
|
||||
UsageAlertGroup(
|
||||
id=str(
|
||||
uuid5(
|
||||
NAMESPACE_URL,
|
||||
json.dumps(
|
||||
["litellm:performance-group", row.scope_type, row.scope_id, row.day.isoformat()]
|
||||
),
|
||||
)
|
||||
),
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
scope_label=labels.get((row.scope_type, row.scope_id)),
|
||||
day=row.day,
|
||||
incident_count=row.incident_count,
|
||||
metrics=row.metrics,
|
||||
activity_path=row.metrics[0].representative.activity_path,
|
||||
)
|
||||
for row in page.data
|
||||
),
|
||||
total=page.total,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
async def _group_labels(
|
||||
self, groups: tuple[_GroupRow, ...]
|
||||
) -> Mapping[tuple[UsageAlertScope, str | None], str | None]:
|
||||
rows: Final = TypeAdapter(tuple[_ResourceLabel, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
"""WITH scopes AS (
|
||||
SELECT DISTINCT scope_type,scope_id,key_id FROM jsonb_to_recordset($1::jsonb)
|
||||
AS x(scope_type text,scope_id text,key_id text)
|
||||
) SELECT s.scope_type,s.scope_id,CASE s.scope_type
|
||||
WHEN 'key' THEN COALESCE(NULLIF(k.key_alias,''),NULLIF(k.key_name,''))
|
||||
WHEN 'team' THEN NULLIF(t.team_alias,'')
|
||||
WHEN 'user' THEN COALESCE(NULLIF(u.user_alias,''),NULLIF(u.user_email,''))
|
||||
WHEN 'model' THEN s.scope_id WHEN 'deployment' THEN 'Gateway' END AS label
|
||||
FROM scopes s LEFT JOIN "LiteLLM_VerificationToken" k ON s.scope_type='key' AND k.token=s.key_id
|
||||
LEFT JOIN "LiteLLM_TeamTable" t ON s.scope_type='team' AND t.team_id=s.scope_id
|
||||
LEFT JOIN "LiteLLM_UserTable" u ON s.scope_type='user' AND u.user_id=s.scope_id""",
|
||||
json.dumps(
|
||||
tuple(
|
||||
{
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"key_id": api_key_from_bucket(row.scope_id)
|
||||
if row.scope_type == "key" and row.scope_id
|
||||
else None,
|
||||
}
|
||||
for row in groups
|
||||
)
|
||||
),
|
||||
)
|
||||
)
|
||||
return MappingProxyType({(row.scope_type, row.scope_id): row.label for row in rows})
|
||||
|
||||
async def set_group_status(self, data: UsageAlertGroupUpdate) -> UsageAlertGroupUpdateResult:
|
||||
rows: Final = TypeAdapter(tuple[_Count, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
"""WITH changed AS (
|
||||
UPDATE "LiteLLM_UsageAlertIncident" SET status=$5,
|
||||
record=jsonb_set(record,'{status}',to_jsonb($5::text)),updated_at=CURRENT_TIMESTAMP
|
||||
WHERE scope_type=$1 AND scope_id IS NOT DISTINCT FROM $2::text AND day=$3 AND status=$4
|
||||
RETURNING id
|
||||
) SELECT count(*)::int AS count FROM changed""",
|
||||
data.scope_type,
|
||||
data.scope_id,
|
||||
data.day.isoformat(),
|
||||
data.expected_status,
|
||||
data.status,
|
||||
)
|
||||
)
|
||||
return UsageAlertGroupUpdateResult(updated_count=rows[0].count)
|
||||
|
||||
async def list_reported_models(self, search: str | None, cursor: str | None, limit: int) -> UsageAlertModelList:
|
||||
today: Final = datetime.now(timezone.utc).date()
|
||||
rows: Final = TypeAdapter(tuple[_ModelIdentity, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
"""SELECT DISTINCT model FROM "LiteLLM_DailyUserSpend"
|
||||
WHERE date >= $1 AND date <= $2 AND model IS NOT NULL AND model <> ''
|
||||
AND ($3::text IS NULL OR strpos(lower(model),lower($3))>0)
|
||||
AND ($4::text IS NULL OR model>$4)
|
||||
ORDER BY model LIMIT $5""",
|
||||
(today - timedelta(days=29)).isoformat(),
|
||||
today.isoformat(),
|
||||
search,
|
||||
cursor,
|
||||
limit + 1,
|
||||
)
|
||||
)
|
||||
selected: Final = rows[:limit]
|
||||
return UsageAlertModelList(
|
||||
data=tuple(UsageAlertModelOption(scope_id=row.model, label=row.model) for row in selected),
|
||||
limit=limit,
|
||||
next_cursor=selected[-1].model if len(rows) > limit else None,
|
||||
start_date=today - timedelta(days=29),
|
||||
end_date=today,
|
||||
)
|
||||
|
||||
async def key_ownership(self, key_id: str) -> tuple[str | None, str | None] | None:
|
||||
rows: Final = TypeAdapter(tuple[_Ownership, ...]).validate_python(
|
||||
await self.database.query_raw(
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ UsageAlertDirection: TypeAlias = Literal["increase", "decrease"]
|
|||
UsageAlertStatus: TypeAlias = Literal["open", "resolved", "dismissed"]
|
||||
UsageAlertChannel: TypeAlias = Literal["email", "slack"]
|
||||
UsageAlertEvaluationStatus: TypeAlias = Literal["pending", "complete", "insufficient_coverage", "insufficient_history"]
|
||||
UsageAlertRuleKind: TypeAlias = Literal["all", "default", "custom"]
|
||||
|
||||
|
||||
class UsageAlertRuleInput(BaseModel):
|
||||
|
|
@ -60,6 +61,7 @@ class UsageAlertRule(UsageAlertRuleInput):
|
|||
created_by: str | None
|
||||
last_evaluated_day: date | None = None
|
||||
evaluation_status: UsageAlertEvaluationStatus = "pending"
|
||||
is_default: bool = False
|
||||
|
||||
|
||||
class UsageAlert(BaseModel):
|
||||
|
|
@ -90,11 +92,25 @@ class UsageAlertUpdate(BaseModel):
|
|||
status: UsageAlertStatus
|
||||
|
||||
|
||||
class UsageAlertEvaluationCounts(BaseModel, frozen=True):
|
||||
pending: int = 0
|
||||
complete: int = 0
|
||||
insufficient_coverage: int = 0
|
||||
insufficient_history: int = 0
|
||||
|
||||
|
||||
class UsageAlertDefaultSummary(BaseModel, frozen=True):
|
||||
total: int = 0
|
||||
enabled: int = 0
|
||||
evaluation_status_counts: UsageAlertEvaluationCounts = Field(default_factory=UsageAlertEvaluationCounts)
|
||||
|
||||
|
||||
class UsageAlertRuleList(BaseModel, frozen=True):
|
||||
data: tuple[UsageAlertRule, ...]
|
||||
total: int
|
||||
limit: int
|
||||
offset: int
|
||||
default_summary: UsageAlertDefaultSummary = Field(default_factory=UsageAlertDefaultSummary)
|
||||
|
||||
|
||||
class UsageAlertList(BaseModel, frozen=True):
|
||||
|
|
@ -104,6 +120,64 @@ class UsageAlertList(BaseModel, frozen=True):
|
|||
offset: int
|
||||
|
||||
|
||||
class UsageAlertGroupMetric(BaseModel, frozen=True):
|
||||
metric: UsageAlertMetric
|
||||
incident_count: int
|
||||
representative: UsageAlert
|
||||
|
||||
|
||||
class UsageAlertGroup(BaseModel, frozen=True):
|
||||
id: str
|
||||
scope_type: UsageAlertScope
|
||||
scope_id: str | None
|
||||
scope_label: str | None
|
||||
day: date
|
||||
incident_count: int
|
||||
metrics: tuple[UsageAlertGroupMetric, ...] = Field(max_length=4)
|
||||
activity_path: str
|
||||
|
||||
|
||||
class UsageAlertGroupList(BaseModel, frozen=True):
|
||||
data: tuple[UsageAlertGroup, ...]
|
||||
total: int
|
||||
limit: int
|
||||
offset: int
|
||||
|
||||
|
||||
class UsageAlertGroupUpdate(BaseModel, frozen=True):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
scope_type: UsageAlertScope
|
||||
scope_id: str | None = Field(default=None, min_length=1, max_length=1024)
|
||||
day: date
|
||||
expected_status: UsageAlertStatus
|
||||
status: UsageAlertStatus
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_group(self) -> Self:
|
||||
if (self.scope_type == "deployment") != (self.scope_id is None):
|
||||
raise ValueError("Specify a resource ID except for deployment alerts")
|
||||
if self.expected_status == self.status:
|
||||
raise ValueError("The new status must differ from the expected status")
|
||||
return self
|
||||
|
||||
|
||||
class UsageAlertGroupUpdateResult(BaseModel, frozen=True):
|
||||
updated_count: int
|
||||
|
||||
|
||||
class UsageAlertModelOption(BaseModel, frozen=True):
|
||||
scope_id: str
|
||||
label: str
|
||||
|
||||
|
||||
class UsageAlertModelList(BaseModel, frozen=True):
|
||||
data: tuple[UsageAlertModelOption, ...]
|
||||
limit: int
|
||||
next_cursor: str | None
|
||||
start_date: date
|
||||
end_date: date
|
||||
|
||||
|
||||
class UsageAlertNotification(BaseModel):
|
||||
model_config = ConfigDict(frozen=True)
|
||||
incident: UsageAlert
|
||||
|
|
|
|||
|
|
@ -1,13 +1,21 @@
|
|||
from datetime import date
|
||||
from datetime import date, timedelta
|
||||
from hashlib import sha256
|
||||
from typing import Final
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from integration._support.actors import ManagementDatabase
|
||||
from integration._support.client import Gateway, string_value
|
||||
|
||||
from litellm.repositories.usage_alerts import UsageAlertsRepository
|
||||
from litellm.types.proxy.usage_alerts import UsageAlertObservation, UsageAlertRule, UsageAlertRuleInput
|
||||
from litellm.types.proxy.usage_alerts import (
|
||||
UsageAlertGroupList,
|
||||
UsageAlertMetric,
|
||||
UsageAlertModelList,
|
||||
UsageAlertObservation,
|
||||
UsageAlertRule,
|
||||
UsageAlertRuleInput,
|
||||
)
|
||||
|
||||
|
||||
def test_scoped_rules_use_real_management_auth_and_preserve_read_only_roles(gateway: Gateway) -> None:
|
||||
|
|
@ -252,3 +260,277 @@ async def test_rule_identity_edits_preserve_old_incident_and_filter_consistency(
|
|||
assert resolved is not None and resolved.status == "resolved"
|
||||
finally:
|
||||
await repository.delete_rule(rule.id)
|
||||
|
||||
|
||||
@pytest.mark.asyncio(loop_scope="session")
|
||||
async def test_grouped_http_pages_preserve_all_metrics_history_and_scoped_actions(
|
||||
gateway: Gateway, db: ManagementDatabase
|
||||
) -> None:
|
||||
repository: Final = UsageAlertsRepository.from_prisma(db.prisma)
|
||||
day: Final = date(2026, 11, 4)
|
||||
with gateway.scenario() as scenario:
|
||||
owner: Final = scenario.user(user_role="internal_user", user_alias="Checkout operator")
|
||||
outsider: Final = scenario.user(user_role="internal_user")
|
||||
viewer: Final = scenario.user(user_role="proxy_admin_viewer")
|
||||
owner_key: Final = scenario.key(user_id=owner, key_type="management")
|
||||
outsider_key: Final = scenario.key(user_id=outsider, key_type="management")
|
||||
viewer_key: Final = scenario.key(user_id=viewer, key_type="read_only")
|
||||
inference_key: Final = scenario.key(user_id=owner, key_type="llm_api")
|
||||
metrics: Final[tuple[UsageAlertMetric, ...]] = (*(("requests",) * 27), "spend", "errors", "latency")
|
||||
rules: Final = tuple(
|
||||
[
|
||||
await repository.save_rule(
|
||||
UsageAlertRuleInput(
|
||||
name=f"Group rule {index}",
|
||||
scope_type="user",
|
||||
scope_id=owner,
|
||||
metric=metric,
|
||||
direction="increase",
|
||||
baseline_days=7 if index % 2 else 14,
|
||||
),
|
||||
owner,
|
||||
)
|
||||
for index, metric in enumerate(metrics)
|
||||
]
|
||||
)
|
||||
assert all(rule is not None for rule in rules)
|
||||
incidents: Final = tuple(
|
||||
[
|
||||
(
|
||||
await repository.record_observations(
|
||||
rule,
|
||||
(
|
||||
UsageAlertObservation(
|
||||
entity_id=owner,
|
||||
day=day,
|
||||
observed=420,
|
||||
baseline=0 if index == 0 else 120,
|
||||
threshold=100 if index == 0 else 360,
|
||||
breached=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
)[0]
|
||||
for index, rule in enumerate(rules)
|
||||
if rule is not None
|
||||
]
|
||||
)
|
||||
old_rule: Final = await repository.save_rule(
|
||||
UsageAlertRuleInput(
|
||||
name="Earlier day", scope_type="user", scope_id=owner, metric="requests", direction="increase"
|
||||
),
|
||||
owner,
|
||||
)
|
||||
assert old_rule is not None
|
||||
old: Final = (
|
||||
await repository.record_observations(
|
||||
old_rule,
|
||||
(
|
||||
UsageAlertObservation(
|
||||
entity_id=owner,
|
||||
day=day - timedelta(days=1),
|
||||
observed=420,
|
||||
baseline=120,
|
||||
threshold=360,
|
||||
breached=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
)[0]
|
||||
other_rule: Final = await repository.save_rule(
|
||||
UsageAlertRuleInput(
|
||||
name="Other resource", scope_type="user", scope_id=outsider, metric="requests", direction="increase"
|
||||
),
|
||||
outsider,
|
||||
)
|
||||
assert other_rule is not None
|
||||
other: Final = (
|
||||
await repository.record_observations(
|
||||
other_rule,
|
||||
(
|
||||
UsageAlertObservation(
|
||||
entity_id=outsider,
|
||||
day=day,
|
||||
observed=420,
|
||||
baseline=120,
|
||||
threshold=360,
|
||||
breached=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
)[0]
|
||||
params: Final = {"scope_type": "user", "scope_id": owner, "status": "open", "limit": "1"}
|
||||
try:
|
||||
response: Final = gateway.request("GET", "/usage/alert-groups", key=owner_key, params=params)
|
||||
assert response.status_code == 200, response.text
|
||||
page: Final = UsageAlertGroupList.model_validate(response.json())
|
||||
assert page.total == 2 and len(page.data) == 1
|
||||
group: Final = page.data[0]
|
||||
assert group.day == day and group.scope_label == "Checkout operator"
|
||||
assert group.incident_count == 30 and len(group.metrics) == 4
|
||||
assert {metric.metric: metric.incident_count for metric in group.metrics} == {
|
||||
"requests": 27,
|
||||
"spend": 1,
|
||||
"errors": 1,
|
||||
"latency": 1,
|
||||
}
|
||||
assert all(metric.representative.id in {incident.id for incident in incidents} for metric in group.metrics)
|
||||
assert group.activity_path in {incident.activity_path for incident in incidents}
|
||||
second: Final = UsageAlertGroupList.model_validate(
|
||||
gateway.request("GET", "/usage/alert-groups", key=owner_key, params={**params, "offset": "1"}).json()
|
||||
)
|
||||
assert second.total == 2 and second.data[0].day == old.day
|
||||
empty: Final = UsageAlertGroupList.model_validate(
|
||||
gateway.request("GET", "/usage/alert-groups", key=owner_key, params={**params, "offset": "2"}).json()
|
||||
)
|
||||
assert empty.total == 2 and empty.data == ()
|
||||
history: Final = gateway.request(
|
||||
"GET",
|
||||
"/usage/alerts",
|
||||
key=owner_key,
|
||||
params={**params, "start_date": day.isoformat(), "end_date": day.isoformat(), "metric": "requests"},
|
||||
)
|
||||
assert history.status_code == 200 and history.json()["total"] == 27
|
||||
await repository.set_status(incidents[0].id, "dismissed")
|
||||
body: Final = {
|
||||
"scope_type": "user",
|
||||
"scope_id": owner,
|
||||
"day": day.isoformat(),
|
||||
"expected_status": "open",
|
||||
"status": "resolved",
|
||||
}
|
||||
for forbidden in (outsider_key, inference_key):
|
||||
assert gateway.request("GET", "/usage/alert-groups", key=forbidden, params=params).status_code == 403
|
||||
for forbidden in (outsider_key, inference_key, viewer_key):
|
||||
assert gateway.request("PATCH", "/usage/alert-groups", body, key=forbidden).status_code == 403
|
||||
assert gateway.request("GET", "/usage/alert-groups", key=viewer_key, params=params).status_code == 200
|
||||
updated: Final = gateway.request("PATCH", "/usage/alert-groups", body, key=owner_key)
|
||||
assert updated.status_code == 200 and updated.json()["updated_count"] == 29
|
||||
assert gateway.request("PATCH", "/usage/alert-groups", body, key=owner_key).json()["updated_count"] == 0
|
||||
assert await repository.get_alert(old.id) == old
|
||||
assert await repository.get_alert(other.id) == other
|
||||
dismissed: Final = await repository.get_alert(incidents[0].id)
|
||||
assert dismissed is not None and dismissed.status == "dismissed" and dismissed.baseline_value == 0
|
||||
resolved: Final = await repository.get_alert(incidents[-1].id)
|
||||
assert resolved == incidents[-1].model_copy(update={"status": "resolved"})
|
||||
assert (
|
||||
gateway.request(
|
||||
"GET", "/usage/alert-groups", params={"scope_type": "user", "scope_id": "x' OR 1=1 --"}
|
||||
).json()["total"]
|
||||
== 0
|
||||
)
|
||||
assert (
|
||||
gateway.request(
|
||||
"GET", "/usage/alert-groups", params={"start_date": "2026-11-05", "end_date": "2026-11-04"}
|
||||
).status_code
|
||||
== 422
|
||||
)
|
||||
finally:
|
||||
for rule in (*rules, old_rule, other_rule):
|
||||
if rule is not None:
|
||||
await repository.delete_rule(rule.id)
|
||||
deleted: Final = await repository.list_groups(scope_type="user", scope_id=owner)
|
||||
assert deleted.total == 2 and all(group.scope_label is None and group.scope_id == owner for group in deleted.data)
|
||||
|
||||
|
||||
@pytest.mark.asyncio(loop_scope="session")
|
||||
async def test_default_rule_summary_and_kind_filters_do_not_require_loading_every_rule(
|
||||
gateway: Gateway, db: ManagementDatabase
|
||||
) -> None:
|
||||
repository: Final = UsageAlertsRepository.from_prisma(db.prisma)
|
||||
default: Final = UsageAlertRuleInput(
|
||||
name="Built-in daily model requests", scope_type="model", metric="requests", direction="increase"
|
||||
)
|
||||
await repository.ensure_default_rules((default,))
|
||||
custom: Final = await repository.save_rule(
|
||||
default.model_copy(update={"scope_id": f"model-{uuid4().hex}"}), "qa-admin"
|
||||
)
|
||||
assert custom is not None
|
||||
try:
|
||||
all_rules: Final = await repository.list_rules(scope_type="model", limit=1)
|
||||
builtins: Final = await repository.list_rules(scope_type="model", limit=1, kind="default")
|
||||
customs: Final = await repository.list_rules(scope_type="model", limit=1, kind="custom")
|
||||
assert builtins.total >= 1 and all(rule.is_default for rule in builtins.data)
|
||||
assert customs.total >= 1 and all(not rule.is_default for rule in customs.data)
|
||||
assert all_rules.total == builtins.total + customs.total
|
||||
assert all_rules.default_summary == builtins.default_summary == customs.default_summary
|
||||
assert builtins.default_summary.total == builtins.total
|
||||
assert (
|
||||
sum(builtins.default_summary.evaluation_status_counts.model_dump().values())
|
||||
== builtins.default_summary.enabled
|
||||
)
|
||||
assert (await repository.list_rules(scope_type="model", kind="default", offset=builtins.total)).data == ()
|
||||
stored: Final = await repository.get_rule(builtins.data[0].id)
|
||||
assert stored is not None and stored.is_default
|
||||
filtered: Final = gateway.request(
|
||||
"GET", "/usage/alert-rules", params={"scope_type": "model", "kind": "custom", "limit": "1"}
|
||||
)
|
||||
assert filtered.status_code == 200 and filtered.json()["total"] == customs.total
|
||||
assert all(not rule["is_default"] for rule in filtered.json()["data"])
|
||||
configuration: Final = stored.model_dump(mode="json", include=frozenset(UsageAlertRuleInput.model_fields))
|
||||
path: Final = f"/usage/alert-rules/{stored.id}"
|
||||
for change in (
|
||||
{"scope_type": "key"},
|
||||
{"scope_id": "model-example"},
|
||||
{"metric": "requests" if stored.metric != "requests" else "spend"},
|
||||
{"direction": "increase" if stored.direction == "decrease" else "decrease"},
|
||||
):
|
||||
assert gateway.request("PUT", path, {**configuration, **change}).status_code == 422
|
||||
assert await repository.get_rule(stored.id) == stored
|
||||
try:
|
||||
updated: Final = gateway.request("PUT", path, {**configuration, "min_value": stored.min_value + 1})
|
||||
assert updated.status_code == 200 and updated.json()["is_default"] is True
|
||||
assert updated.json()["min_value"] == stored.min_value + 1
|
||||
finally:
|
||||
assert gateway.request("PUT", path, configuration).status_code == 200
|
||||
custom_input: Final = custom.model_dump(mode="json", include=frozenset(UsageAlertRuleInput.model_fields))
|
||||
assert (
|
||||
gateway.request("PUT", f"/usage/alert-rules/{custom.id}", {**custom_input, "metric": "latency"}).status_code
|
||||
== 200
|
||||
)
|
||||
finally:
|
||||
await repository.delete_rule(custom.id)
|
||||
|
||||
|
||||
@pytest.mark.asyncio(loop_scope="session")
|
||||
async def test_reported_model_search_preserves_exact_identity_and_cursor(
|
||||
gateway: Gateway, db: ManagementDatabase
|
||||
) -> None:
|
||||
repository: Final = UsageAlertsRepository.from_prisma(db.prisma)
|
||||
prefix: Final = f"reported-{uuid4().hex}"
|
||||
initial: Final = await repository.list_reported_models(prefix, None, 1)
|
||||
assert initial.data == ()
|
||||
models: Final = (f"{prefix}/first", f"{prefix}/literal%quote'", f"{prefix}/second")
|
||||
try:
|
||||
for index, model in enumerate((*models, models[0])):
|
||||
await db.prisma.writer_db.execute_raw(
|
||||
'INSERT INTO "LiteLLM_DailyUserSpend" (id,date,api_key,model,updated_at) VALUES ($1,$2,$3,$4,CURRENT_TIMESTAMP)',
|
||||
f"{prefix}-{index}",
|
||||
initial.end_date.isoformat(),
|
||||
prefix,
|
||||
model,
|
||||
)
|
||||
await db.prisma.writer_db.execute_raw(
|
||||
'INSERT INTO "LiteLLM_DailyUserSpend" (id,date,api_key,model,updated_at) VALUES ($1,$2,$3,$4,CURRENT_TIMESTAMP)',
|
||||
f"{prefix}-old",
|
||||
(initial.start_date - timedelta(days=1)).isoformat(),
|
||||
prefix,
|
||||
f"{prefix}/old",
|
||||
)
|
||||
response: Final = gateway.request("GET", "/usage/alert-models", params={"search": prefix, "limit": "1"})
|
||||
assert response.status_code == 200, response.text
|
||||
first: Final = UsageAlertModelList.model_validate(response.json())
|
||||
assert first.data[0].scope_id == models[0] and first.next_cursor == models[0]
|
||||
second: Final = await repository.list_reported_models(prefix, first.next_cursor, 1)
|
||||
third: Final = await repository.list_reported_models(prefix, second.next_cursor, 1)
|
||||
assert second.data[0].scope_id == models[1] and third.data[0].scope_id == models[2]
|
||||
assert third.next_cursor is None
|
||||
literal: Final = await repository.list_reported_models(f"{prefix}/literal%quote'", None, 20)
|
||||
assert tuple(option.scope_id for option in literal.data) == (models[1],)
|
||||
assert literal.data[0].label == models[1]
|
||||
with gateway.scenario() as scenario:
|
||||
user: Final = scenario.user(user_role="internal_user")
|
||||
key: Final = scenario.key(user_id=user, key_type="management")
|
||||
assert gateway.request("GET", "/usage/alert-models", key=key).status_code == 403
|
||||
finally:
|
||||
await db.prisma.writer_db.execute_raw('DELETE FROM "LiteLLM_DailyUserSpend" WHERE api_key=$1', prefix)
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ from typing import Final
|
|||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from litellm.types.proxy.usage_alerts import UsageAlertRuleInput
|
||||
from litellm.types.proxy.usage_alerts import UsageAlertGroupUpdate, UsageAlertRuleInput
|
||||
|
||||
RULE: Final = {
|
||||
"name": "Daily spend",
|
||||
|
|
@ -41,3 +41,21 @@ def test_zero_request_floor_and_daily_drop_multiplier_are_supported() -> None:
|
|||
assert rule.direction == "decrease"
|
||||
assert rule.multiplier == 2
|
||||
assert rule.min_requests == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"changes",
|
||||
[{"scope_id": None}, {"scope_type": "deployment"}, {"status": "open"}, {"status": "unknown"}],
|
||||
)
|
||||
def test_group_updates_require_exact_resource_and_a_different_valid_status(changes: dict[str, object]) -> None:
|
||||
with pytest.raises(ValidationError):
|
||||
UsageAlertGroupUpdate.model_validate(
|
||||
{
|
||||
"scope_type": "team",
|
||||
"scope_id": "team-example",
|
||||
"day": "2026-11-04",
|
||||
"expected_status": "open",
|
||||
"status": "resolved",
|
||||
**changes,
|
||||
}
|
||||
)
|
||||
|
|
|
|||
285
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
285
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
|
|
@ -18834,6 +18834,41 @@ export interface paths {
|
|||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/usage/alert-groups": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
/** List Usage Alert Groups */
|
||||
get: operations["list_usage_alert_groups_usage_alert_groups_get"];
|
||||
put?: never;
|
||||
post?: never;
|
||||
delete?: never;
|
||||
options?: never;
|
||||
head?: never;
|
||||
/** Update Usage Alert Group */
|
||||
patch: operations["update_usage_alert_group_usage_alert_groups_patch"];
|
||||
trace?: never;
|
||||
};
|
||||
"/usage/alert-models": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
/** List Usage Alert Models */
|
||||
get: operations["list_usage_alert_models_usage_alert_models_get"];
|
||||
put?: never;
|
||||
post?: never;
|
||||
delete?: never;
|
||||
options?: never;
|
||||
head?: never;
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/usage/alert-rules": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
|
|
@ -50472,6 +50507,120 @@ export interface components {
|
|||
/** Threshold */
|
||||
threshold: number;
|
||||
};
|
||||
/** UsageAlertDefaultSummary */
|
||||
UsageAlertDefaultSummary: {
|
||||
/**
|
||||
* Enabled
|
||||
* @default 0
|
||||
*/
|
||||
enabled: number;
|
||||
evaluation_status_counts?: components["schemas"]["UsageAlertEvaluationCounts"];
|
||||
/**
|
||||
* Total
|
||||
* @default 0
|
||||
*/
|
||||
total: number;
|
||||
};
|
||||
/** UsageAlertEvaluationCounts */
|
||||
UsageAlertEvaluationCounts: {
|
||||
/**
|
||||
* Complete
|
||||
* @default 0
|
||||
*/
|
||||
complete: number;
|
||||
/**
|
||||
* Insufficient Coverage
|
||||
* @default 0
|
||||
*/
|
||||
insufficient_coverage: number;
|
||||
/**
|
||||
* Insufficient History
|
||||
* @default 0
|
||||
*/
|
||||
insufficient_history: number;
|
||||
/**
|
||||
* Pending
|
||||
* @default 0
|
||||
*/
|
||||
pending: number;
|
||||
};
|
||||
/** UsageAlertGroup */
|
||||
UsageAlertGroup: {
|
||||
/** Activity Path */
|
||||
activity_path: string;
|
||||
/**
|
||||
* Day
|
||||
* Format: date
|
||||
*/
|
||||
day: string;
|
||||
/** Id */
|
||||
id: string;
|
||||
/** Incident Count */
|
||||
incident_count: number;
|
||||
/** Metrics */
|
||||
metrics: components["schemas"]["UsageAlertGroupMetric"][];
|
||||
/** Scope Id */
|
||||
scope_id: string | null;
|
||||
/** Scope Label */
|
||||
scope_label: string | null;
|
||||
/**
|
||||
* Scope Type
|
||||
* @enum {string}
|
||||
*/
|
||||
scope_type: "deployment" | "key" | "user" | "team" | "model";
|
||||
};
|
||||
/** UsageAlertGroupList */
|
||||
UsageAlertGroupList: {
|
||||
/** Data */
|
||||
data: components["schemas"]["UsageAlertGroup"][];
|
||||
/** Limit */
|
||||
limit: number;
|
||||
/** Offset */
|
||||
offset: number;
|
||||
/** Total */
|
||||
total: number;
|
||||
};
|
||||
/** UsageAlertGroupMetric */
|
||||
UsageAlertGroupMetric: {
|
||||
/** Incident Count */
|
||||
incident_count: number;
|
||||
/**
|
||||
* Metric
|
||||
* @enum {string}
|
||||
*/
|
||||
metric: "spend" | "requests" | "latency" | "errors";
|
||||
representative: components["schemas"]["UsageAlert"];
|
||||
};
|
||||
/** UsageAlertGroupUpdate */
|
||||
UsageAlertGroupUpdate: {
|
||||
/**
|
||||
* Day
|
||||
* Format: date
|
||||
*/
|
||||
day: string;
|
||||
/**
|
||||
* Expected Status
|
||||
* @enum {string}
|
||||
*/
|
||||
expected_status: "open" | "resolved" | "dismissed";
|
||||
/** Scope Id */
|
||||
scope_id?: string | null;
|
||||
/**
|
||||
* Scope Type
|
||||
* @enum {string}
|
||||
*/
|
||||
scope_type: "deployment" | "key" | "user" | "team" | "model";
|
||||
/**
|
||||
* Status
|
||||
* @enum {string}
|
||||
*/
|
||||
status: "open" | "resolved" | "dismissed";
|
||||
};
|
||||
/** UsageAlertGroupUpdateResult */
|
||||
UsageAlertGroupUpdateResult: {
|
||||
/** Updated Count */
|
||||
updated_count: number;
|
||||
};
|
||||
/** UsageAlertList */
|
||||
UsageAlertList: {
|
||||
/** Data */
|
||||
|
|
@ -50483,6 +50632,32 @@ export interface components {
|
|||
/** Total */
|
||||
total: number;
|
||||
};
|
||||
/** UsageAlertModelList */
|
||||
UsageAlertModelList: {
|
||||
/** Data */
|
||||
data: components["schemas"]["UsageAlertModelOption"][];
|
||||
/**
|
||||
* End Date
|
||||
* Format: date
|
||||
*/
|
||||
end_date: string;
|
||||
/** Limit */
|
||||
limit: number;
|
||||
/** Next Cursor */
|
||||
next_cursor: string | null;
|
||||
/**
|
||||
* Start Date
|
||||
* Format: date
|
||||
*/
|
||||
start_date: string;
|
||||
};
|
||||
/** UsageAlertModelOption */
|
||||
UsageAlertModelOption: {
|
||||
/** Label */
|
||||
label: string;
|
||||
/** Scope Id */
|
||||
scope_id: string;
|
||||
};
|
||||
/** UsageAlertRule */
|
||||
UsageAlertRule: {
|
||||
/**
|
||||
|
|
@ -50520,6 +50695,11 @@ export interface components {
|
|||
evaluation_status: "pending" | "complete" | "insufficient_coverage" | "insufficient_history";
|
||||
/** Id */
|
||||
id: string;
|
||||
/**
|
||||
* Is Default
|
||||
* @default false
|
||||
*/
|
||||
is_default: boolean;
|
||||
/** Last Evaluated Day */
|
||||
last_evaluated_day?: string | null;
|
||||
/**
|
||||
|
|
@ -50635,6 +50815,7 @@ export interface components {
|
|||
UsageAlertRuleList: {
|
||||
/** Data */
|
||||
data: components["schemas"]["UsageAlertRule"][];
|
||||
default_summary?: components["schemas"]["UsageAlertDefaultSummary"];
|
||||
/** Limit */
|
||||
limit: number;
|
||||
/** Offset */
|
||||
|
|
@ -78561,6 +78742,109 @@ export interface operations {
|
|||
};
|
||||
};
|
||||
};
|
||||
list_usage_alert_groups_usage_alert_groups_get: {
|
||||
parameters: {
|
||||
query?: {
|
||||
scope_type?: ("deployment" | "key" | "user" | "team" | "model") | null;
|
||||
scope_id?: string | null;
|
||||
status?: ("open" | "resolved" | "dismissed") | null;
|
||||
start_date?: string | null;
|
||||
end_date?: string | null;
|
||||
limit?: number;
|
||||
offset?: number;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody?: never;
|
||||
responses: {
|
||||
/** @description Successful Response */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["UsageAlertGroupList"];
|
||||
};
|
||||
};
|
||||
/** @description Validation Error */
|
||||
422: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["HTTPValidationError"];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
update_usage_alert_group_usage_alert_groups_patch: {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody: {
|
||||
content: {
|
||||
"application/json": components["schemas"]["UsageAlertGroupUpdate"];
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Successful Response */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["UsageAlertGroupUpdateResult"];
|
||||
};
|
||||
};
|
||||
/** @description Validation Error */
|
||||
422: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["HTTPValidationError"];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
list_usage_alert_models_usage_alert_models_get: {
|
||||
parameters: {
|
||||
query?: {
|
||||
search?: string | null;
|
||||
cursor?: string | null;
|
||||
limit?: number;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody?: never;
|
||||
responses: {
|
||||
/** @description Successful Response */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["UsageAlertModelList"];
|
||||
};
|
||||
};
|
||||
/** @description Validation Error */
|
||||
422: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["HTTPValidationError"];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
list_usage_alert_rules_usage_alert_rules_get: {
|
||||
parameters: {
|
||||
query?: {
|
||||
|
|
@ -78568,6 +78852,7 @@ export interface operations {
|
|||
scope_id?: string | null;
|
||||
limit?: number;
|
||||
offset?: number;
|
||||
kind?: "all" | "default" | "custom";
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue