* feat(proxy): add /team/daily/activity/aggregated and use it in the Usage UI
The Team Usage tab drained row-paginated pages client side, which painted
newest days first and drew duplicate bars when a day's rows straddled a
page boundary. Serve the whole range in one SQL GROUPING SETS pass instead:
the aggregated query gains optional per-entity rollup levels (entity as the
most-significant GROUPING bit) so breakdown.entities keeps per-team spend,
aliases, and per-key splits. The endpoint shares the paginated route's
scoping via _resolve_team_daily_activity_scope, accepts the timezone the UI
already sends, and the api_key filter now takes a list so non-admin member
scoping works. The dashboard tries the aggregated endpoint first and falls
back to page draining on failure.
* chore: ratchet B008 budget down by the endpoint converted to Annotated Depends
* chore: keep mutable-ok suppressions on their annotation lines after formatting
* fix(proxy): reject malformed or over-wide ranges on team aggregated activity
The aggregated endpoint has no pagination bounding its work, so validate
start_date and end_date as real dates and cap the span at 400 days. The
dashboard's widest presets fit well inside the cap, and an over-cap range
falls back to the paginated flow. Also trim implementation comments that
restated the grouping-set code.
* fix(proxy): parse aggregated range bounds as UTC to satisfy DTZ007
* refactor(proxy): fetch entity rollups with a companion query instead of extending the main one
The entity-as-extra-GROUPING-bit approach made the bitmask layout
mode-dependent: the same constant meant (date) for normal rows and
(date, entity) for entity rows, disambiguated by masking. Split it out:
the shared WHERE builder feeds both the untouched main query and a small
per-entity rollup query keyed by GROUPING(api_key), run concurrently, and
a fold writes breakdown.entities onto the built response.
* refactor(proxy): share the daily-activity error and entity-metadata shapes
The type-discipline ceiling for LIT002 ratcheted down on staging, so the new
aggregated endpoint had to stop hand-rolling collections the codebase already
builds elsewhere. Funnel the `{"error": ...}` detail through one construction
site, turn the range validator into an error-as-value, reuse a single
entity-metadata lookup for both breakdown paths, and widen
get_api_key_metadata to any set so callers stop copying frozensets.
* fix(proxy): strip callback credentials from the auth object stamped into request metadata
* style(proxy): drop the restating half of the stamp-site comment
* test(proxy): pin that the stamped auth copy carries header-derived identity
GET /guardrails/usage/overview and GET /guardrails/usage/detail/{id} 500ed on a
database that has not applied 20260817143646_add_daily_guardrail_usage_units yet
(pip installs on litellm-proxy-extras 0.4.86 with DISABLE_SCHEMA_UPDATE=true).
Both endpoints now return their metrics with empty units and log one warning
until the migration lands.
The daily guardrail metrics and usage-unit upserts are non-idempotent
increments, but the retry loop re-sent every failed row on any exception.
An ambiguous post-send failure such as a read timeout after the write had
already committed therefore stacked a second increment and inflated the
billable unit totals served by the guardrail usage endpoints.
Retry only DB_RETRY_SAFE_ERROR_TYPES (httpx.ConnectError), the same rule
the spend writer and autorouter rollup use for increment upserts, and log
any other failure once as terminal for that row while the rest of the
batch still lands.
Follows up #37225
The trailing-slash normalization test used gateway.litellm-sandbox.ai as
its base URL. Swap it for gateway.example.com so the test file does not
reference a real-looking hostname. The test is fully mocked, so the host
value has no effect on what is exercised.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
A transient DB error during the spend log flush dropped that batch's guardrail
metrics and usage unit rows for good. Retry only the rows that failed, up to 3
times with 1s/2s/4s backoff, mirroring the daily spend writer, and inject the
sleep so tests stay fast. Lowers the lint budgets the refactor freed up
The flush and the usage endpoints summed units with a scan per distinct key,
quadratic in rows times keys; group sorted rows instead. Skip payloads without
a request_id like the metrics path, type the flush key as a NamedTuple, and drop
the (guardrail_id, date) index that the primary key already covers
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.
Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
Per-row guards in the daily metrics and usage unit flush so a single DB error no longer drops the rest of the batch, plus removal of narrating comments flagged in review
The Noma guardrail sends the conversation to the scanner in `inputs`. It
also forwarded `request_data` whole, which repeats that same conversation
under `messages` (or `input` on the responses API), and attached
`logging_obj.model_call_details`, which repeats it a third time.
For image-heavy calls that duplication is most of the request. A
production scan of a request carrying base64 images measured 100MB total,
of which 94.8MB was `request_data` against 5.1MB of `inputs` - the proxy
was uploading ~95% redundant bytes, and paying to serialize them.
Drop `messages` and `input` from `request_data` and from
`model_call_details`. This is a denylist rather than an allowlist on
purpose: every other key is still forwarded untouched, so a scanner-side
change that starts reading a new `request_data` key needs no matching
release of this hook. The removed keys are ones the scanner never reads -
it takes context only from metadata, litellm_metadata,
provider_specific_header, litellm_session_id/trace_id/call_id, stream,
response/responses ids, and litellm_logging_obj.complete_streaming_response,
all of which still pass through.
The conversation still reaches the scanner in full via `inputs`, so no
detection coverage changes.
Trimming happens before serialization, so the duplicate is never encoded.
Existing payload tests asserted the duplication; they now assert the trim
while keeping what they originally guarded - deep-copy semantics and the
unpicklable-object (uvloop.Loop) regression.
* fix(mcp): scope authorization server issuer
Generated with AI
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(mcp): keep the bare-origin issuer when no server was named
The scoped issuer must key off whether the request actually carried a server
name. _build_oauth_authorization_server_response rebinds mcp_server_name when
root discovery resolves the single configured OAuth2 server, so gating on the
rebound value also scoped /.well-known/openid-configuration, whose document is
served from the bare origin and whose issuer must stay the bare origin
Adds the named-server regression test for the reported mismatch, restores the
bare-origin assertion, and covers the OIDC document
* test(mcp): type the delegate_auth_to_upstream helper parameter
* refactor(mcp): bind the discovery issuer to a local before building the response
---------
Co-authored-by: Irosh <15094153+irosh-colombage-ZocDoc2@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Registry-swap reconciliation used bool(server.url) while registration uses _requires_oauth_discovery, dropping slots for issuer-anchored servers without a url. The preemptive 401 loop awaited discovery before the stamped client_credentials continue, so a deferred discovery failure could 503 requests whose challenge decision never reads metadata
* fix(proxy): cache tag-name registry so unregistered request tags skip Postgres
Request tags are free-form attribution labels, so most have no LiteLLM_TagTable
row. get_tag_objects_batch never cached that absence: every tagged request ran
a find_many that came back empty, and under Prisma pool contention those
per-request queries queued for minutes inside user_api_key_auth.
Cache the bounded set of registered tag names under one aggregate key with the
management-object TTL. Uncached request tags are filtered against it before any
per-tag DB fetch, so unregistered tags cost zero DB reads on a warm path. An
empty registry is cached as a valid answer; DB errors are not cached and fall
back to the per-tag lookup; tables past TAG_REGISTRY_MAX_SIZE cache an overflow
sentinel that disables filtering. Tag create/update/delete endpoints now evict
the registry and per-tag keys and publish cross-worker invalidation (they
previously evicted nothing). The per-tag write-back also gains the management
TTL it was missing, and the hand-built tag:{name} key strings are replaced with
a shared builder.
* fix(proxy): skip per-request end-user DB reads via restricted-id registry
Every request carrying a user id ran get_end_user_object, and with high-cardinality
auto-created end-user rows (hundreds of thousands of ids, all restriction fields
NULL) the per-pod cache missed on nearly every request, so each one paid a Postgres
find_unique that queued behind the Prisma pool during background-job bursts. True
misses were never cached, and unknown ids paid the read twice per request.
Cache the bounded set of end-user ids that carry any restriction (blocked, budget,
region, default model, or object permission) under one aggregate key with the
management-object TTL. When an id misses the per-id cache and is absent from a
usable registry, get_end_user_object returns None with zero DB reads; restricted
ids keep today's fetch-and-cache path. The skip is bypassed whenever
litellm.max_end_user_budget_id is set (default budgets make unrestricted rows
behaviorally distinct from missing rows), validate_end_user_id_in_db is on
(existence checks need the row), or the token carries end_user_max_budget from
custom auth (the row's recorded spend seeds the budget counter). Empty registries
cache as a valid answer, DB errors are never cached, and oversized tables cache an
overflow sentinel that disables filtering. Customer create/update/block/delete now
evict the registry and per-id keys and publish cross-worker invalidation (they
previously evicted nothing), and the per-id write-back gains the management TTL it
was missing so Redis entries no longer live forever.
* refactor(proxy): single generic registry loader with error sentinel and single-flight
Code review follow-ups on the two registry caches. Registry DB errors now cache
the overflow sentinel for a short REGISTRY_ERROR_NEGATIVE_CACHE_TTL window and
log at warning, so a degraded Postgres stops paying the failing registry scan on
every request on top of the per-id fallback. Cold registry loads are single-flight
per worker behind per-registry locks with a recheck after acquire, so a TTL expiry
no longer fans out one full-table scan per in-flight request. The tag and end-user
loaders collapse into one _load_bounded_registry with per-entity fetch closures,
and the triplicated evict-then-broadcast protocol becomes one evict_and_broadcast
helper beside publish_auth_cache_invalidation, shared by the tag, customer, and
project eviction paths.
* chore(lint): suppress fail-safe registry excepts and ratchet BLE001 budget
* docs(proxy): trim registry cache commentary to single-line why docstrings
* fix(lint): move tag fetch return to else block to satisfy TRY300 budget
* fix(guardrails): scan text on /guardrails/apply_guardrail for Azure Content Safety
The two Azure Content Safety guardrails never implemented apply_guardrail, so the
endpoint fell through to the base no-op and answered 200 with the caller's text
echoed back, having scanned nothing.
Implementing that method also flips the proxy's unified-vs-native dispatch, which
would move request traffic off these guardrails' own hooks. Add an opt-out that
keeps every lifecycle event on the native hooks, so only the endpoint changes.
* test(guardrails): cover the remaining native-hook opt-out dispatch sites
Adds regression tests for the parallel post-call path, the MCP post-call hook, and
the policy engine step, so every read of the opt-out flag fails when removed.
