Keeps an explicit empty messages list on the pre-existing path (default
model, provider validation error) instead of dropping the routing
decision and surfacing a misleading tags 401.
Auto-router strategy hook returned None whenever the request carried
input instead of messages, so tagged /v1/responses requests (Codex CLI)
never picked a tier and tag filtering left nothing to route to. Resolve
input through the shared prompt-template helper before matching routes.
* feat(proxy): add /team/daily/activity/aggregated and use it in the Usage UI
The Team Usage tab drained row-paginated pages client side, which painted
newest days first and drew duplicate bars when a day's rows straddled a
page boundary. Serve the whole range in one SQL GROUPING SETS pass instead:
the aggregated query gains optional per-entity rollup levels (entity as the
most-significant GROUPING bit) so breakdown.entities keeps per-team spend,
aliases, and per-key splits. The endpoint shares the paginated route's
scoping via _resolve_team_daily_activity_scope, accepts the timezone the UI
already sends, and the api_key filter now takes a list so non-admin member
scoping works. The dashboard tries the aggregated endpoint first and falls
back to page draining on failure.
* chore: ratchet B008 budget down by the endpoint converted to Annotated Depends
* chore: keep mutable-ok suppressions on their annotation lines after formatting
* fix(proxy): reject malformed or over-wide ranges on team aggregated activity
The aggregated endpoint has no pagination bounding its work, so validate
start_date and end_date as real dates and cap the span at 400 days. The
dashboard's widest presets fit well inside the cap, and an over-cap range
falls back to the paginated flow. Also trim implementation comments that
restated the grouping-set code.
* fix(proxy): parse aggregated range bounds as UTC to satisfy DTZ007
* refactor(proxy): fetch entity rollups with a companion query instead of extending the main one
The entity-as-extra-GROUPING-bit approach made the bitmask layout
mode-dependent: the same constant meant (date) for normal rows and
(date, entity) for entity rows, disambiguated by masking. Split it out:
the shared WHERE builder feeds both the untouched main query and a small
per-entity rollup query keyed by GROUPING(api_key), run concurrently, and
a fold writes breakdown.entities onto the built response.
* refactor(proxy): share the daily-activity error and entity-metadata shapes
The type-discipline ceiling for LIT002 ratcheted down on staging, so the new
aggregated endpoint had to stop hand-rolling collections the codebase already
builds elsewhere. Funnel the `{"error": ...}` detail through one construction
site, turn the range validator into an error-as-value, reuse a single
entity-metadata lookup for both breakdown paths, and widen
get_api_key_metadata to any set so callers stop copying frozensets.
* feat(complexity_router): plan-mode tier floor for coding-agent clients
Claude Code and Copilot signal plan mode only through client-injected prompt
text, which the ask-extraction path deliberately strips, so the router could
never see it. Detect the sentinels on the raw wire body and route those
requests to at least plan_mode_min_tier.
The floor is raise-only and transient: classifier results above it still win,
it overrides a session-affinity pin only on turns carrying the sentinel
without rewriting the pin, and plan_mode decisions are not pinnable, so the
first turn after plan mode exits routes as if plan mode had never happened.
Classification is skipped when the floor is the top configured tier. On
adaptive routers the floor rides _soft_floor_pick as a hard_floor that
excludes below-floor candidates, closing the adaptive_eligible=all gap where
a request classified at or above the floor could still route below it.
Detection is staleness-aware: only leading system content and the newest-ask
tail count, so sentinels surviving in history after plan mode exits, built-in
or operator-supplied, never fire. Custom tier sets are supported with
severity from the tier_definitions list order, same as keyword_tier_rules.
Off by default; decisions are recorded with the new plan_mode cause and the
matched sentinel in matched_keyword
* fix(complexity_router): gate pin writes and the failure exit on sentinel presence, not the floor binding
A plan-mode turn classified at or above the floor keeps its ordinary cause,
but pinning it would carry a plan-mode-shaped choice past plan mode's exit
(on adaptive routers the hard floor constrained that pick), so no
sentinel-carrying turn writes the session pin. The default_model failure
exit is skipped for sentinel turns for the same reason: default_model's
placeholder tier can equal the floor while default_model itself sits in no
pool the floor can vouch for
* fix(proxy): strip callback credentials from the auth object stamped into request metadata
* style(proxy): drop the restating half of the stamp-site comment
* test(proxy): pin that the stamped auth copy carries header-derived identity
GET /guardrails/usage/overview and GET /guardrails/usage/detail/{id} 500ed on a
database that has not applied 20260817143646_add_daily_guardrail_usage_units yet
(pip installs on litellm-proxy-extras 0.4.86 with DISABLE_SCHEMA_UPDATE=true).
Both endpoints now return their metrics with empty units and log one warning
until the migration lands.
The daily guardrail metrics and usage-unit upserts are non-idempotent
increments, but the retry loop re-sent every failed row on any exception.
An ambiguous post-send failure such as a read timeout after the write had
already committed therefore stacked a second increment and inflated the
billable unit totals served by the guardrail usage endpoints.
Retry only DB_RETRY_SAFE_ERROR_TYPES (httpx.ConnectError), the same rule
the spend writer and autorouter rollup use for increment upserts, and log
any other failure once as terminal for that row while the rest of the
batch still lands.
Follows up #37225
A pass that raises (a missing Slack webhook, say) now waits the daily interval instead of logging the
same exception every 30 seconds, and the Admin UI alerting settings list the new alert type so it can be
toggled like the others
The trailing-slash normalization test used gateway.litellm-sandbox.ai as
its base URL. Swap it for gateway.example.com so the test file does not
reference a real-looking hostname. The test is fully mocked, so the host
value has no effect on what is exercised.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
resolve_fireworks_resource_name prefixes bare names with
accounts/fireworks/models/ (or routers/ for *-fast). Azure AI Foundry
hosts Fireworks models under deployment ids like FW-Kimi-K3; rewriting
those yields 404 DeploymentNotFound.
Leave names that already start with FW- unchanged. Native Fireworks
short names still get the accounts/ path.
Co-authored-by: Cursor <cursoragent@cursor.com>
A transient DB error during the spend log flush dropped that batch's guardrail
metrics and usage unit rows for good. Retry only the rows that failed, up to 3
times with 1s/2s/4s backoff, mirroring the daily spend writer, and inject the
sleep so tests stay fast. Lowers the lint budgets the refactor freed up
An empty pass no longer holds the daily lock, a False lock claim (held or redis
error) is retried on the next 30 second poll instead of sleeping a day, and a
sent alert is stamped in the shared cache for a day so sibling pods and restarts
stay quiet
The flush and the usage endpoints summed units with a scan per distinct key,
quadratic in rows times keys; group sorted rows instead. Skip payloads without
a request_id like the metrics path, type the flush key as a NamedTuple, and drop
the (guardrail_id, date) index that the primary key already covers
* fix(shadow_eval): copy messages before router call and raise judge output cap
* fix(shadow_eval): lead failure detail with location and pin post-failure continuation
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.
Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
Per-row guards in the daily metrics and usage unit flush so a single DB error no longer drops the rest of the batch, plus removal of narrating comments flagged in review
The Noma guardrail sends the conversation to the scanner in `inputs`. It
also forwarded `request_data` whole, which repeats that same conversation
under `messages` (or `input` on the responses API), and attached
`logging_obj.model_call_details`, which repeats it a third time.
For image-heavy calls that duplication is most of the request. A
production scan of a request carrying base64 images measured 100MB total,
of which 94.8MB was `request_data` against 5.1MB of `inputs` - the proxy
was uploading ~95% redundant bytes, and paying to serialize them.
Drop `messages` and `input` from `request_data` and from
`model_call_details`. This is a denylist rather than an allowlist on
purpose: every other key is still forwarded untouched, so a scanner-side
change that starts reading a new `request_data` key needs no matching
release of this hook. The removed keys are ones the scanner never reads -
it takes context only from metadata, litellm_metadata,
provider_specific_header, litellm_session_id/trace_id/call_id, stream,
response/responses ids, and litellm_logging_obj.complete_streaming_response,
all of which still pass through.
The conversation still reaches the scanner in full via `inputs`, so no
detection coverage changes.
Trimming happens before serialization, so the duplicate is never encoded.
Existing payload tests asserted the duplication; they now assert the trim
while keeping what they originally guarded - deep-copy semantics and the
unpicklable-object (uvloop.Loop) regression.
* fix(mcp): scope authorization server issuer
Generated with AI
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(mcp): keep the bare-origin issuer when no server was named
The scoped issuer must key off whether the request actually carried a server
name. _build_oauth_authorization_server_response rebinds mcp_server_name when
root discovery resolves the single configured OAuth2 server, so gating on the
rebound value also scoped /.well-known/openid-configuration, whose document is
served from the bare origin and whose issuer must stay the bare origin
Adds the named-server regression test for the reported mismatch, restores the
bare-origin assertion, and covers the OIDC document
* test(mcp): type the delegate_auth_to_upstream helper parameter
* refactor(mcp): bind the discovery issuer to a local before building the response
---------
Co-authored-by: Irosh <15094153+irosh-colombage-ZocDoc2@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
The field itself landed on staging via 0c5c9c79d7; these are the regression tests from PR #31435 for the retrieval-facing half.
(cherry picked from commit a9a322d63f6d4658b1f28d1622335775e94736a4)
