The aggregated daily-activity queries build their WHERE clause as raw SQL, and
an empty entity list rendered as `"team_id" IN ()`, which Postgres rejects with
`syntax error at or near ")"`. Callers reach that state normally: a caller
without admin view and no explicit team_ids has its scope resolved to the teams
it belongs to, so anyone belonging to no teams, an org admin for instance, sent
an empty list and got a 500 back from /team/daily/activity/aggregated.
The paginated endpoint hands the same empty list to Prisma, which renders
`in: []` and matches nothing, so it kept returning 200 with an empty result set.
Emit FALSE for the empty case so the raw-SQL path lands on the same answer,
mirroring what the api_key filter a few lines below already does.
The fix covers both aggregated queries at once because they share one WHERE
clause builder.
true_passthrough and oauth_delegate forward the caller's own bearer and mint
nothing, so their sessions consume no discovered OAuth endpoint. The discovery
completeness gate still failed them closed with a 503 raised before the upstream
was ever contacted, which the tools/list fan-out swallowed into HTTP 200 with an
empty tool list. Any upstream that publishes no RFC 9728 metadata, including
every OpenAPI-backed internal API, was permanently unusable.
A failed discovery is now fatal only to servers whose OAuth flow the gateway
runs itself. Discovery still runs for the forwarding modes, so /authorize,
/token and /register keep resolving their endpoints, and each keeps its own 400
when they are missing.
MCPServer.is_client_forwarded_token now owns the mode pair that five call sites
spelled inline, one of which had already named it is_client_forwarded_mode.
* feat(bedrock): add a config toggle to disable agent-runtime pass-through
The /bedrock pass-through dispatches agents, knowledge bases, flows, rerank,
retrieveAndGenerate, generateQuery and optimize-prompt to bedrock-agent-runtime,
so an operator who only wants to expose model invoke and converse has no way to
narrow that surface
Adds general_settings.disable_bedrock_agent_runtime_passthrough. When set, those
routes are rejected with a 403 before credentials are fetched or the request is
signed. Plain bedrock-runtime model pass-through is unaffected, and the setting
defaults to off, so existing deployments behave exactly as before
The branch is inverted to an early return for the non-agent-runtime case so the
toggle can reject outright instead of falling through to model extraction, which
would surface a confusing 400 about an unparseable model
* style(bedrock): drop redundant docstrings from the agent-runtime toggle
Both surfaces wrote zero bytes for the whole time-to-first-token, so an
intermediary with an idle read timeout drops a healthy connection before the
first token. They reached neither keepalive engine, which is what #37322 left
open.
The streaming assistants run spends that wait inside the awaited call that
produces its response, since create_response buffers the first chunk, so it
takes the same open_sse_before_first_byte seam the native routes use. The A2A
route only contacts the upstream agent once its body iterator is first pulled,
so nothing is awaited before the response exists and the gap has to be filled
from inside the stream instead; wrap_sse_stream_with_keepalive_pings already
does that and now takes the filler as a parameter, so A2A gets an SSE comment
its JSON-RPC clients discard rather than Anthropic's ping event.
Off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds.
A terminal HTTP failure partway through chunking now logs the summed usage
and cost of the ApplyGuardrail calls AWS already billed, mirroring the
blocked-chunk path.
Removes one named callback from a team and leaves the team's other callbacks
registered and firing. Before this, the only removal route was
POST /team/{team_id}/disable_logging, which clears every callback at once, so a
tenant sharing a team could not deregister its own integration
The handler filters metadata["logging"], keeps the survivors encrypted, refreshes
the cached team so the removal applies to keys that are already live, and emits a
redacted audit row, matching what the add and disable routes do
Resolves LIT-5161
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail
call AWS billed for the logical request, not just the blocking chunk.
Client-supplied metadata.standard_logging_guardrail_information is stripped
at the proxy boundary so callers cannot forge (even negative) guardrail
cost into spend, and guardrail_information_cost ignores negative or
non-finite entry costs as defense in depth. The cost map schema test now
allows guardrail_cost_per_unit and the guardrail mode.
- scale batch output-token reservations by the row's n / best_of candidate count
- parse client-supplied output caps defensively instead of 500ing on unparseable values
- exclude project IO descriptors from the first should_rate_limit pass when TPM
reservation is disabled so their buckets are not double-charged
An internal user who administers an organization saw an empty
Organization Usage dashboard and had to be promoted to proxy admin to
see any of it.
Two independent gates were closed on them. The route layer rejected
GET /organization/daily/activity with 401 before the handler ran, since
the route belonged to no list a non-proxy-admin can reach, and the
handler's own org-admin scoping was therefore dead code. In the
dashboard, viewOrganizationUsage was granted by session role alone, and
an org admin's session role is internal_user, so the Organization Usage
option never rendered and its data fetch stayed disabled.
The route now sits in self_managed_routes, where the handler restricts
results to organizations the caller is ORG_ADMIN of and 403s on any
other org, and viewOrganizationUsage joins the existing per-capability
org-admin allowance that already covers viewDeletedTeams.
A caller who administers no organization resolves to an empty id list
rather than to None, so the organization-alias lookup is scoped by that
same list instead of reading the whole table.
The Usage page falls back to the global view when org-admin membership
is revoked while it is open, so the selector never keeps a value it no
longer offers.
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
The Anthropic messages endpoint's exception handler passed the raw
request body dict to the failure hook, but request setup had already
replaced the processor's dict with one carrying the logging object, so
failure rows for /v1/messages never lifted recovered or estimated usage.
Pass the processor's dict instead.
The input-side estimate only counted the messages list, missing the
Anthropic top-level system prompt (string or text-block list) and the
Responses API instructions field, which live in optional_params. Count
them too.
Price ApplyGuardrail usage units recorded by PR #37225 with a new
bedrock/guardrails entry in the model cost map (regional override via
bedrock/{region}/guardrails), add the per-request guardrail_cost to the
standard logging payload's response_cost and CostBreakdown, surface it in
the x-litellm-response-cost header, and bill blocked requests through the
failure hook so key and team budgets see what AWS bills
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin'
Adds a third classification mode where an operator-supplied hook decides the
tier instead of the heuristic scorer or the LLM classifier. The hook implements
an async classify(context) returning a tier name (built-in value, tier_labels
label, or tier_definitions name) or None to decline; failures, timeouts, and
unknown tiers fall back exactly like a failed LLM classifier. The context
carries the request messages and metadata, including caller identity, so a
plugin can route by team, spend, or any business rule.
The plugin resolves from a dotted path at proxy startup with a load-time check
that classify is a coroutine function, and is closed off over HTTP like the
routing plugins list. Routing decisions record the new classifier_plugin cause.
tier_definitions now accepts classifier_type 'plugin' alongside 'llm'.
* fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids
The db-sync reconcile re-reads the raw config and hashes litellm_params to
compute which ids the config wants served, but the router's ids were hashed
from the resolved params where plugin dotted paths are live instances. The
mismatched ids made the reconcile evict every plugin-bearing auto-router one
sync after startup, on any proxy with a database connected. This also affected
the existing routing plugins list, not just the new classifier plugin.
Resolving the plugins in _delete_deployment the same way load_config does makes
both sides hash the same canonical form. A plugin module broken on disk at
reconcile time skips cleanup instead of evicting valid deployments, matching
how a get_config failure is handled
* fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel
A hook returning a non-string raised inside resolve_classified_tier outside the
plugin exception boundary, failing the request instead of falling back. Also
moves the read-only empty mapping to constants.py per repo convention and moves
the classifier plugin product docs out of the package README for the docs repo
* refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom'
The mode value now names the operator's intent while classifier_plugin keeps
naming the mechanism; routing decisions keep the classifier_plugin cause
* refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile
Replaces the previous approach of re-running plugin resolution inside
_delete_deployment, which imported operator modules on every reconcile cycle
and skipped the whole cleanup pass when any one module was broken on disk.
load_config now stamps model_info.id from the raw litellm_params before
resolution swaps dotted paths for live instances, so the reconcile's raw-config
hash matches by construction and needs no resolution at all: a broken module
cannot stall cleanup for unrelated models, and any future param-transforming
resolution is covered by the same pin. _generate_model_id becomes a staticmethod
so the pin can run before the Router exists; its statically dead non-string key
branches are removed. Also documents candidate_models as an informational
snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin
filters
* fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern
The staticmethod conversion accidentally dropped the non-string-key branches
from _generate_model_id, a silent hash change for any params with non-string
keys; they are restored verbatim. The classifier plugin context now follows
the Router-level routing-plugin recipe exactly: structured messages come from
resolve_structured_messages over the raw messages, and the metadata key comes
from the shared get_metadata_variable_name_from_kwargs helper, which also
replaces the duplicated inline sniff in _pick_model_for_tier. This removes the
raw-or-resolved fallback where a plugin could silently receive resolved
messages when a call site forgot to pass the raw ones
* refactor(router): make generate_model_id public, guard classifier context construction
Two modules legitimately hash deployment ids with the same helper now (Router
and the proxy's config-load pin), so the private name was lying about its
audience and the cross-module call needed a pyright suppression; renaming it
public restores the static safety net. The classifier plugin's RoutingContext
construction moves inside the failure boundary, matching the LLM path where
litellm-side prompt building also falls back rather than failing the request,
and a prompt-only call with no message list is now covered by a test
Failure rows in the spend log only carried token counts when a broken
stream stashed recovered partial usage; non-stream requests that reached
the provider and then failed (timeouts, provider 4xx/5xx) logged
0/0/0 even though the provider billed the input tokens. Estimate the
input side in post_call_failure_hook with the same tokenizer fallback
interrupted streams use, gated to requests that were actually dispatched
(first_api_call_start_time set and no litellm_no_upstream_llm_call
marker), and pin response_cost to 0.0 so failed requests never bill
spend. Recovered partial-stream usage still wins over the estimate.
Image, file, video, and previous_response_id requests reserved the whole
project ITPM limit up front, so any window with existing usage rejected
them and one in-flight multimodal request blocked the entire project.
Reserve the token_counter estimate instead, like every other request;
post-call reconciliation already charges actual usage.
* feat(proxy): add /team/daily/activity/aggregated and use it in the Usage UI
The Team Usage tab drained row-paginated pages client side, which painted
newest days first and drew duplicate bars when a day's rows straddled a
page boundary. Serve the whole range in one SQL GROUPING SETS pass instead:
the aggregated query gains optional per-entity rollup levels (entity as the
most-significant GROUPING bit) so breakdown.entities keeps per-team spend,
aliases, and per-key splits. The endpoint shares the paginated route's
scoping via _resolve_team_daily_activity_scope, accepts the timezone the UI
already sends, and the api_key filter now takes a list so non-admin member
scoping works. The dashboard tries the aggregated endpoint first and falls
back to page draining on failure.
* chore: ratchet B008 budget down by the endpoint converted to Annotated Depends
* chore: keep mutable-ok suppressions on their annotation lines after formatting
* fix(proxy): reject malformed or over-wide ranges on team aggregated activity
The aggregated endpoint has no pagination bounding its work, so validate
start_date and end_date as real dates and cap the span at 400 days. The
dashboard's widest presets fit well inside the cap, and an over-cap range
falls back to the paginated flow. Also trim implementation comments that
restated the grouping-set code.
* fix(proxy): parse aggregated range bounds as UTC to satisfy DTZ007
* refactor(proxy): fetch entity rollups with a companion query instead of extending the main one
The entity-as-extra-GROUPING-bit approach made the bitmask layout
mode-dependent: the same constant meant (date) for normal rows and
(date, entity) for entity rows, disambiguated by masking. Split it out:
the shared WHERE builder feeds both the untouched main query and a small
per-entity rollup query keyed by GROUPING(api_key), run concurrently, and
a fold writes breakdown.entities onto the built response.
* refactor(proxy): share the daily-activity error and entity-metadata shapes
The type-discipline ceiling for LIT002 ratcheted down on staging, so the new
aggregated endpoint had to stop hand-rolling collections the codebase already
builds elsewhere. Funnel the `{"error": ...}` detail through one construction
site, turn the range validator into an error-as-value, reuse a single
entity-metadata lookup for both breakdown paths, and widen
get_api_key_metadata to any set so callers stop copying frozensets.
* fix(proxy): strip callback credentials from the auth object stamped into request metadata
* style(proxy): drop the restating half of the stamp-site comment
* test(proxy): pin that the stamped auth copy carries header-derived identity
GET /guardrails/usage/overview and GET /guardrails/usage/detail/{id} 500ed on a
database that has not applied 20260817143646_add_daily_guardrail_usage_units yet
(pip installs on litellm-proxy-extras 0.4.86 with DISABLE_SCHEMA_UPDATE=true).
Both endpoints now return their metrics with empty units and log one warning
until the migration lands.
The daily guardrail metrics and usage-unit upserts are non-idempotent
increments, but the retry loop re-sent every failed row on any exception.
An ambiguous post-send failure such as a read timeout after the write had
already committed therefore stacked a second increment and inflated the
billable unit totals served by the guardrail usage endpoints.
Retry only DB_RETRY_SAFE_ERROR_TYPES (httpx.ConnectError), the same rule
the spend writer and autorouter rollup use for increment upserts, and log
any other failure once as terminal for that row while the rest of the
batch still lands.
Follows up #37225
The trailing-slash normalization test used gateway.litellm-sandbox.ai as
its base URL. Swap it for gateway.example.com so the test file does not
reference a real-looking hostname. The test is fully mocked, so the host
value has no effect on what is exercised.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
A transient DB error during the spend log flush dropped that batch's guardrail
metrics and usage unit rows for good. Retry only the rows that failed, up to 3
times with 1s/2s/4s backoff, mirroring the daily spend writer, and inject the
sleep so tests stay fast. Lowers the lint budgets the refactor freed up
