A terminal HTTP failure partway through chunking now logs the summed usage
and cost of the ApplyGuardrail calls AWS already billed, mirroring the
blocked-chunk path.
Removes one named callback from a team and leaves the team's other callbacks
registered and firing. Before this, the only removal route was
POST /team/{team_id}/disable_logging, which clears every callback at once, so a
tenant sharing a team could not deregister its own integration
The handler filters metadata["logging"], keeps the survivors encrypted, refreshes
the cached team so the removal applies to keys that are already live, and emits a
redacted audit row, matching what the add and disable routes do
Resolves LIT-5161
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail
call AWS billed for the logical request, not just the blocking chunk.
Client-supplied metadata.standard_logging_guardrail_information is stripped
at the proxy boundary so callers cannot forge (even negative) guardrail
cost into spend, and guardrail_information_cost ignores negative or
non-finite entry costs as defense in depth. The cost map schema test now
allows guardrail_cost_per_unit and the guardrail mode.
* feat(vector_stores): add Valkey as a managed vector store provider
Adds a valkey provider for managed vector stores, searchable via the
valkey-search module over RESP. Introduces BaseDirectVectorStoreConfig
for datastores that execute searches directly instead of building an
HTTP request, and refactors the valkey semantic cache to share the new
connection URL helper. Registered in the provider enum, router params,
proxy config registry, Admin UI Add Vector Store modal, and provider
endpoint support matrix.
* fix(vector_stores): join list queries and bound valkey socket timeouts
Review feedback: multi-string queries are now space-joined like every
other embedding-based provider instead of dropping all but the first,
and the request timeout is threaded through the direct vector store
interface into bounded socket_connect_timeout / socket_timeout values
on both redis clients so an unreachable Valkey host cannot pin proxy
workers until the OS TCP timeout.
* chore(ui): regenerate schema.d.ts for valkey vector store fields
* docs(ui): make the Valkey vector store setup note and field tooltips explicit
* feat(ui): pick the Valkey embedding model from the proxy's models like Milvus
* fix(ui): number the setup steps in the vector store provider alerts
- scale batch output-token reservations by the row's n / best_of candidate count
- parse client-supplied output caps defensively instead of 500ing on unparseable values
- exclude project IO descriptors from the first should_rate_limit pass when TPM
reservation is disabled so their buckets are not double-charged
An internal user who administers an organization saw an empty
Organization Usage dashboard and had to be promoted to proxy admin to
see any of it.
Two independent gates were closed on them. The route layer rejected
GET /organization/daily/activity with 401 before the handler ran, since
the route belonged to no list a non-proxy-admin can reach, and the
handler's own org-admin scoping was therefore dead code. In the
dashboard, viewOrganizationUsage was granted by session role alone, and
an org admin's session role is internal_user, so the Organization Usage
option never rendered and its data fetch stayed disabled.
The route now sits in self_managed_routes, where the handler restricts
results to organizations the caller is ORG_ADMIN of and 403s on any
other org, and viewOrganizationUsage joins the existing per-capability
org-admin allowance that already covers viewDeletedTeams.
A caller who administers no organization resolves to an empty id list
rather than to None, so the organization-alias lookup is scoped by that
same list instead of reading the whole table.
The Usage page falls back to the global view when org-admin membership
is revoked while it is open, so the selector never keeps a value it no
longer offers.
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
On /v1/messages, AnthropicStreamWrapper synthesizes the content_block_start for
the first content block, queues it, then hits a bare `continue` when that same
upstream chunk's translated delta is empty. The queue is only drained at the top
of the next __next__ / __anext__, so the queued content_block_start waits for a
further upstream chunk to arrive.
An empty delta on the opening chunk is the normal tool-call shape: Bedrock
Converse's contentBlockStart carries the tool id and name with no arguments, and
OpenAI-format streams send arguments: "" on the chunk that names the function.
So a client learns a tool call started one upstream event late, and when the
provider delivers argument fragments as a trailing burst it sees nothing at all
after message_start for the whole generation.
Flush the queued event before continuing, in both the sync and async paths. The
sibling block-transition path already returns from the queue, so only the
first-block-open case changed.
The Anthropic messages endpoint's exception handler passed the raw
request body dict to the failure hook, but request setup had already
replaced the processor's dict with one carrying the logging object, so
failure rows for /v1/messages never lifted recovered or estimated usage.
Pass the processor's dict instead.
The input-side estimate only counted the messages list, missing the
Anthropic top-level system prompt (string or text-block list) and the
Responses API instructions field, which live in optional_params. Count
them too.
Price ApplyGuardrail usage units recorded by PR #37225 with a new
bedrock/guardrails entry in the model cost map (regional override via
bedrock/{region}/guardrails), add the per-request guardrail_cost to the
standard logging payload's response_cost and CostBreakdown, surface it in
the x-litellm-response-cost header, and bill blocked requests through the
failure hook so key and team budgets see what AWS bills
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin'
Adds a third classification mode where an operator-supplied hook decides the
tier instead of the heuristic scorer or the LLM classifier. The hook implements
an async classify(context) returning a tier name (built-in value, tier_labels
label, or tier_definitions name) or None to decline; failures, timeouts, and
unknown tiers fall back exactly like a failed LLM classifier. The context
carries the request messages and metadata, including caller identity, so a
plugin can route by team, spend, or any business rule.
The plugin resolves from a dotted path at proxy startup with a load-time check
that classify is a coroutine function, and is closed off over HTTP like the
routing plugins list. Routing decisions record the new classifier_plugin cause.
tier_definitions now accepts classifier_type 'plugin' alongside 'llm'.
* fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids
The db-sync reconcile re-reads the raw config and hashes litellm_params to
compute which ids the config wants served, but the router's ids were hashed
from the resolved params where plugin dotted paths are live instances. The
mismatched ids made the reconcile evict every plugin-bearing auto-router one
sync after startup, on any proxy with a database connected. This also affected
the existing routing plugins list, not just the new classifier plugin.
Resolving the plugins in _delete_deployment the same way load_config does makes
both sides hash the same canonical form. A plugin module broken on disk at
reconcile time skips cleanup instead of evicting valid deployments, matching
how a get_config failure is handled
* fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel
A hook returning a non-string raised inside resolve_classified_tier outside the
plugin exception boundary, failing the request instead of falling back. Also
moves the read-only empty mapping to constants.py per repo convention and moves
the classifier plugin product docs out of the package README for the docs repo
* refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom'
The mode value now names the operator's intent while classifier_plugin keeps
naming the mechanism; routing decisions keep the classifier_plugin cause
* refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile
Replaces the previous approach of re-running plugin resolution inside
_delete_deployment, which imported operator modules on every reconcile cycle
and skipped the whole cleanup pass when any one module was broken on disk.
load_config now stamps model_info.id from the raw litellm_params before
resolution swaps dotted paths for live instances, so the reconcile's raw-config
hash matches by construction and needs no resolution at all: a broken module
cannot stall cleanup for unrelated models, and any future param-transforming
resolution is covered by the same pin. _generate_model_id becomes a staticmethod
so the pin can run before the Router exists; its statically dead non-string key
branches are removed. Also documents candidate_models as an informational
snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin
filters
* fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern
The staticmethod conversion accidentally dropped the non-string-key branches
from _generate_model_id, a silent hash change for any params with non-string
keys; they are restored verbatim. The classifier plugin context now follows
the Router-level routing-plugin recipe exactly: structured messages come from
resolve_structured_messages over the raw messages, and the metadata key comes
from the shared get_metadata_variable_name_from_kwargs helper, which also
replaces the duplicated inline sniff in _pick_model_for_tier. This removes the
raw-or-resolved fallback where a plugin could silently receive resolved
messages when a call site forgot to pass the raw ones
* refactor(router): make generate_model_id public, guard classifier context construction
Two modules legitimately hash deployment ids with the same helper now (Router
and the proxy's config-load pin), so the private name was lying about its
audience and the cross-module call needed a pyright suppression; renaming it
public restores the static safety net. The classifier plugin's RoutingContext
construction moves inside the failure boundary, matching the LLM path where
litellm-side prompt building also falls back rather than failing the request,
and a prompt-only call with no message list is now covered by a test
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live
(default, unchanged behavior), record (pass through to the live proxy while
writing every interaction to a fixture bundle), or replay (serve every
interaction from the bundle with no proxy and no provider spend). Both new
transports fulfil the existing Transport protocol, so no test changes shape.
A bundle is a directory with a manifest (record timestamp, harness version,
format version) and one JSON file per interaction, grouped per test in call
order. Replay against a manifest older than seven days hard-fails at
collection time naming the bundle age. Record always wipes and never reads
the previous bundle, refusing to wipe a directory that is not a bundle.
Auth header values are redacted on write; uploads store a sha256 digest.
unique_marker() becomes deterministic per test in record/replay modes so a
replay run regenerates exactly the requests the record run sent.
Content-based match keys, streaming chunk fidelity, and provider-scoping are
follow-ups (LIT-5741, LIT-5742, LIT-5745).
Failure rows in the spend log only carried token counts when a broken
stream stashed recovered partial usage; non-stream requests that reached
the provider and then failed (timeouts, provider 4xx/5xx) logged
0/0/0 even though the provider billed the input tokens. Estimate the
input side in post_call_failure_hook with the same tokenizer fallback
interrupted streams use, gated to requests that were actually dispatched
(first_api_call_start_time set and no litellm_no_upstream_llm_call
marker), and pin response_cost to 0.0 so failed requests never bill
spend. Recovered partial-stream usage still wins over the estimate.
Image, file, video, and previous_response_id requests reserved the whole
project ITPM limit up front, so any window with existing usage rejected
them and one in-flight multimodal request blocked the entire project.
Reserve the token_counter estimate instead, like every other request;
post-call reconciliation already charges actual usage.
openai/responses/<model> deployments reached OpenAI as the literal model id
responses/<model> on /v1/responses and on /v1/messages (which rides the
Responses API for the openai provider) and 400ed with model_not_found, while
/v1/chat/completions already stripped the prefix. Strip a leading responses/
right after provider resolution so every Responses API entrypoint (HTTP,
websocket, compaction, the /v1/messages adapter) sends the real model id
Keeps an explicit empty messages list on the pre-existing path (default
model, provider validation error) instead of dropping the routing
decision and surfacing a misleading tags 401.
Auto-router strategy hook returned None whenever the request carried
input instead of messages, so tagged /v1/responses requests (Codex CLI)
never picked a tier and tag filtering left nothing to route to. Resolve
input through the shared prompt-template helper before matching routes.
* feat(proxy): add /team/daily/activity/aggregated and use it in the Usage UI
The Team Usage tab drained row-paginated pages client side, which painted
newest days first and drew duplicate bars when a day's rows straddled a
page boundary. Serve the whole range in one SQL GROUPING SETS pass instead:
the aggregated query gains optional per-entity rollup levels (entity as the
most-significant GROUPING bit) so breakdown.entities keeps per-team spend,
aliases, and per-key splits. The endpoint shares the paginated route's
scoping via _resolve_team_daily_activity_scope, accepts the timezone the UI
already sends, and the api_key filter now takes a list so non-admin member
scoping works. The dashboard tries the aggregated endpoint first and falls
back to page draining on failure.
* chore: ratchet B008 budget down by the endpoint converted to Annotated Depends
* chore: keep mutable-ok suppressions on their annotation lines after formatting
* fix(proxy): reject malformed or over-wide ranges on team aggregated activity
The aggregated endpoint has no pagination bounding its work, so validate
start_date and end_date as real dates and cap the span at 400 days. The
dashboard's widest presets fit well inside the cap, and an over-cap range
falls back to the paginated flow. Also trim implementation comments that
restated the grouping-set code.
* fix(proxy): parse aggregated range bounds as UTC to satisfy DTZ007
* refactor(proxy): fetch entity rollups with a companion query instead of extending the main one
The entity-as-extra-GROUPING-bit approach made the bitmask layout
mode-dependent: the same constant meant (date) for normal rows and
(date, entity) for entity rows, disambiguated by masking. Split it out:
the shared WHERE builder feeds both the untouched main query and a small
per-entity rollup query keyed by GROUPING(api_key), run concurrently, and
a fold writes breakdown.entities onto the built response.
* refactor(proxy): share the daily-activity error and entity-metadata shapes
The type-discipline ceiling for LIT002 ratcheted down on staging, so the new
aggregated endpoint had to stop hand-rolling collections the codebase already
builds elsewhere. Funnel the `{"error": ...}` detail through one construction
site, turn the range validator into an error-as-value, reuse a single
entity-metadata lookup for both breakdown paths, and widen
get_api_key_metadata to any set so callers stop copying frozensets.
