The prior commit ran ruff format on the whole file to type the
_batch_cost_result helper, reflowing hundreds of unrelated pre-existing
lines that were never ruff-format-clean to begin with (confirmed at the
PR's base commit, before any of these changes). CI's ruff-format gate
only checks litellm/**/*.py, not tests/, so this reformatting served no
CI purpose and only bloated the diff. Restores everything except the
intended TYPE_CHECKING import and _batch_cost_result annotations.
- ruff format litellm/batches/batch_utils.py
- reconcile the upstream output_file_id=None guard (merged in from
litellm_internal_staging) with BatchCostUsageResult, and count that
batch's error_file_id failures instead of always reporting 0
- fix test_handle_completed_batch_no_output_file_is_zero's tuple
unpacking, which predated the BatchCostUsageResult refactor
- commit the batch_successful_requests/batch_failed_requests fixture
fix to test_spend_management_endpoints.py that was left uncommitted
Both surfaces wrote zero bytes for the whole time-to-first-token, so an
intermediary with an idle read timeout drops a healthy connection before the
first token. They reached neither keepalive engine, which is what #37322 left
open.
The streaming assistants run spends that wait inside the awaited call that
produces its response, since create_response buffers the first chunk, so it
takes the same open_sse_before_first_byte seam the native routes use. The A2A
route only contacts the upstream agent once its body iterator is first pulled,
so nothing is awaited before the response exists and the gap has to be filled
from inside the stream instead; wrap_sse_stream_with_keepalive_pings already
does that and now takes the filler as a parameter, so A2A gets an SSE comment
its JSON-RPC clients discard rather than Anthropic's ping event.
Off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds.
* fix(otel): bound and shut down credential-scoped tracer providers
Each credential-scoped TracerProvider owns a BatchSpanProcessor worker thread that
only stops on shutdown, and the v1 cache holding them was an unbounded, unsynchronized
dict that never shut anything down. Every distinct team/key credential set therefore
added a thread for the life of the process, and concurrent first-requests for the same
credential set orphaned duplicate providers outright.
Make the cache a lock-guarded bounded LRU that shuts down whatever it drops, matching
the v2 TenantTracerCache. Providers wrapping a caller-supplied SpanExporter instance
share that exporter with the logger's own provider, so they are dropped without
shutdown; those use SimpleSpanProcessor and own no thread.
* fix(otel): reclaim dropped providers on a dedicated executor
Sustained credential churn queues one blocking shutdown per eviction, so using the
shared logging executor let an unreachable tenant endpoint stall unrelated logging
work behind the OTLP retry budget. Give provider shutdown its own bounded pool; its
threads spawn lazily, so a proxy that never evicts still pays nothing.
* fix(otel): decide provider shutdown from the victim, not the evicting request
Both dynamic entry points share one provider cache, so it can hold providers of
mixed exporter ownership. Reading the ownership flag from the evicting request
therefore stopped a shared caller-supplied exporter in one direction, silencing
telemetry process-wide, and leaked a BatchSpanProcessor thread in the other.
Cache ownership alongside the provider so the drop decision reads the victim's
own flag.
* fix(otel): honor the widened header mapping type instead of dict only
Widening the header parameter to Mapping left the isinstance check on dict, so a
non-dict Mapping silently returned no headers at all, which for the OTLP path means
an unauthenticated exporter and no traces with nothing raised. The dict branch also
returned the caller's own object, and dropping the defensive copy at the call site
let that alias reach a long-lived exporter. Match on Mapping and copy.
* fix(otel): do not give a provider we may never stop an interpreter-exit hook
Every TracerProvider registers an atexit hook by default, and that hook holds a strong
reference. Providers wrapping a caller-supplied exporter are dropped without shutdown,
so they stayed pinned for the life of the process and then stopped the shared exporter
at exit. Tie shutdown_on_exit to ownership: those providers use SimpleSpanProcessor and
buffer nothing, so they lose no flush, while providers that own their exporter keep the
hook and their exit flush.
Also stop the victim the eviction test leaves behind, and trim the added comments.
Key and team router_settings set enable_tag_filtering on the request kwargs,
and get_deployments_for_tag already treats that as authoritative, but
_select_pre_routing_strategy only consulted the router-wide flag, so tagged
auto-router markers still captured untagged requests from keys that enabled
filtering. The e2e auto-router module now enables tag filtering through
key-level router_settings instead of flipping /config/update module-wide,
which was denying concurrently running tagged requests from other suites on
the shared per-build CI proxy.
A terminal HTTP failure partway through chunking now logs the summed usage
and cost of the ApplyGuardrail calls AWS already billed, mirroring the
blocked-chunk path.
Removes one named callback from a team and leaves the team's other callbacks
registered and firing. Before this, the only removal route was
POST /team/{team_id}/disable_logging, which clears every callback at once, so a
tenant sharing a team could not deregister its own integration
The handler filters metadata["logging"], keeps the survivors encrypted, refreshes
the cached team so the removal applies to keys that are already live, and emits a
redacted audit row, matching what the add and disable routes do
Resolves LIT-5161
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail
call AWS billed for the logical request, not just the blocking chunk.
Client-supplied metadata.standard_logging_guardrail_information is stripped
at the proxy boundary so callers cannot forge (even negative) guardrail
cost into spend, and guardrail_information_cost ignores negative or
non-finite entry costs as defense in depth. The cost map schema test now
allows guardrail_cost_per_unit and the guardrail mode.
* feat(vector_stores): add Valkey as a managed vector store provider
Adds a valkey provider for managed vector stores, searchable via the
valkey-search module over RESP. Introduces BaseDirectVectorStoreConfig
for datastores that execute searches directly instead of building an
HTTP request, and refactors the valkey semantic cache to share the new
connection URL helper. Registered in the provider enum, router params,
proxy config registry, Admin UI Add Vector Store modal, and provider
endpoint support matrix.
* fix(vector_stores): join list queries and bound valkey socket timeouts
Review feedback: multi-string queries are now space-joined like every
other embedding-based provider instead of dropping all but the first,
and the request timeout is threaded through the direct vector store
interface into bounded socket_connect_timeout / socket_timeout values
on both redis clients so an unreachable Valkey host cannot pin proxy
workers until the OS TCP timeout.
* chore(ui): regenerate schema.d.ts for valkey vector store fields
* docs(ui): make the Valkey vector store setup note and field tooltips explicit
* feat(ui): pick the Valkey embedding model from the proxy's models like Milvus
* fix(ui): number the setup steps in the vector store provider alerts
- scale batch output-token reservations by the row's n / best_of candidate count
- parse client-supplied output caps defensively instead of 500ing on unparseable values
- exclude project IO descriptors from the first should_rate_limit pass when TPM
reservation is disabled so their buckets are not double-charged
An internal user who administers an organization saw an empty
Organization Usage dashboard and had to be promoted to proxy admin to
see any of it.
Two independent gates were closed on them. The route layer rejected
GET /organization/daily/activity with 401 before the handler ran, since
the route belonged to no list a non-proxy-admin can reach, and the
handler's own org-admin scoping was therefore dead code. In the
dashboard, viewOrganizationUsage was granted by session role alone, and
an org admin's session role is internal_user, so the Organization Usage
option never rendered and its data fetch stayed disabled.
The route now sits in self_managed_routes, where the handler restricts
results to organizations the caller is ORG_ADMIN of and 403s on any
other org, and viewOrganizationUsage joins the existing per-capability
org-admin allowance that already covers viewDeletedTeams.
A caller who administers no organization resolves to an empty id list
rather than to None, so the organization-alias lookup is scoped by that
same list instead of reading the whole table.
The Usage page falls back to the global view when org-admin membership
is revoked while it is open, so the selector never keeps a value it no
longer offers.
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
On /v1/messages, AnthropicStreamWrapper synthesizes the content_block_start for
the first content block, queues it, then hits a bare `continue` when that same
upstream chunk's translated delta is empty. The queue is only drained at the top
of the next __next__ / __anext__, so the queued content_block_start waits for a
further upstream chunk to arrive.
An empty delta on the opening chunk is the normal tool-call shape: Bedrock
Converse's contentBlockStart carries the tool id and name with no arguments, and
OpenAI-format streams send arguments: "" on the chunk that names the function.
So a client learns a tool call started one upstream event late, and when the
provider delivers argument fragments as a trailing burst it sees nothing at all
after message_start for the whole generation.
Flush the queued event before continuing, in both the sync and async paths. The
sibling block-transition path already returns from the queue, so only the
first-block-open case changed.
The Anthropic messages endpoint's exception handler passed the raw
request body dict to the failure hook, but request setup had already
replaced the processor's dict with one carrying the logging object, so
failure rows for /v1/messages never lifted recovered or estimated usage.
Pass the processor's dict instead.
The input-side estimate only counted the messages list, missing the
Anthropic top-level system prompt (string or text-block list) and the
Responses API instructions field, which live in optional_params. Count
them too.
Price ApplyGuardrail usage units recorded by PR #37225 with a new
bedrock/guardrails entry in the model cost map (regional override via
bedrock/{region}/guardrails), add the per-request guardrail_cost to the
standard logging payload's response_cost and CostBreakdown, surface it in
the x-litellm-response-cost header, and bill blocked requests through the
failure hook so key and team budgets see what AWS bills
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin'
Adds a third classification mode where an operator-supplied hook decides the
tier instead of the heuristic scorer or the LLM classifier. The hook implements
an async classify(context) returning a tier name (built-in value, tier_labels
label, or tier_definitions name) or None to decline; failures, timeouts, and
unknown tiers fall back exactly like a failed LLM classifier. The context
carries the request messages and metadata, including caller identity, so a
plugin can route by team, spend, or any business rule.
The plugin resolves from a dotted path at proxy startup with a load-time check
that classify is a coroutine function, and is closed off over HTTP like the
routing plugins list. Routing decisions record the new classifier_plugin cause.
tier_definitions now accepts classifier_type 'plugin' alongside 'llm'.
* fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids
The db-sync reconcile re-reads the raw config and hashes litellm_params to
compute which ids the config wants served, but the router's ids were hashed
from the resolved params where plugin dotted paths are live instances. The
mismatched ids made the reconcile evict every plugin-bearing auto-router one
sync after startup, on any proxy with a database connected. This also affected
the existing routing plugins list, not just the new classifier plugin.
Resolving the plugins in _delete_deployment the same way load_config does makes
both sides hash the same canonical form. A plugin module broken on disk at
reconcile time skips cleanup instead of evicting valid deployments, matching
how a get_config failure is handled
* fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel
A hook returning a non-string raised inside resolve_classified_tier outside the
plugin exception boundary, failing the request instead of falling back. Also
moves the read-only empty mapping to constants.py per repo convention and moves
the classifier plugin product docs out of the package README for the docs repo
* refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom'
The mode value now names the operator's intent while classifier_plugin keeps
naming the mechanism; routing decisions keep the classifier_plugin cause
* refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile
Replaces the previous approach of re-running plugin resolution inside
_delete_deployment, which imported operator modules on every reconcile cycle
and skipped the whole cleanup pass when any one module was broken on disk.
load_config now stamps model_info.id from the raw litellm_params before
resolution swaps dotted paths for live instances, so the reconcile's raw-config
hash matches by construction and needs no resolution at all: a broken module
cannot stall cleanup for unrelated models, and any future param-transforming
resolution is covered by the same pin. _generate_model_id becomes a staticmethod
so the pin can run before the Router exists; its statically dead non-string key
branches are removed. Also documents candidate_models as an informational
snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin
filters
* fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern
The staticmethod conversion accidentally dropped the non-string-key branches
from _generate_model_id, a silent hash change for any params with non-string
keys; they are restored verbatim. The classifier plugin context now follows
the Router-level routing-plugin recipe exactly: structured messages come from
resolve_structured_messages over the raw messages, and the metadata key comes
from the shared get_metadata_variable_name_from_kwargs helper, which also
replaces the duplicated inline sniff in _pick_model_for_tier. This removes the
raw-or-resolved fallback where a plugin could silently receive resolved
messages when a call site forgot to pass the raw ones
* refactor(router): make generate_model_id public, guard classifier context construction
Two modules legitimately hash deployment ids with the same helper now (Router
and the proxy's config-load pin), so the private name was lying about its
audience and the cross-module call needed a pyright suppression; renaming it
public restores the static safety net. The classifier plugin's RoutingContext
construction moves inside the failure boundary, matching the LLM path where
litellm-side prompt building also falls back rather than failing the request,
and a prompt-only call with no message list is now covered by a test
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live
(default, unchanged behavior), record (pass through to the live proxy while
writing every interaction to a fixture bundle), or replay (serve every
interaction from the bundle with no proxy and no provider spend). Both new
transports fulfil the existing Transport protocol, so no test changes shape.
A bundle is a directory with a manifest (record timestamp, harness version,
format version) and one JSON file per interaction, grouped per test in call
order. Replay against a manifest older than seven days hard-fails at
collection time naming the bundle age. Record always wipes and never reads
the previous bundle, refusing to wipe a directory that is not a bundle.
Auth header values are redacted on write; uploads store a sha256 digest.
unique_marker() becomes deterministic per test in record/replay modes so a
replay run regenerates exactly the requests the record run sent.
Content-based match keys, streaming chunk fidelity, and provider-scoping are
follow-ups (LIT-5741, LIT-5742, LIT-5745).
