The docstring cited session_affinity_ttl_seconds as the keepalive bound,
but the Router-level knob feeding ttl_seconds is
deployment_affinity_ttl_seconds; session_affinity_ttl_seconds is the
separate per-request PreRoutingHookResponse override. Anyone grepping
the docstring's name to shrink the Router default finds only the
override. Name both, scoped correctly.
Starlette scans the route table in registration order, so a request pays one
regex match per route registered ahead of its own. The proxy registers several
hundred routes and left the liveness probe near position 280 and the lazy
loaded /v1/messages at the very end. Move /health/liveliness, /health/liveness,
/v1/chat/completions, /chat/completions and /v1/messages to the front of the
route table after startup registration and again after a lazy router loads.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): make the in-memory management cache capacity configurable
Add general_settings.user_api_key_cache_max_size (positive int, default 200) to resize the
in-memory tier of the shared user_api_key_cache at startup and on DB config reloads, expose it
in the Admin UI general settings, and cover it with behavioral tests. Prior art: #34726
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): resize the in-memory tier from DualCache so any cache instance honours the cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): wrap the cache capacity field description to the 120 col limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): parse Bedrock stream spend incrementally instead of buffering the whole response
Bedrock pass-through streaming kept every relayed chunk in memory until EOF and
then decoded, parsed and translated the whole stream again for spend logging.
Large or concurrent streams could exhaust proxy worker memory.
Sync and async passthrough wrappers now hand each chunk to a provider stream
collector as it is relayed. Bedrock decodes event-stream frames incrementally,
folds consecutive text deltas, and keeps only what stream_chunk_builder needs
for usage, tool calls and metadata. Text deltas are no longer retained in the
Bedrock and Anthropic stream decoders either. Providers without a collector
keep the previous raw-bytes behavior. Collector failures are isolated so spend
tracking can never interrupt the customer stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): assert the spend payload the collector builds instead of mock internals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): type the Bedrock collector helpers by the collector protocol instead of asserting the class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Key objects share the 200-entry UserApiKeyCache in-memory store with teams,
end users, tags and memberships, so churn in those objects evicts hot keys
and forces a LiteLLM_VerificationToken lookup on the next request. Route
bare hashed-token keys to a dedicated InMemoryCache inside UserApiKeyCache
while keeping Redis, TTL, serialization and invalidation shared
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The provider budget push runs inside the request success callback, so
awaiting the Redis pipeline there made every request wait for the round
trip. Hand it back to a task whose failure is logged through the breaker
aware logger, so an open breaker stays a debug line and a real Redis error
is one error line instead of an unretrieved task traceback
The componentized chart rendered litellm.connectionPoolEnv into the gateway
container only, so with database.connectionPool.enabled and
gateway.collector.enabled the collector's Prisma client opened its own pool
straight to Postgres instead of going through the pod-local PgBouncer.
Render the same include in the collector container, drop the gateway.extraEnv
workaround from the collector test, and add enabled/disabled regression
assertions for the collector in both helm charts and the terraform aws and gcp
modules, which already pass the pool env to their collector containers. The
module READMEs note the IAM token-auth exception, where the collector keeps a
direct connection by design.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared user popover copied alias, email and ID through three copy
buttons that all announced themselves as "Copy ID", so a screen reader
could not tell them apart. IdCell now takes the label, defaulting to the
old text everywhere else.
Also drops the closest("tr") the new link tests used, which put the
testing-library/no-node-access budget over its ceiling, and asserts the
sentinel row leaves User Email and the admin badge unlinked too.
Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
The team detail page's Virtual Keys table showed Organization ID, User
Email, User ID and Created By as dead text, so getting from a key to the
org or user behind it meant copying an id and searching for it.
Those four cells now render as links, reusing the sentinel-aware href
helpers, so default_user_id and the litellm-dashboard team stay plain
text instead of pointing at pages that do not exist.
The Created By cell was a verbatim copy of the Virtual Keys page's user
popover, so that moved into the shared table_cells kit and both tables
now use the one implementation.
Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
* fix(streaming): keep usage-only chunks from crashing streams with empty stream_options
The usage-only chunk branch in CustomStreamWrapper.chunk_creator indexed stream_options["include_usage"] directly, so a caller passing stream_options={} hit a KeyError that surfaced as MidStreamFallbackError. Streaming mock_response with an admission input_tokens count (#40637) now always emits such a chunk, which made the crash reachable. Reuse the send_stream_usage policy computed at init instead. Also annotate the #40637 test bindings with Final.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): report admitted zero prompt tokens instead of recounting in mock streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A call admitted before the breaker opened could finish while the breaker was
HALF_OPEN and close it before the designated probe reported, so Redis traffic
resumed on a stale answer. The admission now records whether the call is the
probe and only the probe's success closes a half-open breaker.
The cron job lock manager also logged an error every cycle the open breaker
refused its Redis call, one line per job per pod. That refusal is now a debug
line like every other guarded call, while real Redis errors still log at error
The sync Redis read now raises while the circuit breaker is open, and the
health state merge caught that as a generic error, skipping the local write
and logging an error on every background health check cycle. Read the shared
snapshot through a helper that treats the refused read as a miss so the merge
falls back to the pod-local copy the way a swallowed connection error already did
* fix(logging): finish response metadata before the sync logging thread reads it
The async and sync client wrappers handed the response to the threaded success handler before computing its cost, call id, and api_base, so that thread inserted into the same metadata dict the request coroutine was still iterating and a finished chat completion turned into a 500 (dictionary changed size during iteration). Metadata is now finalized first, and the merge and header copies snapshot their dicts before iterating.
* fix(logging): snapshot metadata with a dict copy and drop redundant comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): copy metadata via dict.copy and dedupe Final import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Same treatment as User, Team and Created By in the previous commit: the
Organization column rendered the alias as dead text, so it now goes through
IdentityCell with an orgDetailHref.
Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
The sync get path was unguarded, logged with a stray format argument, and never fed the
breaker. The sync batch read swallowed the breaker's refusal as an ERROR plus a service
failure event per call, so DualCache dropped its in-memory hits and left batch reservations
behind. record_success closed an OPEN breaker on stale in-flight successes, skipping the
recovery timeout and the half-open probe. The spend counter pipeline re-raised the refusal
into the cost callback, which logged an ERROR and fired the failed-tracking alert per request.
A team or key priority that is not Latin-1 encodable (for example CJK text) was
attached as a response header by the dynamic rate limiter v3 post-call hook, and
Starlette then raised UnicodeEncodeError while writing headers, turning a
successful /v1/messages call into HTTP 500. The header is now omitted for such
values while x-litellm-rate-limiter-version and the v3 rate limit headers are
still attached.
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(mock): emit admission-time usage chunk on streaming mock_response
Streaming mock_response chunks carried no usage, so the chunk builder re-tokenized the whole prompt in Python after the stream ended even when budget reservation had already counted it at admission. The mock streaming generators now yield a final usage-only chunk carrying the admission prompt count (same completion count as the non-streaming path). Without an admission count the old tokenizer fallback stays.
* fix(mock): type the mock stream generators and keep the usage chunk on the content stream id
Review follow-up: the usage-only chunk was built with a fresh id, so CustomStreamWrapper switched response_id for the finish-reason and usage chunks. It now copies the content stream id. The generators also get full parameter and return annotations.
---------
Co-authored-by: yassin <yassin@berri.ai>
Take the staging tree outright. #40620 already landed the typed
RedisCircuitBreakerOpenError, DEBUG-level refusal logging at every
DualCache, Cache, limiter, and router catch site, and sync timeout
classification, so the overlapping parts of this branch are superseded.
The pieces that are still missing on staging are rebuilt on top in the
next commit
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar
py-spy on the gateway showed the post-response _PROXY_track_cost_callback,
spend-log and DBSpendUpdateWriter work running on the inference workers'
event loop, so a DB or Redis stall backed up the request path.
When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact
typed SpendEvent per success and hands it to a SpendEventProducer that ships
it over a unix socket (default) or loopback-only TCP to a sidecar started as
`python -m gateway.spend_worker`. The sidecar runs the unchanged
_ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url).
When the sidecar is unreachable, the buffer is full, or the gateway shuts
down with events still queued or in flight, the producer applies
LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The
sidecar half-closes producers on SIGTERM and drains, the producer treats
EOF as unavailable, and the gateway flushes buffered spend counters on
shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its
own process log.
Helm: both charts gain an opt-in spend-worker sidecar container sharing an
emptyDir socket dir, and the componentized chart's HPA uses a
ContainerResource CPU metric scoped to the gateway container so sidecar
CPU does not drive inference scaling.
* feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks
Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): retrieve a completed batch in the in-process spend path test
The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename the spend worker sidecar to collector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown
The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime
stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the
pod-local pgbouncer URL itself, replacing gateway/collector.py
The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it.
SpendEventProducer shields an in-progress fallback from the writer task cancellation so close()
no longer loses an event already handed to the in-process pipeline
Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs,
flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(terraform): drop the gcp job duplicated by the aws/gcp matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports
The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format pooled_database_url after the pgbouncer rebase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain
Cache hits updated the logging object after the early return, so the offloaded spend event carried
preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against
producer transports uvloop already closed so one dead connection cannot abort the drain
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): keep the gcp collector port off the metrics sidecar health port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): collector connects to Postgres directly under IAM or Entra token auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
