The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call
deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss
tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left
Deployment affinity and the complexity router's session and tier pins move
off the router's DualCache onto RouterSessionPinStore, backed by the native
redis-store. The read and claim Lua scripts run byte-identical to main, the
pin read rides the request's pre-call batch, and keys and values keep main's
format.
* refactor(router): count the whole router_usage family on the native store
usage-based-routing-v2, the model rate limit check (tpm, rpm, tpd, rpd) and the
io-token check (itpm, otpm) count through the native router_usage store, next
to the router core's global_router tpm and rpm. The pick's usage read rides the
native pre-call batch and settles on the batch reply, keeping DualCache's
per-key read throttle, and the success tpm increment rides the native
post-call batch after counting in memory. io-token reservations stash their
counter's fields instead of a Redis key. Redis keys and values are unchanged.
A Redis service event is reported in a context named for whose it is, either
the calling store op's, the request whose post-call batch its deadline sends,
or nobody's for breaker transitions and unscoped calls. The deadline's tag is
built when a post-call declaration arms it, so that flush parents under the
request's span even though auth sets the span after the batch scope opens. An
SDK call anchors the span its caller passed, so its usage reads and io-token
refunds parent under it without a span argument threaded through the family.
* fix(redis): keep an int counter past i64 above the limit instead of wrapping
Number::add summed two ints with plain i64 addition. With Redis off and the OTPM counter at 1, a request with max_tokens=2**63-1 passed the MAX_INCREMENT check, and the memory count wrapped negative in a release build, so it passed admission and let other requests past the cap. Python's int never overflows, so main sees a count above the limit, rolls it back and answers 429
An int sum past i64 now becomes the nearest float, which compares above any limit the way Python's int does. The rollback after such a refusal can leave the count one token low, since the float cannot hold 2**63 exactly
* fix(logging): run every callback under the span an SDK caller passed
Native Redis service spans take their parent from the request span contextvar. The proxy sets it at auth and the SDK router scopes it around the routed call, but a streamed response is iterated after that scope closes, so its callbacks saw no span. Three callbacks reopened the scope from kwargs by hand, and the sync stream path had no wrap at all because its success handler runs on an executor thread
The four Logging handlers now anchor the span from model_call_details around their dispatch, so every CustomLogger callback sees it, streamed or not, sync or async, with or without a Router. The per-callback wraps in lowest_tpm_rpm_v2 and model_rate_limit_check are gone. A span already anchored, as on the proxy, is kept
* test(logging): cover a mid-stream failure, and drop the span wrap in Router._release
The callback span test now also breaks a stream after its first chunk, sync and async, through a custom provider, so the failure handlers the stream wrapper starts on a bare thread are covered too
A probe through the Router showed Router._release only runs inside the routed attempt, where the router has already anchored the span, and never after a stream returns. Its own wrap was a no-op, so it is gone
* fix(router): refuse an io-token reservation past its limit before counting it
The overflow fix let a reservation near i64::MAX become a float, so the count still read above the limit and the request got its 429. But the rollback subtracted the same amount from that float, and with OTPM usage at 40 the count came back as 0.0, so later requests reused capacity already spent
A reservation above its limit now gets the 429 before the store is touched, in the async and sync paths. On main the count went up and straight back down for such a reservation, so the answer and the count after it are the same as main's. Only the usage named in the error message differs, which now shows the reservation instead of the count plus the reservation
* test(redis): send the failed SDK call's refund through a real completion
The refund span test called ModelRateLimitingCheck.async_log_failure_event directly, so it relied on the span wrap that callback used to carry. The Logging failure handlers now anchor the span for every callback, so the test sends a failing litellm.acompletion with the reservation in its metadata and checks the refund's Redis span still lands under the caller's span
* feat(redis): serve the health checks' deployment states from a native store
DeploymentHealthCache reads and writes its one Redis entry through a new
deployment_health store family instead of the router's DualCache, like the
other router families.
* refactor(redis): type the deployment health states in the store
The health store kept a raw Value from Redis to Python and back, so Python re-checked the shape by hand and router-core only got a typed record after a Python pass. The family now has a types.rs with HealthState and HealthStates, the bridge converts through PyData, and the Python store returns DeploymentHealthStateValue mappings
Reads keep main's leniency for records another writer left: extra fields are ignored, a falsy flag is unhealthy and a missing one healthy, a timestamp that is no number reads as 0, and a record that is no dict is dropped alone. A rewrite writes only is_healthy, timestamp and reason
* fix(dynamic_rate_limiter_v3): read a model's saturation from Redis so every worker agrees on it
A worker's in-memory copy of the saturation counters carried only the usage it served
itself, and steady traffic kept that copy alive past saturation_check_cache_ttl, so two
workers behind one Redis disagreed on whether a model was saturated. The saturation read
now goes to Redis and is written back to memory for the TTL; without Redis memory is all
there is.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): read the saturation count from Redis itself, not DualCache's memory-first get
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): back the Redis saturation count into memory only when memory holds nothing, and split the counter-pair comprehensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): fall back to the worker's own saturation count when the Redis read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL
* fix(prometheus): carry live spend for the user, org and customer budget gauges too
The remaining budget gauges for users, orgs and customers read the same cached rows as the key and
team gauges, so they read high for about a minute after a charge. Auth now carries the live counter
for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded.
The x-litellm-key-spend header and logging spend fields keep their current values.
* fix(prometheus): show committed spend in the remaining budget gauges
Reads twin counters that take only charged cost, so a worst-case budget hold
never shows as spent and a request that finishes last cannot set a gauge back.
The twins exist only while the Prometheus logger is registered, and every path
that resets a spend counter drops them. Failed requests now set the org gauge
from the carried org row and committed spend, not the cached org row.
* fix(prometheus): count a charge the cached customer row already holds once
* fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets
Auth now reads committed spend through the admission MGET and names the counters a charge
may move, so the master key and unbudgeted entities get no twin. Organization spend edits
and the global spend reset drop their twins, and a long request no longer reads as charged.
* fix(prometheus): read the committed org spend on failed requests without the auth object
* test(prometheus): cover a failed request with no carried org
* fix(prometheus): move the committed counters for charges with no auth on the row
A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them.
* fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes
Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a
charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed
counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys
and teams in bounded pages.
* fix(prometheus): count a charged failure's own cost in the org gauge
A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request
* fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog
* fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete
Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins
* fix(prometheus): keep committed spend values out of logger dumps of pass-through rows
CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values
* fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster
The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files
* fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores
RedisCache and the native Redis client each kept their own breaker, so once
the cache learned the coordination Redis was wedged, a spend-counter write
through the native store still paid a full socket timeout. Every Redis target
now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected
into both the RedisCache guard and the native client. The Python FSM is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(rust): cargo fmt the shared breaker
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(redis): close the breaker when the recovery probe gets a Redis reply error
A command error such as WRONGTYPE means Redis answered, but the probe was left
unsettled and its drop reopened the breaker for another recovery timeout. Both
the RedisCache guard and the native client now settle such a probe as a success.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test
The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw.
* fix(redis): key the shared breaker by server and close the three audit gaps
The breaker registry now keys on the Redis server (host and port, unix socket,
cluster node set or sentinel service) instead of the full connection target, so
caches that differ only in db, credentials or url spelling share one breaker.
A READONLY reply from a replica mid-failover no longer counts as Redis answering,
so it can't reset a run of failures. A new RedisCache refreshes the breaker state
gauge, which the deleted Python breaker did at construction. A shared native client
is only reused by a store that passes the same breaker.
* refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations
lib.rs now only declares modules and re-exports, with resolution, target, tls,
auth and endpoint each in their own file. Derived FromPyObject already reads a
struct field as the attribute of the same name, so #[pyo3(attribute)] on the
breaker and store settings did nothing.
* refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot
Derived FromPyObject already reads each field as the attribute of the same name.
The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in
the same order.
* refactor(redis): return a HashMap from the three memory-tier snapshots
The snapshots read cache-memory's HashMap, so there is no order to keep.
PyO3 builds the same dict from a std HashMap, which replaces the hand-written
PyDict loops in the spend counter and router usage stores and the IndexMap in
the cooldown store. The rate-limit slot registry keeps IndexMap because it
hands back a Python dict in the order Python wrote it.
* refactor(redis): share Redis breakers and clients through one RedisHosts per host
redis-connect's RedisHosts now owns the per-server breaker and the shared clients
that python-bridge kept in two process-wide statics. A client takes breaker
settings and admits through its server's breaker, so no caller can pair a client
with a breaker of its own, and clients are keyed by Resolution alone. This drops
RedisClient::guarded_by and Breaker::same.
python-bridge keeps one RedisHosts per process and calls it with the GIL released.
The gateway builds one per connect, so its coordination and response cache on the
same server now share a breaker. RedisStoreSettings.breaker is back to breaker
settings, as on main.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(redis): route the router, rate limiters and auth through the native store families
* style(redis): satisfy the python lint gates
Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): give the registry's default usage store a memory-only handle
RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): build stores from native handles and wait for router tasks
Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Reworks the redis-client, connect, resolve and testing crates, adds python-compat json loads and
TTL seconds, replaces the redis-store memory tier with a DualCache plus a shared failure log, and
gives the Python side the Redis timeout log throttle and a request OTEL parent span for service events
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
