litellm/litellm-rust/crates/redis-connect/AGENTS.md at beluga_decrease_anys_20261107

mirror of https://github.com/BerriAI/litellm.git synced 2026-11-13 08:52:02 +00:00

fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473 )

* fix(redis): share one native circuit breaker between RedisCache and the Rust stores

RedisCache and the native Redis client each kept their own breaker, so once
the cache learned the coordination Redis was wedged, a spend-counter write
through the native store still paid a full socket timeout. Every Redis target
now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected
into both the RedisCache guard and the native client. The Python FSM is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(rust): cargo fmt the shared breaker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(redis): close the breaker when the recovery probe gets a Redis reply error

A command error such as WRONGTYPE means Redis answered, but the probe was left
unsettled and its drop reopened the breaker for another recovery timeout. Both
the RedisCache guard and the native client now settle such a probe as a success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test

The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw.

* fix(redis): key the shared breaker by server and close the three audit gaps

The breaker registry now keys on the Redis server (host and port, unix socket,
cluster node set or sentinel service) instead of the full connection target, so
caches that differ only in db, credentials or url spelling share one breaker.

A READONLY reply from a replica mid-failover no longer counts as Redis answering,
so it can't reset a run of failures. A new RedisCache refreshes the breaker state
gauge, which the deleted Python breaker did at construction. A shared native client
is only reused by a store that passes the same breaker.

* refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations

lib.rs now only declares modules and re-exports, with resolution, target, tls,
auth and endpoint each in their own file. Derived FromPyObject already reads a
struct field as the attribute of the same name, so #[pyo3(attribute)] on the
breaker and store settings did nothing.

* refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot

Derived FromPyObject already reads each field as the attribute of the same name.
The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in
the same order.

* refactor(redis): return a HashMap from the three memory-tier snapshots

The snapshots read cache-memory's HashMap, so there is no order to keep.
PyO3 builds the same dict from a std HashMap, which replaces the hand-written
PyDict loops in the spend counter and router usage stores and the IndexMap in
the cooldown store. The rate-limit slot registry keeps IndexMap because it
hands back a Python dict in the order Python wrote it.

* refactor(redis): share Redis breakers and clients through one RedisHosts per host

redis-connect's RedisHosts now owns the per-server breaker and the shared clients
that python-bridge kept in two process-wide statics. A client takes breaker
settings and admits through its server's breaker, so no caller can pair a client
with a breaker of its own, and clients are keyed by Resolution alone. This drops
RedisClient::guarded_by and Breaker::same.

python-bridge keeps one RedisHosts per process and calls it with the GIL released.
The gateway builds one per connect, so its coordination and response cache on the
same server now share a breaker. RedisStoreSettings.breaker is back to breaker
settings, as on main.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-07 17:50:06 +09:00

4 KiB

Raw Permalink Blame History

Turns a litellm_redis_types::Resolution into a RedisClient, the same way for every host. It depends on litellm-redis-types, never on litellm-config, so a host that resolves Redis some other way connects the same way. redis-client stays cloud-agnostic, so credential providers and cloud token sources are built here
The host passes only what it alone can build in HostServices: the event sink, the HTTP client and the environment lookup. The breaker config comes from Resolution::breaker, which config reads from REDIS_CIRCUIT_BREAKER_* with Python's fallback, so a bad value keeps the default instead of failing connect. Each cloud source asks its auth crate for one explicit identity, the way the Python SDK builds it: AwsAuthService::default_chain_credentials (botocore's default chain), CredentialSource::application_default (google.auth.default) and AzureIdentity (the class _build_azure_credential picks). LiteLLM's provider settings such as AWS_ROLE_NAME, AZURE_AD_TOKEN or VERTEXAI_CREDENTIALS belong to Bedrock, Azure OpenAI and Vertex and never reach those constructors. Auth::Provider (a handle for a Python credential_provider object) fails at connect with an error naming the key, since no host can supply one yet
RedisHosts is the one place a host shares Redis state. Breakers are keyed by Endpoint, so every caller on a server sees one health signal whatever its db or credentials. A client takes only breaker settings and admits through its server's breaker, so no caller can pair a client with a breaker of its own, and clients are keyed by Resolution alone. Both are held weakly, so a breaker or client is freed with its last user, and the lock is held while a client connects so each is built once. A caller holding the Python GIL releases it first. Hosts own one instance (python-bridge per process, the gateway per connect) and never keep a registry of their own
Every expiring credential goes through the one refresh loop in refresh.rs. A cloud source only fetches. Never add a second loop or let a source sleep. The expiry deadline lives only in the published Latest::Ready, so expiring it leaves Ready and cannot fire again. Never keep a copy of the deadline in the loop
AWS SDK identity settings must be in the process environment. aws-config cannot take the host lookup, so config-only overrides for profiles, role/web-identity settings, credentials, files and metadata sources fail before connecting rather than silently using another identity. Equal process/config values are allowed. Bedrock-only settings remain unrelated to Redis
redis-rs subscribes once for the connection handshake and again for re-authentication, so a new connection sends a redundant initial AUTH. Unchanged refreshes do not trigger another AUTH. Do not skip a provider's first or alternating subscriptions, because concurrent connections and retries each need the current token. Removing the initial duplicate requires an upstream connection-level fix
The loop starts on the first subscribe(), which redis-rs calls inside its runtime, so connect is safe off a runtime
A token-only credential (Azure without REDIS_USERNAME, GCP) is sent as AUTH default <token>, because redis-rs always sends a username with a credentials provider. On Redis 6+ that is the same as Python's AUTH <token>
Tests for the refresh loop use a fake source and real Redis ACL users. Cloud sources are proven against the live service, unit tests only pin request shapes we own
Rotation tests must retain the same server-side connection IDs across re-AUTH, then force a reconnect after removing the old user and verify the new identity still works
Live cloud tests live in tests/live/. Set LITELLM_REDIS_AWS_LIVE_CONFIG, LITELLM_REDIS_AZURE_LIVE_CONFIG or LITELLM_REDIS_GCP_LIVE_CONFIG to a JSON file containing redis settings and hold_seconds. Set the waiting period from the deployment's token lifetime when checking expiry. Optional fixtures return early when unset, and configured tests run under plain cargo test. An unconfigured run never counts as live proof

4 KiB Raw Permalink Blame History

4 KiB

Raw Permalink Blame History