From 8abb1c97aa496128272616a0832a5002662df409 Mon Sep 17 00:00:00 2001 From: Yujong Lee Date: Thu, 29 Oct 2026 18:34:37 -0700 Subject: [PATCH] refactor(redis-store): add the store layer and its rules Co-Authored-By: Claude Opus 5.5 --- .agents/skills/redis-access/SKILL.md | 14 +++- .../skills/redis-access/references/store.md | 84 +++++++++++++++++++ litellm-rust/Cargo.lock | 4 + litellm-rust/Cargo.toml | 1 + litellm-rust/crates/redis-store/AGENTS.md | 22 +++++ litellm-rust/crates/redis-store/Cargo.toml | 6 ++ litellm-rust/crates/redis-store/src/lib.rs | 0 litellm/redis/AGENTS.md | 3 + litellm/redis/stores/__init__.py | 0 9 files changed, 130 insertions(+), 4 deletions(-) create mode 100644 .agents/skills/redis-access/references/store.md create mode 100644 litellm-rust/crates/redis-store/AGENTS.md create mode 100644 litellm-rust/crates/redis-store/Cargo.toml create mode 100644 litellm-rust/crates/redis-store/src/lib.rs create mode 100644 litellm/redis/stores/__init__.py diff --git a/.agents/skills/redis-access/SKILL.md b/.agents/skills/redis-access/SKILL.md index 7f77801623d..161ba2b6e75 100644 --- a/.agents/skills/redis-access/SKILL.md +++ b/.agents/skills/redis-access/SKILL.md @@ -1,17 +1,23 @@ --- name: redis-access -description: Add Redis operations or move raw redis-py clients and scripts behind RedisTransport in litellm/redis, without porting them to Rust or expanding migration scope +description: Add Redis operations, move raw redis-py clients, scripts or keyspaces behind RedisTransport and stores in litellm/redis, without porting them to Rust or expanding migration scope metadata: internal: true --- # Redis access -Python reaches Redis through `RedisTransport` (`litellm/redis/transport/contract.py`). Read `references/transport.md` for adding a command, the failure kinds, the namespace and TTL rules, and moving raw `init_async_client` / `async_register_script` / `.redis_client` access onto the transport +Python reaches Redis through `RedisTransport` (`litellm/redis/transport/contract.py`), and each keyspace lives behind a store in `litellm/redis/stores/`. Callers get typed domain results from a store and keep policy (limits, fail open or closed, retries, logging) + +Read the reference for the task + +- `references/transport.md` for adding a command, the failure kinds, the namespace and TTL rules, and moving raw `init_async_client` / `async_register_script` / `.redis_client` access onto the transport +- `references/store.md` for adding coordination state, moving a keyspace behind a store, store tests, and the Rust port Rules that always apply - New code never calls `init_async_client`, `init_pubsub_client`, `async_register_script` or `.redis_client`, and never imports redis-py outside `litellm/redis/transport/redis_py.py` -- Keys and values in Redis are a wire format shared with older pods and Rust. A move keeps them byte for byte +- New coordination state gets a store, not a new `RedisCache` method, and a Python store doesn't need a Rust port +- Keys and values in Redis are a wire format shared with older pods and Rust. A move keeps them byte for byte, and a change is a versioned migration of its own - A `transport` failure on a write is an unknown outcome, never "not applied" -- Do only what was asked. Confirm before any Rust work +- Do only what was asked. An extraction isn't a store move, and a store move isn't a Rust port. Confirm before any Rust work diff --git a/.agents/skills/redis-access/references/store.md b/.agents/skills/redis-access/references/store.md new file mode 100644 index 00000000000..3bd9b9850f2 --- /dev/null +++ b/.agents/skills/redis-access/references/store.md @@ -0,0 +1,84 @@ +# Redis stores + +A store owns one keyspace. Read this before adding coordination state, moving a keyspace out of its caller, or porting a store to Rust + +## Why stores + +Keys, encoding, TTLs, Lua and reply decoding for each keyspace are spread across Python modules today, and Rust already has a second response-cache key builder in `cache_response::cache_key`. Python and Rust pods share one Redis, so if their keys or values differ they silently split counters, cooldowns or locks. Each shared keyspace needs one definition of its data and one set of operations, and the store is that definition + +## Shape + +- One module per keyspace in `litellm/redis/stores/.py` +- The module defines a `Protocol` of domain operations, named for what the caller wants (`acquire`, `release`, `increment`, `get`), never for Redis commands +- The Python implementation is a frozen dataclass that takes a `RedisTransport` and runs its own commands on it. It owns, and nothing outside it knows + - the key builders, with logical keys and no namespace + - the value codec + - expiry, taken from the caller + - Lua, once the transport runs it + - reply decoding into typed results +- Every operation returns its result or a `RedisFailure`. A value that doesn't decode is a `decode` failure, never a miss or a default +- No logging, no retries, no fallback. Those are policy + +```python +class SessionBudgetStore(Protocol): + async def increment(self, session_id: str, cost: float, *, ttl: timedelta) -> float | RedisFailure: ... + async def read(self, session_id: str) -> float | None | RedisFailure: ... + + +@dataclass(frozen=True, slots=True) +class TransportSessionBudgetStore: + transport: RedisTransport + ... +``` + +## Callers + +- Depend on the keyspace's `Protocol`, built once from `redis_cache.transport()` and injected where the caller can take it +- `match` the result, ending in `assert_never` when it has more than two cases +- Keep policy in the caller: which TTL applies, fail open or closed, and the `with_service_target` that labels its commands. The transport already logs failures and emits service events +- A caller never builds a key, touches a client or runs a command for a keyspace that has a store. Exposing a key builder for an existing public helper is fine as a delegate + +## Adding new coordination state + +Add a store, not a new `RedisCache` method. A Python store is the finished state for now, and adding one does not require a Rust port + +## Moving an existing keyspace into a store + +- Move the key builders, codec, TTLs, Lua and decoding verbatim. Keep policy in the caller +- Keys and values already in Redis are a wire format. Old and new pods read each other's writes during a rolling deploy, so a move keeps them byte for byte +- Changing a key, codec or TTL unit is never part of a move. It is its own change that writes a new versioned key and reads both for one release +- Move one keyspace per change + +Keyspaces still in their callers, as candidates (paths under `litellm/`, not exhaustive) + +- response cache, `caching/caching.py` +- rate limit windows, `proxy/hooks/parallel_request_limiter_v3.py` +- deployment cooldowns, `router_utils/cooldown_cache.py` +- routing affinity, `caching/affinity_cache.py` +- pod and MCP refresh locks, `proxy/db/db_transaction_queue/pod_lock_manager.py` and `proxy/mcp_server/outbound_credentials/redis_distributed_lock.py`, which share one owner-checked release +- login throttle, `proxy/auth/login_throttle.py` +- session caps, `proxy/hooks/max_budget_per_session_limiter.py` and `max_iterations_limiter.py`, which share one counter pattern +- enqueued batch tokens, `proxy/hooks/batch_enqueued_tokens.py` +- spend counters, `proxy/spend_tracking/spend_counter_batch.py` + +Response cache identity (normalization, hashing into a logical key) is backend-independent and is not store logic. The store only maps that logical key to storage. Reuse or move `cache_response::cache_key`, never write a third key builder + +## Tests + +- Test the store through its caller against `fake_redis_cache(server)` from `tests/_support/fake_redis.py`. That is a real `RedisCache` on a `fakeredis.FakeServer`, so namespace, breaker and failure mapping are the production ones +- Cover each result the caller acts on, that the TTL lands (`pttl` on a raw `FakeAsyncRedis`), and an outage (`server.connected = False`) +- When a keyspace moves, seed with the old writer and read back with the old reader, so old and new pods read each other's writes +- Assert returned values and Redis state, never the commands sent + +## Porting a store to Rust + +Confirm with the user before starting the first port, since the bridge for it doesn't exist yet + +- The keyspace's module in `litellm-rust/crates/redis-store` implements the same operations as async fns on `crates/redis-client`. Its `AGENTS.md` has the crate rules +- A `#[pyclass]` per store in `litellm-rust/crates/python-bridge/src/redis/` wraps that module, built with the shared `NativeRedisClient`, so Python opens no extra connections +- A `NativeStore` in the Python module implements the same `Protocol` over the pyclass, so callers don't change. The implementation is picked at construction, and the transport-backed one stays for config the native client doesn't support (Sentinel, cloud auth, TLS cert options) +- Results come back as Pydantic models generated from the Rust types, frozen, so callers keep matching on the same classes. A decode error maps to `RedisFailure(kind="decode")` +- Rust callers use the same module, so there is one implementation per operation +- Native stores skip the Python guard, so the breaker and the REDIS service events have to come from Rust before the first port +- The codec has to write the exact bytes Python writes. Pod lock owners are `json.dumps` output, which escapes non-ASCII, and `serde_json` doesn't, so a Rust pod couldn't release a lock a Python pod took +- Each port ships cross-implementation tests against one Redis, covering keys, values, expiry and atomicity through a rolling deploy diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock index e959f77917b..1c82935c901 100644 --- a/litellm-rust/Cargo.lock +++ b/litellm-rust/Cargo.lock @@ -4402,6 +4402,10 @@ dependencies = [ "tokio", ] +[[package]] +name = "litellm-redis-store" +version = "0.1.0" + [[package]] name = "litellm-redis-testing" version = "0.1.0" diff --git a/litellm-rust/Cargo.toml b/litellm-rust/Cargo.toml index 49a33e08456..20f007d9176 100644 --- a/litellm-rust/Cargo.toml +++ b/litellm-rust/Cargo.toml @@ -72,6 +72,7 @@ litellm-host-python = { path = "crates/host-python" } litellm-python-compat = { path = "crates/python-compat" } litellm-circuit-breaker = { path = "crates/circuit-breaker" } litellm-redis-client = { path = "crates/redis-client" } +litellm-redis-store = { path = "crates/redis-store" } litellm-redis-testing = { path = "crates/redis-testing" } litellm-cost = { path = "crates/cost" } litellm-model-catalog = { path = "crates/model-catalog" } diff --git a/litellm-rust/crates/redis-store/AGENTS.md b/litellm-rust/crates/redis-store/AGENTS.md new file mode 100644 index 00000000000..e206ec6ab7a --- /dev/null +++ b/litellm-rust/crates/redis-store/AGENTS.md @@ -0,0 +1,22 @@ +- Redis-backed domain operations shared by the Rust gateway and Python, like a `crates/db` repository. Callers ask for domain operations (`acquire`, `increment`, `get`) and never see a Redis command +- One module per keyspace (rate limit windows, session caps, enqueued tokens, cooldowns, affinity pins, locks, login throttle, spend counters, update buffers, response polling). The module owns its key layout, value codec, expiry, Lua and reply decoding, and nothing outside it builds those keys +- Operations are async fns that take a `&litellm_redis_client::RedisClient` and run their commands on it. The client applies the namespace and the breaker, so a store never does either and never goes back through Python's `RedisTransport` +- Results are domain types (`Acquire::Taken`, `Option`), never a raw `redis::Value`. A reply that doesn't decode is an error, never a miss or a default count +- Callers keep policy, like which limit applies, fail open or closed, retries and logging + +## Wire contract +- Keys, value bytes, TTL units and Lua behavior are a wire format shared with Python pods and older releases on the same Redis. A difference doesn't error, it silently splits counters, budgets or locks between pods during a rolling deploy or rollback +- A keyspace ported from Python keeps the bytes Python writes exactly, including hash tags, `json.dumps` spacing and ASCII escaping, float formatting and seconds vs milliseconds +- Never change a key, codec or TTL unit in place. Write under a new versioned key, read both for one release, then drop the old one, as its own change + +## Shape +- Logical keys only, the client adds the namespace. Lua that touches more than one key keeps them in one slot with a `{hash tag}`, and declares every key in KEYS +- Lua is a `&'static str` next to the operation that runs it +- Response-cache identity (normalization, hashing into a logical key) is not store logic. This crate only maps that logical key to storage. Reuse or move `cache-response`'s key builder, never write a third +- Pub/sub, admin commands and vector search are not keyspaces and don't belong here +- Python reaches a store through one `#[pyclass]` per module in `crates/python-bridge/src/redis/`, built with the shared `NativeRedisClient`, behind the keyspace's Python store Protocol in `litellm/redis/stores/` + +## Tests +- Run operations against real Redis from `litellm-redis-testing`, on each topology through `#[values(Topology::Standalone, Topology::Cluster)]`. Never assert command bytes or a mocked wire +- Assert decoded results, TTLs and atomicity through the store's own operations and Redis state +- A keyspace ported from Python ships cross-implementation tests against one Redis, where the Python store writes and this crate reads and the reverse diff --git a/litellm-rust/crates/redis-store/Cargo.toml b/litellm-rust/crates/redis-store/Cargo.toml new file mode 100644 index 00000000000..6a3b8d41bf3 --- /dev/null +++ b/litellm-rust/crates/redis-store/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "litellm-redis-store" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true diff --git a/litellm-rust/crates/redis-store/src/lib.rs b/litellm-rust/crates/redis-store/src/lib.rs new file mode 100644 index 00000000000..e69de29bb2d diff --git a/litellm/redis/AGENTS.md b/litellm/redis/AGENTS.md index 7f035e4b385..2dcc68456df 100644 --- a/litellm/redis/AGENTS.md +++ b/litellm/redis/AGENTS.md @@ -9,3 +9,6 @@ For adding Redis access or moving raw Redis access here, follow [.agents/skills/ - `transport/redis_py.py` is the only module that touches redis-py's client. The breaker and the exception mapping come from `RedisCache` through the injected guard - Every backend runs each command through a `Guard` that emits one REDIS service event and logs Redis health failures, so swapping backends never drops metrics. Callers don't log Redis health - `litellm/caching` imports `litellm/redis`, never the reverse +- `stores/` holds one module per keyspace, and a keyspace that has a store is only reached through it. A store is a `Protocol` of domain operations plus implementations of it. It owns its keys, value codec, expiry, Lua and decoding, does its own IO, and callers never see a Redis command +- The Python implementation runs on any `RedisTransport`. A native implementation wraps the keyspace's `crates/redis-store` module and runs on `crates/redis-client` directly, never through `RedisTransport` +- Keys, value bytes and TTL units in Redis are a wire format shared with older pods and Rust. Never change them in place, see `crates/redis-store/AGENTS.md` diff --git a/litellm/redis/stores/__init__.py b/litellm/redis/stores/__init__.py new file mode 100644 index 00000000000..e69de29bb2d