mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
ae8dc1f39f
With enable_redis_auth_cache and multiple replicas, /key/update and
/key/delete delete the Redis auth blob and the handling pod's in-memory
entry, but two read-path writers re-published the stale blob from any other
replica's per-pod memory back to Redis with a fresh 60s TTL on every
request: the post-auth re-cache in user_api_key_auth and the spend writeback
in update_cache. Replicas whose in-memory entries expired then re-primed
themselves from the poisoned Redis entry, so key limit and access changes
never took effect fleet-wide while traffic continued, and a deleted key kept
authenticating.
The auth object is now written only by the DB-load paths
(IdentityStore._resolve_key, get_key_object): the post-auth re-cache is
removed outright (even a local-only write could race an invalidation and
resurrect a revoked key on this worker) and spend tracking no longer writes
the auth object back at all; spend is tracked through the spend🔑*
counters. The remaining spend writebacks for user, team, end-user, and tag
objects become local-only so they cannot republish stale management objects
either, with one deliberate exception: the proxy-wide
{litellm_proxy_admin_name}:spend scalar keeps its shared Redis write because
the global max_budget check reads it between authoritative DB reloads, and
it carries no limits or permissions so sharing it cannot resurrect an
invalidated auth blob.
DualCache's redis-to-memory read backfill also ignored default_in_memory_ttl,
pinning backfilled entries for InMemoryCache's 600s default instead of the
configured 60s auth TTL; the backfill now injects the configured default like
every write path already does, so a replica primed from Redis converges
within the auth cache TTL as well.
Consolidates the sibling stale-auth-recache branch; the delete-propagation
case is the duplicate ticket LIT-4350.
Resolves LIT-4219
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _embedding_router.py | ||
| _internal_lru_cache.py | ||
| azure_blob_cache.py | ||
| base_cache.py | ||
| caching.py | ||
| caching_handler.py | ||
| disk_cache.py | ||
| dual_cache.py | ||
| gcs_cache.py | ||
| in_memory_cache.py | ||
| llm_caching_handler.py | ||
| qdrant_semantic_cache.py | ||
| Readme.md | ||
| redis_cache.py | ||
| redis_cluster_cache.py | ||
| redis_semantic_cache.py | ||
| s3_cache.py | ||
| valkey_semantic_cache.py | ||
Caching on LiteLLM
LiteLLM supports multiple caching mechanisms. This allows users to choose the most suitable caching solution for their use case.
The following caching mechanisms are supported:
- RedisCache
- RedisSemanticCache
- QdrantSemanticCache
- InMemoryCache
- DiskCache
- S3Cache
- AzureBlobCache
- DualCache (updates both Redis and an in-memory cache simultaneously)
Folder Structure
litellm/caching/
├── base_cache.py
├── caching.py
├── caching_handler.py
├── disk_cache.py
├── dual_cache.py
├── in_memory_cache.py
├── qdrant_semantic_cache.py
├── redis_cache.py
├── redis_semantic_cache.py
├── s3_cache.py