scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-12 08:42:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 207
litellm/litellm/caching
History
Exact
Emerson Gomes f8933ecae4
fix(redis): free each request's Redis batch scope once the request ends (#50751) 
* fix(spend): respect use_redis_transaction_buffer when selecting the SpendLogs ledger

* revert: keep the SpendLogs ledger independent of use_redis_transaction_buffer

Measured on a live proxy, ledger on and ledger off grow RSS at the same rate per request,
so gating the ledger on the Redis buffer moved deployments off it without fixing the growth.

* fix(redis): free each request's Redis batch scope once the request ends

The native request batch keeps a copy of the context it is built in. It was built inside the
request scope, so that context pointed back at the scope, and the cycle collector cannot see
through the native object. Every request's batches, spend counter batches and native Redis
state stayed alive, growing worker RSS about 40 KiB per request and slowing each request as
the heap grew. Build the native batch in the context the scope was opened from instead.
2026-11-10 16:14:12 +00:00
..
__init__.py chore(caching)!: remove the disk cache backend (#48114) 2026-10-29 18:02:31 -07:00
_embedding_router.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
_internal_lru_cache.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
affinity_cache.py fix(router): keep session tier pins coherent across workers (#48645) 2026-10-27 18:22:39 -07:00
azure_blob_cache.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
base_cache.py refactor(typing): widen delete_nested_value to Mapping and drop two unused Span aliases (#48001) 2026-10-24 19:39:27 +00:00
caching.py refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
caching_handler.py fix(caching): don't cache or replay a reply that has no output (#50057) 2026-11-06 21:32:59 -08:00
dual_cache.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
evicted_client_closer.py fix(caching): key cached async clients by a per-loop token instead of id(loop) (#48877) 2026-10-29 13:14:48 -07:00
gcs_cache.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
in_memory_cache.py feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits (#48875) 2026-10-28 15:52:44 -07:00
llm_caching_handler.py fix(caching): key cached async clients by a per-loop token instead of id(loop) (#48877) 2026-10-29 13:14:48 -07:00
qdrant_semantic_cache.py refactor(types): replace Any with proven types in 14 files (#48928) 2026-10-29 07:26:07 -07:00
Readme.md chore(caching)!: remove the disk cache backend (#48114) 2026-10-29 18:02:31 -07:00
redis_batch.py fix(redis): free each request's Redis batch scope once the request ends (#50751) 2026-11-10 16:14:12 +00:00
redis_cache.py fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) 2026-11-07 17:50:06 +09:00
redis_cluster_cache.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
redis_cluster_node_isolation.py fix(caching): keep a node timeout from forcing a cluster-wide topology reinit on redis-py 8.x (#39349) 2026-09-03 17:31:58 -07:00
redis_semantic_cache.py refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
s3_cache.py chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom 2026-08-05 02:37:24 -07:00
valkey_semantic_cache.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00

Readme.md

Caching on LiteLLM

LiteLLM supports multiple caching mechanisms. This allows users to choose the most suitable caching solution for their use case.

The following caching mechanisms are supported:

  1. RedisCache
  2. RedisSemanticCache
  3. QdrantSemanticCache
  4. InMemoryCache
  5. S3Cache
  6. AzureBlobCache
  7. DualCache (updates both Redis and an in-memory cache simultaneously)

Folder Structure

litellm/caching/
├── base_cache.py
├── caching.py
├── caching_handler.py
├── dual_cache.py
├── in_memory_cache.py
├── qdrant_semantic_cache.py
├── redis_cache.py
├── redis_semantic_cache.py
├── s3_cache.py

Documentation