scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-16 23:41:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102
litellm/litellm/caching
History
mateo-berri ef5f51abca fix(least-busy): count for every router, and keep the expiry through a clamp 
Two routers in one process shared a single handler, because the callback
manager dedupes on the class name plus the handler's public attributes and the
handler had none. The second router's requests were never counted. The handler
now carries the id of the cache it was built on, so routers with different
caches both register while the two selectors one router builds for its routing
groups still collapse into one.

Clamping a negative count back to zero used SET, which drops the key's TTL, so
the next write started the hour over. It uses INCRBY by the negative amount now,
which leaves the expiry alone.

The Lua script had no test that ran it, so tests/local_testing covers both the
sync and async paths against a real Redis, and the file is wired into the
CircleCI job that provides one.
2026-09-06 01:59:15 -07:00
..
__init__.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
_embedding_router.py fix(caching): bound the semantic cache embedding lookup 2026-08-20 16:09:30 -07:00
_internal_lru_cache.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
azure_blob_cache.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
base_cache.py chore(lint): clear grandfathered over-limit lint drift and ratchet budgets down 2026-08-05 12:18:13 -07:00
caching.py feat(caching): add semantic_cache_scope to isolate semantic cache hits per end user (#39590) 2026-09-03 18:44:54 -07:00
caching_handler.py refactor(typing): replace Any with proven types in 42 more backend files 2026-09-02 15:35:01 +00:00
disk_cache.py chore(lint): clear grandfathered over-limit lint drift and ratchet budgets down 2026-08-05 12:18:13 -07:00
dual_cache.py refactor: drop fresh Any annotations and suppressions from admission control, spend summary, and dual cache 2026-09-04 08:05:04 +00:00
evicted_client_closer.py fix(caching): clear strict-lint budget breaches in re-landed closer code 2026-08-05 11:08:19 -07:00
gcs_cache.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
in_memory_cache.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
llm_caching_handler.py fix(caching): clear strict-lint budget breaches in re-landed closer code 2026-08-05 11:08:19 -07:00
qdrant_semantic_cache.py refactor(types): replace Any with precise types across 73 modules 2026-09-01 11:05:02 +00:00
Readme.md add azure blob cache support (#12587) 2025-07-15 11:47:38 -07:00
redis_cache.py fix(least-busy): count for every router, and keep the expiry through a clamp 2026-09-06 01:59:15 -07:00
redis_cluster_cache.py fix(redis): support credential providers across clients 2026-08-24 19:07:13 -04:00
redis_cluster_node_isolation.py fix(caching): keep a node timeout from forcing a cluster-wide topology reinit on redis-py 8.x (#39349) 2026-09-03 17:31:58 -07:00
redis_semantic_cache.py refactor(typing): replace Any with proven types in 65 backend files 2026-09-02 09:11:36 +00:00
s3_cache.py chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom 2026-08-05 02:37:24 -07:00
valkey_semantic_cache.py chore(lint): fix post-merge type regressions and ratchet lint budgets 2026-08-29 20:16:04 +00:00

Readme.md

Caching on LiteLLM

LiteLLM supports multiple caching mechanisms. This allows users to choose the most suitable caching solution for their use case.

The following caching mechanisms are supported:

  1. RedisCache
  2. RedisSemanticCache
  3. QdrantSemanticCache
  4. InMemoryCache
  5. DiskCache
  6. S3Cache
  7. AzureBlobCache
  8. DualCache (updates both Redis and an in-memory cache simultaneously)

Folder Structure

litellm/caching/
├── base_cache.py
├── caching.py
├── caching_handler.py
├── disk_cache.py
├── dual_cache.py
├── in_memory_cache.py
├── qdrant_semantic_cache.py
├── redis_cache.py
├── redis_semantic_cache.py
├── s3_cache.py

Documentation