litellm/litellm/caching
norkelbyte e0778bc9dd
refactor(types): replace Any with proven types in 21 files (#48921)
* refactor(types): replace Any with proven types in 35 files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): keep anthropic chat transformation json path untyped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): accept null gemini vector store fields and load encryption TypedDicts on python 3.10

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): validate Python client user responses and type delete_user as the count it returns

* fix(types): keep loose payloads working under the sweep's validators

An off-schema Anthropic stream event is now skipped instead of ending the
guardrail's text and stop_reason scan, and MAI image usage accepts the loose
numbers it did before. New dict and list adapters hide their input from
validation errors, RerankResponse indexing keeps its field types, and the
delete_user test mocks the int /user/delete returns.

* fix(types): make the changed DynamoAI fields ReadOnly and skip non-object A2A stream items

* refactor(types): tighten round 2 parse sites and keep payloads out of errors

Parse Anthropic SSE events with validate_json and ignore extra fields, validate admin team and key rows per row with hidden inputs, type rerank get and __getitem__ per key, and move the Phoenix prompt version response to a frozen BaseModel

* fix(types): keep MAI image calls alive on unreadable usage counts

An unreadable MAI usage count now counts as absent instead of failing a
billed image call, and the provider's usage fields pass through as sent.
The Anthropic stream scan warnings no longer log raw model output, and the
off-schema skip has tests that fail on the old scan.

* fix(types): count non-object MAI usage as absent and parse Anthropic SSE with json.loads

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 18:43:24 -07:00
..
__init__.py chore(caching)!: remove the disk cache backend (#48114) 2026-10-29 18:02:31 -07:00
_embedding_router.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
_internal_lru_cache.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
affinity_cache.py fix(router): keep session tier pins coherent across workers (#48645) 2026-10-27 18:22:39 -07:00
azure_blob_cache.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
base_cache.py refactor(typing): widen delete_nested_value to Mapping and drop two unused Span aliases (#48001) 2026-10-24 19:39:27 +00:00
caching.py refactor(types): replace Any with proven types in 21 files (#48921) 2026-10-29 18:43:24 -07:00
caching_handler.py chore(caching)!: remove the disk cache backend (#48114) 2026-10-29 18:02:31 -07:00
dual_cache.py fix(proxy)!: stop update_cache from resurrecting evicted auth cache entries (#46997) 2026-10-20 17:09:53 +00:00
evicted_client_closer.py fix(caching): key cached async clients by a per-loop token instead of id(loop) (#48877) 2026-10-29 13:14:48 -07:00
gcs_cache.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
in_memory_cache.py feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits (#48875) 2026-10-28 15:52:44 -07:00
llm_caching_handler.py fix(caching): key cached async clients by a per-loop token instead of id(loop) (#48877) 2026-10-29 13:14:48 -07:00
qdrant_semantic_cache.py refactor(types): replace Any with proven types in 14 files (#48928) 2026-10-29 07:26:07 -07:00
Readme.md chore(caching)!: remove the disk cache backend (#48114) 2026-10-29 18:02:31 -07:00
redis_batch.py fix(otel): nest cache spans under their operation and name service spans by purpose (#44150) 2026-10-03 09:20:27 -07:00
redis_cache.py feat(router): add opt-in session repinning after successful failover (#48206) 2026-10-24 13:00:58 -07:00
redis_cluster_cache.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
redis_cluster_node_isolation.py fix(caching): keep a node timeout from forcing a cluster-wide topology reinit on redis-py 8.x (#39349) 2026-09-03 17:31:58 -07:00
redis_semantic_cache.py refactor(types): replace Any with proven types in 14 files (#48928) 2026-10-29 07:26:07 -07:00
s3_cache.py chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom 2026-08-05 02:37:24 -07:00
valkey_semantic_cache.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00

Caching on LiteLLM

LiteLLM supports multiple caching mechanisms. This allows users to choose the most suitable caching solution for their use case.

The following caching mechanisms are supported:

  1. RedisCache
  2. RedisSemanticCache
  3. QdrantSemanticCache
  4. InMemoryCache
  5. S3Cache
  6. AzureBlobCache
  7. DualCache (updates both Redis and an in-memory cache simultaneously)

Folder Structure

litellm/caching/
├── base_cache.py
├── caching.py
├── caching_handler.py
├── dual_cache.py
├── in_memory_cache.py
├── qdrant_semantic_cache.py
├── redis_cache.py
├── redis_semantic_cache.py
├── s3_cache.py

Documentation