scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9
litellm/tests/test_litellm/caching
History
Exact
Cursor Agent eb1a13f7a7 fix: don't close httpx clients on cache eviction (fixes 'client has been closed' errors) 
Root cause: LLMClientCache._remove_key() was calling aclose()/close() on
httpx clients when their cache entries expired (TTL) or were evicted (LRU).
However, callers of get_async_httpx_client() hold direct references to
these client objects (e.g. litellm.module_level_aclient, streaming handlers,
passthrough request handlers). Closing the underlying httpx client while
these references are still in use causes:

    RuntimeError: Cannot send a request, as the client has been closed.

This was introduced in commits 6931fea9 and bfeed0e5 (Feb 20, 2026) which
added close-on-eviction to prevent connection pool leaks. The fix removes
the close-on-eviction behavior: when a cache entry is evicted, only the
cache's reference is dropped. Python GC will finalize transport/connection
pool resources when all external references are released.

The bug manifests after ~1 hour of traffic (_DEFAULT_TTL_FOR_HTTPX_CLIENTS
= 3600s) when cache entries expire, or earlier under high concurrency if
the 200-entry LRU limit is reached. It affects all providers using the
shared httpx client cache (Vertex AI, Bedrock, etc.).

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
2026-02-26 17:59:20 +00:00
..
test_azure_blob_cache.py add azure blob cache support (#12587) 2025-07-15 11:47:38 -07:00
test_caching_handler.py Helicone base url support + fix for embedding cache hits on str input (#11211) 2025-05-28 22:02:55 -07:00
test_dual_cache.py fix(cache): prevent DualCache async batch check-then-act race (#20986) 2026-02-13 18:32:41 +05:30
test_gcs_cache.py Add GCS bucket caching support (#13122) 2025-08-04 16:09:33 -07:00
test_in_memory_cache.py [Performance] Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) (#15000) 2025-09-30 16:49:35 -07:00
test_qdrant_semantic_cache.py feat(semantic-cache): support configurable vector dimensions for Qdrant (#21649) 2026-02-21 00:51:15 -08:00
test_redis_cache.py fix(proxy): add LPOP pipeline error checking and fix org spend ServiceType 2026-02-24 14:22:57 -08:00
test_redis_cluster_cache.py Litellm fix GitHub action testing (#11163) 2025-05-26 14:41:42 -07:00
test_redis_connection_pool.py fix: don't close httpx clients on cache eviction (fixes 'client has been closed' errors) 2026-02-26 17:59:20 +00:00
test_redis_semantic_cache.py Litellm fix GitHub action testing (#11163) 2025-05-26 14:41:42 -07:00
test_s3_cache.py verify expires field prior to serving cache entry 2025-08-25 10:42:12 +02:00