mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
a06404451d
Root cause: LLMClientCache._remove_key() was closing httpx clients when
they were evicted from cache (TTL or size-based). But other code still
held references to those clients (e.g., litellm.module_level_aclient
stored in module __dict__, in-flight requests). This caused
RuntimeError('Cannot send a request, as the client has been closed.')
for all subsequent users of those client references.
The issue manifested after ~1 hour (TTL=3600s) of uptime, as clients
were evicted from the LLMClientCache. It persisted until pod restart
because litellm.module_level_aclient in the module __dict__ was never
re-created (Python's __getattr__ is not called when the attribute
already exists in __dict__).
Fix (3 layers of defense):
1. LLMClientCache._remove_key() no longer closes clients on eviction.
Client cleanup is deferred to GC (__del__) and atexit handlers,
which only run when no references remain.
2. get_async_httpx_client() and _get_httpx_client() now check if a
cached client's underlying httpx client is closed before returning
it. If closed, they create a new one.
3. AsyncHTTPHandler.post/put/patch/delete/get catch RuntimeError with
'client has been closed' and transparently retry with a fresh
httpx.AsyncClient.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| test_azure_blob_cache.py | ||
| test_caching_handler.py | ||
| test_dual_cache.py | ||
| test_gcs_cache.py | ||
| test_in_memory_cache.py | ||
| test_qdrant_semantic_cache.py | ||
| test_redis_cache.py | ||
| test_redis_cluster_cache.py | ||
| test_redis_connection_pool.py | ||
| test_redis_semantic_cache.py | ||
| test_s3_cache.py | ||