scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8
litellm/tests/test_litellm/caching
History
Exact
Ishaan Jaff 503eb2fd4c
fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22925) 
* fix: don't close HTTP/SDK clients on LLMClientCache eviction

Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:

  RuntimeError: Cannot send a request, as the client has been closed.

This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().

Fixes production crashes after the 1-hour cache TTL expires.

* test: update LLMClientCache unit tests for no-close-on-eviction behavior

Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.

Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.

* test: add e2e tests for OpenAI SDK client surviving cache eviction

Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
  doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
  eviction doesn't close the client

Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.

Also expand the module docstring to explain why the sleep is required.

* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction

* docs(CLAUDE.md): add HTTP client cache safety guideline
2026-03-05 12:00:38 -08:00
..
test_azure_blob_cache.py add azure blob cache support (#12587) 2025-07-15 11:47:38 -07:00
test_caching_handler.py Helicone base url support + fix for embedding cache hits on str input (#11211) 2025-05-28 22:02:55 -07:00
test_dual_cache.py fix(cache): prevent DualCache async batch check-then-act race (#20986) 2026-02-13 18:32:41 +05:30
test_gcs_cache.py Add GCS bucket caching support (#13122) 2025-08-04 16:09:33 -07:00
test_in_memory_cache.py [Performance] Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) (#15000) 2025-09-30 16:49:35 -07:00
test_llm_caching_handler.py fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22925) 2026-03-05 12:00:38 -08:00
test_llm_client_cache_e2e.py fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22925) 2026-03-05 12:00:38 -08:00
test_qdrant_semantic_cache.py feat(semantic-cache): support configurable vector dimensions for Qdrant (#21649) 2026-02-21 00:51:15 -08:00
test_redis_cache.py fix(proxy): add LPOP pipeline error checking and fix org spend ServiceType 2026-02-24 14:22:57 -08:00
test_redis_cluster_cache.py Litellm fix GitHub action testing (#11163) 2025-05-26 14:41:42 -07:00
test_redis_connection_pool.py Revert "Add LLMClientCache regression tests for httpx client eviction safety" 2026-02-27 12:43:03 -08:00
test_redis_semantic_cache.py Litellm fix GitHub action testing (#11163) 2025-05-26 14:41:42 -07:00
test_s3_cache.py verify expires field prior to serving cache entry 2025-08-25 10:42:12 +02:00