From 0b934e126c7e2276d09499db6db94984baaa0c60 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Thu, 5 Mar 2026 19:27:22 +0000 Subject: [PATCH] docs: add cache eviction safety rule to AGENTS.md and CLAUDE.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prevent future regressions by documenting that LLMClientCache must never close HTTP/SDK clients on eviction — evicted clients may still be held by in-flight requests. Co-authored-by: Ishaan Jaff --- AGENTS.md | 2 ++ CLAUDE.md | 3 +++ 2 files changed, 5 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 546f2997bf5..d330961793a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -212,6 +212,8 @@ When opening issues or pull requests, follow these templates: Using helpers like `supports_reasoning` (which read from `model_prices_and_context_window.json` / `get_model_info`) allows future model updates to "just work" without code changes. +9. **Never close HTTP/SDK clients on cache eviction**: Do not add `close()`, `aclose()`, or `create_task(close_fn())` inside `LLMClientCache._remove_key()` or any cache eviction path. Evicted clients may still be held by in-flight requests; closing them causes `RuntimeError: Cannot send a request, as the client has been closed.` in production after the cache TTL (1 hour) expires. Connection cleanup is handled at shutdown by `close_litellm_async_clients()`. See PR #22247 for the full incident history. + ## HELPFUL RESOURCES - Main documentation: https://docs.litellm.ai/ diff --git a/CLAUDE.md b/CLAUDE.md index 5b36c2be8ac..104a751ecaf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -116,6 +116,9 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components: - Optional features enabled via environment variables - Separate licensing and authentication for enterprise features +### HTTP Client Cache Safety +- **Never close HTTP/SDK clients on cache eviction.** `LLMClientCache._remove_key()` must not call `close()`/`aclose()` on evicted clients — they may still be used by in-flight requests. Doing so causes `RuntimeError: Cannot send a request, as the client has been closed.` after the 1-hour TTL expires. Cleanup happens at shutdown via `close_litellm_async_clients()`. + ### Troubleshooting: DB schema out of sync after proxy restart `litellm-proxy-extras` runs `prisma migrate deploy` on startup using **its own** bundled migration files, which may lag behind schema changes in the current worktree. Symptoms: `Unknown column`, `Invalid prisma invocation`, or missing data on new fields.