The re-landed closer test asserted a reaped handler's client stays
closed; with #35862 the handler heals on next access, so the test now
pins the inner client up front and asserts the heal as the contract.
Also adds an end-to-end regression test that evicts an init-held
handler through LLMClientCache, waits out the grace close, and proves
the next request succeeds.
This reverts commit 66bc70365f and the
follow-up 2-line type fix a6d4654261 (#35706), which only retyped a
signature #35492 introduced.
Closing evicted litellm-owned clients breaks every object that fetches
get_async_httpx_client once in __init__ and holds the handler for the
life of the process: 40 guardrail classes plus the pagerduty and email
callbacks. Once the cache entry is evicted (TTL 3600s or the 200-entry
size cap) and the 900s grace passes, the held client is closed and every
subsequent request through it fails with RuntimeError: Cannot send a
request, as the client has been closed. On a production deployment with
a default-on guardrail this surfaced as every request 500ing roughly 75
minutes after boot.
The connection-reclaim goal of #35492 can re-land once handlers survive
their inner client being closed.
An evicted client was left for the garbage collector, but every OpenAI/Azure
SDK client is a reference cycle, so nothing freed the client or its pooled TCP
connections until a generational sweep ran. Driving 2000 azure calls through
the official image with no forced collection, live clients and open sockets
climbed from 202 to 1361 while the cache stayed at its 200-entry bound, and RSS
grew 279 MB to 456 MB against a TLS upstream.
Closing on eviction is what caused the earlier 'Cannot send a request, as the
client has been closed' regression, so an evicted client litellm created is now
closed only once a grace window has passed, by which point any request that was
already holding it has finished. A client the caller supplied is never closed,
since litellm does not own its lifecycle.
Resolves LIT-4883
