mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-21 00:21:49 +00:00
3f00e6fa46
`AsyncHTTPHandler` and `HTTPHandler` each closed their httpx client from `__del__`. Closing an httpx client tears down the connection pool every in-flight response is streaming through, and it permanently invalidates the client for later requests. Neither is safe on garbage collection, because a handler is routinely collected while its client is still in use: - Streaming. Nothing in a response's reference graph points back at the handler: httpx binds the response to the transport stream and the pool, never to the wrapper that built them. A cached handler therefore becomes collectable mid-stream when its one-hour TTL expires, and every body still being read off that pool is cut. That is #24929; the window recurs hourly per process, because the TTL clock starts at first client construction. - Borrowed clients. `litellm/a2a_protocol/main.py` and `litellm/proxy/pass_through_endpoints/pass_through_endpoints.py` keep only `handler.client`. Both take the handler from `get_async_httpx_client`, so the cache pins it for the same one-hour TTL and then lets it go on eviction, while the borrowed client is still serving a longer-lived consumer: `create_a2a_client` hands its client to the a2a SDK and documents it as "create client once, reuse for multiple requests". The next request on that client raises `RuntimeError: Cannot send a request, as the client has been closed.` - Someone else's client. `litellm/llms/azure/azure.py` wraps `litellm.client_session` in a throwaway `HTTPHandler` for a single image generation, and `HTTPHandler.close()` closes whatever client it was handed. One such call left the user's shared session closed for the rest of the process. `LLMClientCache` already documents the invariant these finalizers broke: evicted clients "may still be in use by in-flight requests", so the cache deliberately does not close them and leaves them to normal garbage collection. Closing on collection is precisely what that rules out. Deleting both finalizers removes the cause instead of narrowing the window. The async one was never dependable regardless: it needs a running event loop, so collection outside one silently did nothing. Explicit `close()`/`aclose()` is untouched, and `close_litellm_async_clients()` still closes cached async clients at exit. The cost is that an evicted client is no longer closed eagerly. On the default aiohttp transport that shows up as five lines of asyncio ERROR output per evicted handler ("Unclosed client session", "Unclosed connector", and their context keys) with no change in descriptors or RSS, since aiohttp's own connector finalizer closes the connections as it warns. On the httpcore transport there is no log output, but idle keep-alive sockets from evicted pools are held until a generation-2 collection reclaims them: over 5000 evictions that plateaus at 56 descriptors and ~1 MB of RSS, and with 20 concurrent requests per pool it peaks around 650 descriptors before each gen-2 pass returns it to baseline. Bounded by the GC cycle, not by the eviction count. Fixes #24929 |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.