mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-06 02:48:13 +00:00
17bcc04800
The four shared reqwest clients (core messages/chat_completions/ audio_transcription, ai-gateway) now set pool_max_idle_per_host(64) and tcp_keepalive(60s): - reqwest defaults pool_max_idle_per_host to usize::MAX, so a burst of N concurrent calls to one provider host leaves N idle TLS connections in the pool after completion, each held until its idle timeout; under sustained bursts this grows file descriptors without bound. 64 keeps a warm set large enough for handshake-free reuse. - tcp_keepalive lets silent half-open peers (NAT/LB reaping, vanished upstreams) be detected during long streaming responses instead of blocking until the 600s full-request timeout. The python-bridge cdylib now uses mimalloc as its Rust global allocator (the pydantic-core/orjson pattern). It only governs allocations made by Rust code in the extension - CPython's pymalloc heap is untouched - but Rust-side request/response buffering in a long-lived interpreter process gets mimalloc's sharded free lists and eager page purging instead of glibc malloc, which favors RSS growth under churn. No behavior change beyond the socket/allocator characteristics; timeouts and request paths are untouched. |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| AGENTS.md | ||
| Cargo.toml | ||
| CLAUDE.md | ||