mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
0da3f6b72a
Without a limit, every host call becomes an in-flight upstream request with an unbounded response buffer: a burst of N concurrent calls means N open provider sockets, N buffered bodies in memory, and a provider-side 429 storm. reqwest pools cap only idle sockets, the router is a pure selector, and the axum entrypoint has no tower limits, so nothing bounds this today. - core: new concurrency module. Hosts resolve the config-shaped env at startup and install a process-wide semaphore via init_limits (uninitialized = unlimited, so rollouts keep today's behavior). acquire() is an OwnedSemaphorePermit held across the whole call, including response buffering (that buffering is the memory being capped); queue mode is FIFO and cancel-safe (dropping the future releases nothing it did not hold), shed mode fails fast with a new Error::Overloaded before any provider call. - core entrypoints messages, chat_completions, audio_transcription hold the permit for the call; the gateway OCR handler acquires it too (interim until the handler moves into core). Streaming entrypoints are not capped yet: their in-flight window outlives the call that started them, so the permit belongs on the returned stream (follow-up once the frame-stream route lands). - errors: Error::Overloaded maps to RustBridgeDeclined in both bridge mappers (nothing reached the provider; the Python host may fall back to its own path) and to 429 in the gateway. - hosts: the bridge module init and the gateway main read LITELLM_RUST_MAX_IN_FLIGHT and LITELLM_RUST_SHED_ON_LIMIT (invalid values warn and are ignored); the bridge also applies LITELLM_RUST_WORKER_THREADS to the shared runtime, which must happen at module init because pyo3-async-runtimes builds the runtime lazily and cannot resize it afterwards. - diagnostics: new native_stats() beside gil_stats() reporting max_in_flight, in_flight, and shed_on_limit. |
||
|---|---|---|
| .. | ||
| ai-gateway | ||
| CODING_STANDARDS | ||
| core | ||
| python-bridge | ||
| python-interop | ||