0da3f6b72a
Without a limit, every host call becomes an in-flight upstream request with an unbounded response buffer: a burst of N concurrent calls means N open provider sockets, N buffered bodies in memory, and a provider-side 429 storm. reqwest pools cap only idle sockets, the router is a pure selector, and the axum entrypoint has no tower limits, so nothing bounds this today. - core: new concurrency module. Hosts resolve the config-shaped env at startup and install a process-wide semaphore via init_limits (uninitialized = unlimited, so rollouts keep today's behavior). acquire() is an OwnedSemaphorePermit held across the whole call, including response buffering (that buffering is the memory being capped); queue mode is FIFO and cancel-safe (dropping the future releases nothing it did not hold), shed mode fails fast with a new Error::Overloaded before any provider call. - core entrypoints messages, chat_completions, audio_transcription hold the permit for the call; the gateway OCR handler acquires it too (interim until the handler moves into core). Streaming entrypoints are not capped yet: their in-flight window outlives the call that started them, so the permit belongs on the returned stream (follow-up once the frame-stream route lands). - errors: Error::Overloaded maps to RustBridgeDeclined in both bridge mappers (nothing reached the provider; the Python host may fall back to its own path) and to 429 in the gateway. - hosts: the bridge module init and the gateway main read LITELLM_RUST_MAX_IN_FLIGHT and LITELLM_RUST_SHED_ON_LIMIT (invalid values warn and are ignored); the bridge also applies LITELLM_RUST_WORKER_THREADS to the shared runtime, which must happen at module init because pyo3-async-runtimes builds the runtime lazily and cannot resize it afterwards. - diagnostics: new native_stats() beside gil_stats() reporting max_in_flight, in_flight, and shed_on_limit. |
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call
that makes the LLM call and hands back a typed response, the same shape as
litellm.messages() in Python.
let response = litellm_core::messages::messages(MessagesRequest {
model: "claude-sonnet-4-5",
body,
api_key: Some(key),
..
})
.await?;
Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role |
|---|---|
| litellm-core | The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router. |
| litellm-ai-gateway | The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.
Layout
crates/
core/ The SDK: route modules + provider transforms.
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
src/providers/anthropic/messages/transformation.rs
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
python-bridge/ PyO3 API adapter for Python LiteLLM.
The folder shape follows the Python provider tree:
core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one
function per top-level route, mirroring the core entrypoints.
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace