1562339aab
Inherits a [workspace.lints] table into all four crates: - clippy::await_holding_lock (deny): a std::sync Mutex/RwLock guard held across .await blocks its Tokio worker and is one step from deadlock. Async-aware locking belongs to tokio::sync. - clippy::await_holding_refcell_ref (deny): same failure mode for RefCell guards resumed on an arbitrary worker thread. - clippy::undocumented_unsafe_blocks (warn): every future unsafe block must carry a SAFETY justification; the workspace currently has zero unsafe blocks, so this only gates what comes next (free-threading work, buffer protocol). - rustc::unsafe_op_in_unsafe_fn (warn): unsafe operations inside unsafe fns need their own explicit unsafe block. |
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call
that makes the LLM call and hands back a typed response, the same shape as
litellm.messages() in Python.
let response = litellm_core::messages::messages(MessagesRequest {
model: "claude-sonnet-4-5",
body,
api_key: Some(key),
..
})
.await?;
Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role |
|---|---|
| litellm-core | The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router. |
| litellm-ai-gateway | The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.
Layout
crates/
core/ The SDK: route modules + provider transforms.
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
src/providers/anthropic/messages/transformation.rs
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
python-bridge/ PyO3 API adapter for Python LiteLLM.
The folder shape follows the Python provider tree:
core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one
function per top-level route, mirroring the core entrypoints.
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace