litellm/litellm-rust

LiteLLM Rust

This workspace contains the staged Rust implementation for LiteLLM.

litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call that makes the LLM call and hands back a typed response, the same shape as litellm.messages() in Python.

let response = litellm_core::messages::messages(MessagesRequest {
    model: "claude-sonnet-4-5",
    body,
    api_key: Some(key),
    ..
})
.await?;

Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.

Crates

Crate	Role
litellm-core	The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router.
litellm-ai-gateway	The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers.
litellm-python-bridge	PyO3 cdylib exposing Rust to the litellm Python SDK — marshals Python objects and calls core entrypoints.

Dependency direction (acyclic): litellm-core ← litellm-ai-gateway ← litellm-python-bridge.

Layout

crates/
  core/           The SDK: route modules + provider transforms.
    src/messages/   mod.rs (entrypoint), types, transformation, prepare, handler, client
    src/providers/anthropic/messages/transformation.rs
  ai-gateway/     Axum server + WebSocket hosts; calls core entrypoints.
  python-bridge/  PyO3 bridge for Python LiteLLM.

The folder shape follows the Python provider tree: core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one function per top-level route, mirroring the core entrypoints.

Checks

Run these before pushing Rust changes. GitHub Actions runs the same checks for changes under litellm-rust/.

cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace