mirror of https://github.com/BerriAI/litellm.git synced 2026-09-17 23:51:30 +00:00

History

mateo-berri b55833b1f9 fix(ai-gateway): build the release image again and cover it in CI The image had two independent breaks. The Dockerfile pinned rust 1.90 while the repo pins 1.98 in rust-toolchain.toml and never copied it in, so the first cargo call died on crates needing a newer rustc. The runtime stage then ran pip install on the root pyproject, which builds with maturin against the python-bridge crate, so metadata generation failed with no Cargo manifest and no Rust toolchain in that stage. Copy rust-toolchain.toml into the builder so every cargo call uses the pinned channel, build the wheel in the builder stage where cargo and python3-dev already live, and have the runtime stage install that artifact instead of compiling anything. Add the ai-gateway image job to the rust workflow so a broken build fails a PR instead of surfacing on a release.		2026-09-03 01:40:00 -07:00
..
crates	fix(ai-gateway): build the release image again and cover it in CI	2026-09-03 01:40:00 -07:00
.gitignore	feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033 )	2026-06-23 13:16:47 -07:00
ADDING_A_PROVIDER.md	docs(litellm-rust): fix the gateway run commands and point ADDING_A_PROVIDER at the one checks runbook	2026-09-03 00:07:19 -07:00
AGENTS.md	refactor(rust): extract domain-neutral Python interop	2026-09-02 12:16:26 -07:00
Cargo.lock	fix(python-bridge): harden sync and async route boundaries (#39332 )	2026-09-02 16:26:35 -07:00
Cargo.toml	fix(python-bridge): harden sync and async route boundaries (#39332 )	2026-09-02 16:26:35 -07:00
CLAUDE.md	ci(rust): lint every gateway feature and keep one checks runbook	2026-09-02 22:16:40 -07:00
README.md	ci(rust): lint every gateway feature and keep one checks runbook	2026-09-02 22:16:40 -07:00

README.md

LiteLLM Rust

This workspace contains the staged Rust implementation for LiteLLM.

litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call that makes the LLM call and hands back a typed response, the same shape as litellm.messages() in Python.

let response = litellm_core::messages::messages(MessagesRequest {
    model: "claude-sonnet-4-5",
    body,
    api_key: Some(key),
    ..
})
.await?;

Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.

Crates

Crate	Role
litellm-core	The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router.
litellm-ai-gateway	The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers.
litellm-python-interop	Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion.
litellm-python-bridge	PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping.

Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.

Layout

crates/
  core/           The SDK: route modules + provider transforms.
    src/messages/   mod.rs (entrypoint), types, transformation, prepare, handler, client
    src/providers/anthropic/messages/transformation.rs
  ai-gateway/     Axum server + WebSocket hosts; calls core entrypoints.
  python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
  python-bridge/  PyO3 API adapter for Python LiteLLM.

The folder shape follows the Python provider tree: core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one function per top-level route, mirroring the core entrypoints.

Checks

Run the commands under "Checks" in CLAUDE.md before pushing Rust changes. That list is the single source of truth and matches what GitHub Actions runs for changes under litellm-rust/.