466b44e318
Adds crates/python-bridge/src/architecture.rs, a source-scan test in the spirit of workspace_crate_allowlist.rs: layering rules that currently live only in AGENTS.md become executable. Rules on production code (text before each file's trailing #[cfg(test)] module): - GIL attach/detach and block_on appear in python-bridge only in execution.rs; scattered interpreter calls are how GIL-ordering deadlocks and per-handoff contention creep in. - Tokio runtime construction appears only in execution.rs and the #[pymodule] init site in lib.rs; one shared runtime per process. - SendWrapper is banned in both PyO3 crates; it converts !Send Python values into cross-thread panics on Tokio workers. - python-interop stays domain-neutral and never blocks on futures or builds runtimes. Deletes the dead routes/runtime.rs: an undeclared byte-for-byte duplicate of execution.rs whose Python::attach/block_on usage would violate the new boundary (also deleted independently in #39577; both sides delete the same file, so the merge is trivial). AGENTS.md gains the enforcement note, mirroring the crate-allowlist convention. |
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call
that makes the LLM call and hands back a typed response, the same shape as
litellm.messages() in Python.
let response = litellm_core::messages::messages(MessagesRequest {
model: "claude-sonnet-4-5",
body,
api_key: Some(key),
..
})
.await?;
Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role |
|---|---|
| litellm-core | The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router. |
| litellm-ai-gateway | The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.
Layout
crates/
core/ The SDK: route modules + provider transforms.
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
src/providers/anthropic/messages/transformation.rs
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
python-bridge/ PyO3 API adapter for Python LiteLLM.
The folder shape follows the Python provider tree:
core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one
function per top-level route, mirroring the core entrypoints.
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace