2c08e7abf8
The two tests that shipped with the fix both called ensure_crypto_provider themselves, so deleting the call from connect_upstream left the whole suite green, and swapping ring for aws-lc-rs did too. Adds an integration test, which gets its own process, that dials wss:// at a local plain-TCP listener through the public Responses WebSocket entrypoint and asserts an Err plus an installed provider. Without the install in the dial it panics with the original CryptoProvider message. A unit test now compares the installed provider's cipher suites and key-exchange groups against ring's, so the choice of backend is pinned rather than assumed. Also names tls12 in the workspace rustls features: it already arrives through reqwest and tokio-rustls, so the graph is unchanged, but a direct dependency should say it needs TLS 1.2 rather than inherit it. |
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call
that makes the LLM call and hands back a typed response, the same shape as
litellm.messages() in Python.
let response = litellm_core::messages::messages(MessagesRequest {
model: "claude-sonnet-4-5",
body,
api_key: Some(key),
..
})
.await?;
Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role |
|---|---|
| litellm-core | The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router. |
| litellm-ai-gateway | The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.
Layout
crates/
core/ The SDK: route modules + provider transforms.
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
src/providers/anthropic/messages/transformation.rs
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
python-bridge/ PyO3 API adapter for Python LiteLLM.
The folder shape follows the Python provider tree:
core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one
function per top-level route, mirroring the core entrypoints.
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace