490d7dd983
Streaming requests through the Rust bridge previously buffered the full provider response and faked the SSE stream on the Python side, so the first chunk only arrived after the last. Add a native streaming path: - core: SseFrameStream reassembles upstream bytes into complete SSE frames (blank-line delimited, keep-alive frames dropped, trailing partial frame preserved); messages_stream_frames exposes it and forces stream:true upstream so a host cannot misuse the entrypoint. - bridge: amessages_stream returns a MessagesStream async iterator yielding one complete SSE frame as bytes per __anext__ (recv_text pattern: per-call future_into_py, pull-based backpressure). Cancelling __anext__ drops only the frame fetch; dropping the iterator aborts the upstream request; mid-stream failures map to exceptions then terminate iteration. trace=true is rejected (no terminal response). - python: rust_bridge.messages_stream wrapper + aclose-aware adapter; streaming in llm_http_handler now routes through the native stream gate when eligible (same rust gates + agentic-hook fallback) and the fake-stream path is removed. - delete the orphaned duplicate routes/runtime.rs (never declared). |
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call
that makes the LLM call and hands back a typed response, the same shape as
litellm.messages() in Python.
let response = litellm_core::messages::messages(MessagesRequest {
model: "claude-sonnet-4-5",
body,
api_key: Some(key),
..
})
.await?;
Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role |
|---|---|
| litellm-core | The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router. |
| litellm-ai-gateway | The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.
Layout
crates/
core/ The SDK: route modules + provider transforms.
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
src/providers/anthropic/messages/transformation.rs
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
python-bridge/ PyO3 API adapter for Python LiteLLM.
The folder shape follows the Python provider tree:
core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one
function per top-level route, mirroring the core entrypoints.
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace