mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
76 lines
3.3 KiB
Markdown
76 lines
3.3 KiB
Markdown
# LiteLLM Rust
|
|
|
|
This workspace contains the staged Rust implementation for LiteLLM.
|
|
|
|
`litellm-core` is the LiteLLM SDK in Rust: one entrypoint per top-level call
|
|
that makes the LLM call and hands back a typed response, the same shape as
|
|
`litellm.messages()` in Python.
|
|
|
|
```rust
|
|
let response = litellm_core::messages::messages(
|
|
MessagesRequest {
|
|
model: "claude-sonnet-4-5",
|
|
body,
|
|
options: RequestOptions {
|
|
api_key: Some(key.to_string()),
|
|
..Default::default()
|
|
},
|
|
},
|
|
&LiteLlmRequestContext::default(),
|
|
)
|
|
.await?;
|
|
```
|
|
|
|
Python continues to own configuration, retries, routing policy, logging,
|
|
callbacks, spend tracking, and customer plugins until each Rust path has parity
|
|
coverage and production evidence.
|
|
|
|
## Native request boundary
|
|
|
|
Native HTTP routes and Responses WebSocket connections accept
|
|
`native(request, *, options, context)`. The request carries only endpoint payload.
|
|
`NativeRequestOptions` carries credentials, typed provider configuration, routing,
|
|
headers, query parameters, and timeout. `NativeRequestContext` carries call identity,
|
|
attribution, and typed capability facts separately from the provider payload.
|
|
|
|
Python builds the frozen request dataclasses in `litellm/rust_bridge/request.py` and
|
|
PyO3 extracts their fields before execution. AWS credentials and metadata policy
|
|
belong in `options.bedrock`; Vertex project/location belongs in `options.vertex`.
|
|
|
|
This boundary preserves existing Python provider preparation, preflight decisions,
|
|
fallback, and callbacks
|
|
|
|
## Crates
|
|
|
|
| Crate | Role |
|
|
|-------|------|
|
|
| litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. |
|
|
| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
|
|
| litellm-ai-gateway | The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
|
|
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
|
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
|
|
|
|
Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop.
|
|
|
|
## Layout
|
|
|
|
```text
|
|
crates/
|
|
core/ The SDK: route modules + provider transforms.
|
|
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
|
|
src/providers/anthropic/messages/transformation.rs
|
|
config/ Config loading and resolved deployments.
|
|
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
|
|
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
|
|
python-bridge/ PyO3 API adapter for Python LiteLLM.
|
|
```
|
|
|
|
The folder shape follows the Python provider tree:
|
|
`core/src/providers/<provider>/<route>/transformation.rs`. The bridge exposes one
|
|
function per top-level route, mirroring the core entrypoints.
|
|
|
|
## Checks
|
|
|
|
Run the commands under "Checks" in [CLAUDE.md](CLAUDE.md) before pushing Rust
|
|
changes. That list is the single source of truth and matches what GitHub Actions
|
|
runs for changes under `litellm-rust/`.
|