mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
c4f19c3e4c
* feat(messages): route Azure Anthropic /messages through Rust behind rust:true
Adds an opt-in Rust path for non-streaming Azure Anthropic Messages. A
deployment sets rust: true in litellm_params to route litellm.messages()
and the proxy /v1/messages endpoint through the native Rust bridge; a
missing flag or rust: false keeps the existing Python path, and non-Azure
providers, streaming, an unavailable bridge, or a None result all fall
back to Python. Rust-backed responses carry an x-litellm-rust: true
response header so callers can see which path served the request.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(docs): exclude LITELLM_USE_RUST_MESSAGES rollout flag from env-doc check
Mirrors the existing LITELLM_USE_RUST_OCR entry; the flag is an internal
rollout toggle that is intentionally not in the public environment settings
docs yet.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(rust_bridge): isolate OCR enable flag and drop dead messages global toggle
use_litellm_rust only mutates the OCR enabled flag when configuring OCR (or
called with no bridge kwargs, preserving the legacy contract), so configuring
only the messages bridge no longer flips OCR state.
Remove the vestigial global enabled/env state from the messages bridge. Routing
is controlled per deployment by rust:true in the shared handler gate, so the
messages module never consulted the global toggle; drop it rather than leave a
no-op switch.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(rust/messages): split Anthropic config into its own provider file and type the request/response contract
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(messages): route eligible Azure Anthropic streaming through Rust via buffered fake-stream
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(messages): fold system-role messages for Azure Anthropic and fall back to Python on Rust bridge errors
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(rust_bridge): use Python::attach for amessages after pyo3 bump
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): mock get_configured_token_limits in model_info tests
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* ci: run rust_bridge unit tests in misc shard
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* Revert "ci: run rust_bridge unit tests in misc shard"
This reverts commit
|
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
Rust starts as a pure transform core used by the existing Python host. Python continues to own auth, configuration, network I/O, retries, routing, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role | Pure / I/O |
|---|---|---|
| litellm-core | Translation layer — types, route contracts (traits), provider transforms (modules under providers/), and the router. Builds requests/responses; no network. | Pure |
| litellm-ai-gateway | Routes + host — the only crate that touches the network. HTTP/WebSocket I/O (modules under io/) plus the axum server binary (behind the server feature). |
I/O |
| litellm-python-bridge | PyO3 cdylib exposing Rust to the litellm Python SDK — a thin adapter over litellm-ai-gateway's I/O. | Binding |
Dependency direction (acyclic): litellm-core ← litellm-ai-gateway ← litellm-python-bridge.
Layout
crates/
core/ Route contracts, shared pure types, errors, and templates.
src/ocr/
providers/ Provider-specific pure transforms.
src/mistral/ocr/transformation.rs
python-bridge/ PyO3 bridge for Python LiteLLM.
The folder shape should follow the Python provider tree:
providers/src/<provider>/<route>/transformation.rs. The bridge should expose
one function per top-level route, starting with ocr(payload).
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace