scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-16 23:41:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 227
litellm/litellm-rust
History
yuneng-jiang 9140911892
build(rust): raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 (#33798) 
pyo3 0.23.5 hard-caps the interpreter at Python 3.13, so building the
native bridge against a 3.14 interpreter aborts inside pyo3-ffi's build
script before anything links. This raises pyo3 and pyo3-async-runtimes
to 0.29 (currently the newest line, and the range starting at 0.26 that
supports 3.14) and migrates the three call sites whose APIs were renamed
across that range: Python::with_gil is now Python::attach and
Python::allow_threads is now Python::detach. On a GIL-enabled interpreter
those are pure renames with identical semantics, so behavior on 3.10
through 3.13 is unchanged

Verified by compiling the native module for cp313 and cp314 and driving
it directly on both interpreters: gil_stats reports exactly one GIL
release per sync OCR call and the async path completes, matching the
0.23.5 baseline. cargo fmt, clippy, and the workspace tests pass on both
3.13 and 3.14 with the lockfile locked, and the lock churn is confined to
the pyo3 crates

Part of #26343; addresses the pyo3 build failure reported in #33116

(cherry picked from commit f3d20153b3)
2026-07-18 13:33:15 -07:00
..
crates build(rust): raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 (#33798) 2026-07-18 13:33:15 -07:00
.gitignore feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033) 2026-06-23 13:16:47 -07:00
ADDING_A_PROVIDER.md feat: add openai realtime translation layer to litellm-rust (1/2) (#31129) 2026-06-23 15:48:51 -07:00
AGENTS.md refactor(litellm-rust): dissolve providers into core + ai-gateway (strict 3-crate layers) (#31218) 2026-06-24 12:22:43 -07:00
Cargo.lock build(rust): raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 (#33798) 2026-07-18 13:33:15 -07:00
Cargo.toml build(rust): raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 (#33798) 2026-07-18 13:33:15 -07:00
CLAUDE.md feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134) 2026-06-24 15:25:10 -07:00
README.md refactor(litellm-rust): dissolve providers into core + ai-gateway (strict 3-crate layers) (#31218) 2026-06-24 12:22:43 -07:00

README.md

LiteLLM Rust

This workspace contains the staged Rust implementation for LiteLLM.

Rust starts as a pure transform core used by the existing Python host. Python continues to own auth, configuration, network I/O, retries, routing, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.

Crates

Crate Role Pure / I/O
litellm-core Translation layer — types, route contracts (traits), provider transforms (modules under providers/), and the router. Builds requests/responses; no network. Pure
litellm-ai-gateway Routes + host — the only crate that touches the network. HTTP/WebSocket I/O (modules under io/) plus the axum server binary (behind the server feature). I/O
litellm-python-bridge PyO3 cdylib exposing Rust to the litellm Python SDK — a thin adapter over litellm-ai-gateway's I/O. Binding

Dependency direction (acyclic): litellm-core ← litellm-ai-gateway ← litellm-python-bridge.

Layout

crates/
  core/           Route contracts, shared pure types, errors, and templates.
    src/ocr/
  providers/      Provider-specific pure transforms.
    src/mistral/ocr/transformation.rs
  python-bridge/  PyO3 bridge for Python LiteLLM.

The folder shape should follow the Python provider tree: providers/src/<provider>/<route>/transformation.rs. The bridge should expose one function per top-level route, starting with ocr(payload).

Checks

Run these before pushing Rust changes. GitHub Actions runs the same checks for changes under litellm-rust/.

cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace