mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
dc585235be
Some checks are pending
LiteLLM Rust / rustfmt, clippy, test (push) Waiting to run
* feat(vertex): add shared Rust Google authentication Mint and refresh Vertex OAuth access tokens in the ai-gateway host layer via the official google-cloud-auth crate, preserving the inline service-account JSON, ADC and GOOGLE_APPLICATION_CREDENTIALS contract with library-managed caching/refresh and no hand-rolled signing. Core stays auth/IO-free: it only classifies the bearer source and rejects Google AIza API keys rather than sending them as OAuth. Also fix Vertex Mistral rawPredict endpoint construction so the global location targets aiplatform.googleapis.com. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(vertex): address review; SHA-256 cache key and env credentials Move newly introduced Vertex constants into crate-level constants.rs, drop the doc/inline comments added by the auth PR, and replace the DefaultHasher u64 credential-cache key with a collision-resistant SHA-256 digest (sha2 is now a non-optional ai-gateway dependency so the non-server host path can use it). When no explicit vertex_credentials optional param is supplied, the host now reads VERTEXAI_CREDENTIALS from the environment before falling back to standard ADC/GOOGLE_APPLICATION_CREDENTIALS, without exposing credential content. Adds tests covering env-based inline credential selection and that distinct credential sources cannot collide on one cache entry. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * build(vertex): pin google-cloud-auth to =1.13.0 for dependency-age policy Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(vertex): bound credential cache, reject AIza in auth header, data-minimize auth errors Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * build(vertex): pin google-cloud-auth to =1.9.0 and adopt MSRV-aware resolver for Rust 1.86 Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(vertex): require well-formed Bearer scheme on caller Authorization header Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(vertex): require exactly one well-formed Bearer authorization header Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(vertex): key credential cache by content and pin rustls exactly Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * chore(bridge): remove accidentally committed native extension binary Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(vertex): split base auth and shared cache Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| .cargo | ||
| crates | ||
| .gitignore | ||
| ADDING_A_PROVIDER.md | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| README.md | ||
LiteLLM Rust
This workspace contains the staged Rust implementation for LiteLLM.
Rust starts as a pure transform core used by the existing Python host. Python continues to own auth, configuration, network I/O, retries, routing, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.
Crates
| Crate | Role | Pure / I/O |
|---|---|---|
| litellm-core | Translation layer — types, route contracts (traits), provider transforms (modules under providers/), and the router. Builds requests/responses; no network. | Pure |
| litellm-ai-gateway | Routes + host — the only crate that touches the network. HTTP/WebSocket I/O (modules under io/) plus the axum server binary (behind the server feature). |
I/O |
| litellm-python-bridge | PyO3 cdylib exposing Rust to the litellm Python SDK — a thin adapter over litellm-ai-gateway's I/O. | Binding |
Dependency direction (acyclic): litellm-core ← litellm-ai-gateway ← litellm-python-bridge.
Layout
crates/
core/ Route contracts, shared pure types, errors, and templates.
src/ocr/
providers/ Provider-specific pure transforms.
src/mistral/ocr/transformation.rs
python-bridge/ PyO3 bridge for Python LiteLLM.
The folder shape should follow the Python provider tree:
providers/src/<provider>/<route>/transformation.rs. The bridge should expose
one function per top-level route, starting with ocr(payload).
Checks
Run these before pushing Rust changes. GitHub Actions runs the same checks for
changes under litellm-rust/.
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace