mirror of https://github.com/BerriAI/litellm.git synced 2026-09-15 23:31:29 +00:00

History

devin-ai-integration[bot] 46a185d3cd feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes (#40381 ) * feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594) The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission count in the gateway pod). A hand-written scanner that yields the same pieces, then feeds the model directly, counts the same text in 10 ms. It only engages for tokenizers with the Anthropic shape (optional NFKC, ByteLevel without prefix space, no post-processor) and falls back to the full encoder when the text contains an added token. Parity with encode_fast is tested on random texts, the pieces are compared with the real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against oniguruma for every code point. NFKC runs through unicode-normalization-alignments, the crate and Unicode tables NormalizedString::nfkc already uses, so the fast path normalizes exactly what the full encoder would. Using the newer unicode-normalization crate changed the count for 171 code points that gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..). The fast normalizer is compared with the tokenizer's for every scalar value and on random texts. The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value. Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>		2026-09-10 13:56:30 -07:00
..
crates	feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes (#40381 )	2026-09-10 13:56:30 -07:00
.gitignore	feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033 )	2026-06-23 13:16:47 -07:00
ADDING_A_PROVIDER.md	docs(litellm-rust): fix the gateway run commands and point ADDING_A_PROVIDER at the one checks runbook	2026-09-03 00:07:19 -07:00
AGENTS.md	feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes (#40381 )	2026-09-10 13:56:30 -07:00
Cargo.lock	feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes (#40381 )	2026-09-10 13:56:30 -07:00
Cargo.toml	feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes (#40381 )	2026-09-10 13:56:30 -07:00
CLAUDE.md	refactor(rust): extract config crate (#39706 )	2026-09-04 08:17:07 -07:00
README.md	refactor(rust): extract config crate (#39706 )	2026-09-04 08:17:07 -07:00

README.md

LiteLLM Rust

This workspace contains the staged Rust implementation for LiteLLM.

litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call that makes the LLM call and hands back a typed response, the same shape as litellm.messages() in Python.

let response = litellm_core::messages::messages(MessagesRequest {
    model: "claude-sonnet-4-5",
    body,
    api_key: Some(key),
    ..
})
.await?;

Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.

Crates

Crate	Role
litellm-core	The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router.
litellm-config	Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python.
litellm-ai-gateway	The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers.
litellm-python-interop	Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion.
litellm-python-bridge	PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping.

Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop.

Layout

crates/
  core/           The SDK: route modules + provider transforms.
    src/messages/   mod.rs (entrypoint), types, transformation, prepare, handler, client
    src/providers/anthropic/messages/transformation.rs
  config/         Config loading and resolved deployments.
  ai-gateway/     Axum server + WebSocket hosts; calls core entrypoints.
  python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
  python-bridge/  PyO3 API adapter for Python LiteLLM.

The folder shape follows the Python provider tree: core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one function per top-level route, mirroring the core entrypoints.

Checks

Run the commands under "Checks" in CLAUDE.md before pushing Rust changes. That list is the single source of truth and matches what GitHub Actions runs for changes under litellm-rust/.