mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
46a185d3cd
* feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594) The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission count in the gateway pod). A hand-written scanner that yields the same pieces, then feeds the model directly, counts the same text in 10 ms. It only engages for tokenizers with the Anthropic shape (optional NFKC, ByteLevel without prefix space, no post-processor) and falls back to the full encoder when the text contains an added token. Parity with encode_fast is tested on random texts, the pieces are compared with the real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against oniguruma for every code point. NFKC runs through unicode-normalization-alignments, the crate and Unicode tables NormalizedString::nfkc already uses, so the fast path normalizes exactly what the full encoder would. Using the newer unicode-normalization crate changed the count for 171 code points that gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..). The fast normalizer is compared with the tokenizer's for every scalar value and on random texts. The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value. Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
45 lines
3.1 KiB
Markdown
45 lines
3.1 KiB
Markdown
# AGENTS.md
|
|
|
|
litellm-rust has six crates. A crate is a layer or shared foundation, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers.
|
|
|
|
## Crates
|
|
|
|
| Crate | Role |
|
|
|-------|------|
|
|
| litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. |
|
|
| litellm-token-counter | Standalone input token counting shared by host integrations without pulling in the full SDK. |
|
|
| litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. |
|
|
| litellm-ai-gateway | The axum server (behind the `server` feature) plus the WebSocket hosts. Translates HTTP/WS to core entrypoints; owns no provider logic and no handlers. |
|
|
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
|
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
|
|
|
|
Dependency direction is acyclic: `litellm-config` depends on `litellm-core`, the gateway depends on both, and `litellm-python-bridge` depends on the domain layers, `litellm-token-counter`, and `litellm-python-interop`. The token counter and interop foundations depend on no LiteLLM domain crate.
|
|
|
|
## Where a route lives
|
|
|
|
A top-level LiteLLM call is a module under `crates/core/src/<route>/`, shaped like `messages`:
|
|
|
|
```
|
|
core/src/messages/
|
|
mod.rs # pub async fn messages(..) -> CoreResult<..> (+ messages_stream for SSE)
|
|
types.rs # request/response types, MessagesRequest
|
|
transformation.rs # the provider template trait
|
|
prepare.rs # provider resolution, auth headers, URL
|
|
handler.rs # the provider call
|
|
client.rs # the shared reqwest client
|
|
```
|
|
|
|
Handlers never live in `ai-gateway`. `ocr`, `audio_transcription`, and `realtime` are still hosted there from before this rule; they move to `core` as they are touched.
|
|
|
|
Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these.
|
|
|
|
Adding a crate fails crates/core/tests/workspace_crate_allowlist.rs until you update its allowlist and this file — intentional.
|
|
|
|
## Style
|
|
|
|
All Rust in `litellm-rust/` follows the official Rust Style Guide:
|
|
https://doc.rust-lang.org/style-guide/
|
|
|
|
`rustfmt` implements its formatting by default, so run `cargo fmt` before committing; CI gates every PR on `cargo fmt --check`. Do not hand-format against rustfmt or add a `rustfmt.toml` that diverges from the default style.
|
|
|
|
Beyond formatting, follow the guide's naming and idiom conventions rustfmt cannot auto-apply: `snake_case` items/functions/modules, `UpperCamelCase` types/traits/variants, `SCREAMING_SNAKE_CASE` constants/statics (acronyms as one word, e.g. `HttpClient`), and the import grouping and item ordering it prescribes. See CLAUDE.md for the detailed version.
|