Split the Rust workflow into fmt, clippy, nextest and wheel jobs so they run in parallel, replace manual actions/cache with Swatinem/rust-cache, and install a pinned checksum-verified cargo-nextest. Make two python-bridge tests self-contained so they pass when nextest runs each test in its own process.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CallbackFamily::delivery is gone. plan_success, plan_failure and
plan_request return a Dispatch carrying family, delivery and gate, and
DispatchCursor takes that Dispatch plus CursorFacts. Delivery was never a
property of the family: legacy runs failure_handler inline from @client but
on the executor from dispatch_failure_handlers, and the streaming port will
need both.
object_target_eligible encodes the is_sync_request gate from
success_handler and failure_handler: on an async SDK request the sync
handler pass runs string integrations only, so CustomLogger and plain
callable targets are not logged twice alongside their async methods.
Parity-neutral for OCR because _is_sync_litellm_request does not know aocr,
but required before the first async route lands.
async_pre_call_deployment_hook, async_post_call_success_deployment_hook and
async_post_call_failure_deployment_hook run on every SDK call, not only on
the proxy, and the native OCR path still delegated them to the Python chain
in litellm.utils. DeploymentBody now iterates litellm.callbacks through
CallbackFamily::Deployment* with Delivery::Await, chaining the replaced
kwargs or response through each CustomLogger and containing failure-hook
errors per target while pre and post hook errors propagate.
Runner::invoke returns an error when an awaited leaf yields a non-awaitable
instead of guessing from is_none. The legacy-disabled test registers a
deployment hook and forbids the three utils entry points.
Finalize calls update_response_metadata directly instead of through a
lifecycle.py passthrough. GC tests cover WorkerJob and DeferredSuccess
traversal through logger, targets and response, and DeferredSuccess close
and release idempotence.
Success, failure and request callbacks on the native OCR path no longer go
through Logging.pre_call, post_call, success_handler or failure_handler.
Core's DispatchCursor selects each target and leaf method; the bridge
invokes it directly or through one labeled leaf in
litellm.rust_bridge.leaves. Delivery follows the family: request and sync
failure run inline, deployment hooks and async failure are awaited by
drive(), sync success runs as one grouped WorkerJob on the executor, async
success is a coroutine enqueued on the logging worker with the deferred gate
held natively in DeferredSuccess.
PrepareLogging (cost, standard payload, redaction) runs inside the dispatch
delivery and never fails the request; Finalize keeps only public response
metadata. Dedup markers stay on the shared logging object and are read as
an eligibility fact. Caller-supplied Logging instances take a separate
compat path that calls their own handlers.
Tests prove the two-pass hook order, per-target hook containment, hook
result replacement, marker write and honour, best-effort preparation, and a
positive run with every legacy orchestration entry point patched to raise.
Native OCR setup no longer calls utils.function_setup. The bridge reads
registration facts (coroutine-ness, CustomLogger, known names, existing list
membership) and core's plan_registration decides every public registry
mutation, which the bridge writes back through litellm.rust_bridge.setup.
The Logging object is built by a narrow Python factory with the same
constructor arguments. Caller-supplied Logging instances keep identity and
skip registration.
A differential test asserts registry side effects equal function_setup for
nine registration shapes; mutating the planner fails four of them.
The request crossed the boundary four times: Python bind_request into a
dataclass, Rust re-reading each field back with its own precedence rules,
a BridgeOcrRequest copy, then the core request. The dataclass was retained
for the whole call only so map_failure could read model and kwargs at the
end.
Add a route-agnostic Signature binder that reproduces Python's positional
and keyword binding rules and TypeError messages, and project straight
from the bound arguments into the core request. This removes the Python
LiteLLMOcrRequest dataclass, bind_request, PythonOcrInput, BridgeOcrRequest,
and the retained boundary_request, document, api_key and callback_inputs
Python objects. api_key for pre_call now comes from core's resolved
connection, matching the legacy handler, and core no longer carries
retained_fields or retains_document since the bridge stopped re-aliasing.
The bridge kept a Rust-side shadow of Logging's callback registries and
elided pre_call/post_call/success_handler when it thought nothing was
listening. That forked the logging contract and hid a bug: with callbacks
present, the during-call hook re-inserted the caller's original document
and unmapped optional params over the provider-transformed body.
Call the real Logging handlers unconditionally, like the Python wrapper
does, and hand callbacks a fresh dict built from core's composed body
instead of re-aliasing caller objects into it.
* feat(rust): count tiktoken cl100k_base admission tokens in Rust
The Rust admission token counter only had the Anthropic tokenizer, so every
other model (OpenAI gpt-4 family, Azure, Gemini, Bedrock non-Claude, Mistral)
tokenized with tiktoken on the Python inference worker.
Add an exact cl100k_base counter to litellm-token-counter: the vendored rank
file (base64 token / rank lines, the bytes Python's tiktoken uses) is parsed
into a byte-level BPE model and the cl100k split pattern is a handwritten
scanner over the shared Unicode classes, so no regex engine runs per request.
Both tokenizers share the message, tool and reply-priming accounting.
The PyO3 TokenCounter gains a from_cl100k_ranks constructor; Python reads the
rank file and passes it in, the way claude_json_str already works. The bridge
selects the counter through the same predicates litellm.token_counter uses
(huggingface_tokenizer_kind, openai_tokenizer_encoding), declines o200k_base,
downloaded HuggingFace and custom tokenizers to Python, and budget reservation
counts once per distinct tokenizer a request names.
The legacy gpt-3.5-turbo-0301 message accounting (4 per message, -1 per name)
stays in Python: the selector declines it through the predicate token_counter
itself uses.
* feat(rust): count tiktoken o200k_base admission tokens in Rust (#40794)
Add a handwritten o200k_base split scanner and TokenCounter::from_o200k_ranks
next to the cl100k_base counter, sharing MergeRanks and the request
accounting. The Python bridge selects it when openai_tokenizer_encoding
names o200k_base, so gpt-4o, gpt-4.1, gpt-5, o1/o3/o4 and chatgpt-4o
requests stop tokenizing on the Python worker under LITELLM_RUST=true
Co-authored-by: yassin <yassin@berri.ai>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes
Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594)
The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of
encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission
count in the gateway pod). A hand-written scanner that yields the same
pieces, then feeds the model directly, counts the same text in 10 ms.
It only engages for tokenizers with the Anthropic shape (optional NFKC,
ByteLevel without prefix space, no post-processor) and falls back to the
full encoder when the text contains an added token. Parity with
encode_fast is tested on random texts, the pieces are compared with the
real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against
oniguruma for every code point.
NFKC runs through unicode-normalization-alignments, the crate and
Unicode tables NormalizedString::nfkc already uses, so the fast path
normalizes exactly what the full encoder would. Using the newer
unicode-normalization crate changed the count for 171 code points that
gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..).
The fast normalizer is compared with the tokenizer's for every scalar
value and on random texts.
The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(python-bridge): split non-streaming bridge modules
* refactor(python-bridge): bring shared function tracing into route layer
* feat(dev): list Python route functions and call sites
* feat(dev): list Rust route functions and call sites
* docs(dev): record OCR parity gaps across Python and Rust
* feat(dev): list executed SDK calls with runtime tracing
* feat(dev): report Python vs Rust SDK pipeline steps in one CLI
* feat(dev): side-by-side pipeline step report in compare CLI
* fix(dev): drop invalid Final annotations in compare cell loop
* feat(dev): blue python-only and yellow rust-only steps in compare CLI
* feat(dev): vertical layout with section spacing in compare CLI
* fix(dev): validate SDK trace stages across sync and async routes
* refactor(rust): align SDK route call structure with Python
* refactor(python-bridge): share sync and async route call wrappers
* refactor(dev): split compare CLI into fixtures, runtime, and report modules
* fix(ci): run SDK trace tests and satisfy test lint
