CallbackFamily::delivery is gone. plan_success, plan_failure and
plan_request return a Dispatch carrying family, delivery and gate, and
DispatchCursor takes that Dispatch plus CursorFacts. Delivery was never a
property of the family: legacy runs failure_handler inline from @client but
on the executor from dispatch_failure_handlers, and the streaming port will
need both.
object_target_eligible encodes the is_sync_request gate from
success_handler and failure_handler: on an async SDK request the sync
handler pass runs string integrations only, so CustomLogger and plain
callable targets are not logged twice alongside their async methods.
Parity-neutral for OCR because _is_sync_litellm_request does not know aocr,
but required before the first async route lands.
Native OCR setup no longer calls utils.function_setup. The bridge reads
registration facts (coroutine-ness, CustomLogger, known names, existing list
membership) and core's plan_registration decides every public registry
mutation, which the bridge writes back through litellm.rust_bridge.setup.
The Logging object is built by a narrow Python factory with the same
constructor arguments. Caller-supplied Logging instances keep identity and
skip registration.
A differential test asserts registry side effects equal function_setup for
nine registration shapes; mutating the planner fails four of them.
Adds CallbackId, Delivery, CallbackInvocation, CallbackFamily, LoggedMarker,
DispatchCursor, plan_success and plan_failure to call_lifecycle. Core now
owns target ordering, get_combined_callback_list dedup, the two-pass success
shape, marker placement, per-family delivery and error policy, and the
sync/async success dispatch plan. No host wiring yet.
The request crossed the boundary four times: Python bind_request into a
dataclass, Rust re-reading each field back with its own precedence rules,
a BridgeOcrRequest copy, then the core request. The dataclass was retained
for the whole call only so map_failure could read model and kwargs at the
end.
Add a route-agnostic Signature binder that reproduces Python's positional
and keyword binding rules and TypeError messages, and project straight
from the bound arguments into the core request. This removes the Python
LiteLLMOcrRequest dataclass, bind_request, PythonOcrInput, BridgeOcrRequest,
and the retained boundary_request, document, api_key and callback_inputs
Python objects. api_key for pre_call now comes from core's resolved
connection, matching the legacy handler, and core no longer carries
retained_fields or retains_document since the bridge stopped re-aliasing.
* feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes
Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594)
The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of
encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission
count in the gateway pod). A hand-written scanner that yields the same
pieces, then feeds the model directly, counts the same text in 10 ms.
It only engages for tokenizers with the Anthropic shape (optional NFKC,
ByteLevel without prefix space, no post-processor) and falls back to the
full encoder when the text contains an added token. Parity with
encode_fast is tested on random texts, the pieces are compared with the
real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against
oniguruma for every code point.
NFKC runs through unicode-normalization-alignments, the crate and
Unicode tables NormalizedString::nfkc already uses, so the fast path
normalizes exactly what the full encoder would. Using the newer
unicode-normalization crate changed the count for 171 code points that
gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..).
The fast normalizer is compared with the tokenizer's for every scalar
value and on random texts.
The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
