litellm.user_url_validation and litellm.user_url_allowed_hosts are only implemented by the Python document fetcher, so an allowlisted internal document was rejected by the Rust route's network policy. The bridge now declines when either is changed from its default
Python's aiohttp transport reads HTTP(S)_PROXY on every request unless disable_aiohttp_trust_env is set, so the Rust clients now do the same instead of requiring aiohttp_trust_env. Transport error messages include reqwest's source chain, so a rejected certificate or refused connection is no longer reported as just 'error sending request'
Honor environment proxies whenever Python would use httpx (sync calls, HTTP/2, aiohttp disabled), apply the per-call ssl_verify argument, ignore empty or missing SSL env values the way http_handler.py does, expire pooled clients after an hour so rotated certificates reload, keep the client certificate off media downloads, and decline instead of raising when a litellm global has an unexpected type
Drop the core ocr() facade so VertexAuth and the http pool stay out of litellm-core's
public API, move the http Error enum to error.rs, and inject the media DNS resolver into
HttpClientPool instead of a per-call builder hook the cache key ignored.
The bridge now reads litellm.* HTTP settings only through litellm/rust_bridge/settings.py,
pinned by python_settings.json, while env overrides stay in Rust. This adds the Python
default User-Agent, parses string ssl_verify globals like get_ssl_verify, drops per-call
ssl_verify that Python OCR never honored, and removes the unused request_timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every legacy callback call from callbacks-legacy now goes through one typed
Python shim, litellm.rust_bridge.legacy_callbacks, the only Python module
the crate reaches. Before, the crate called Logging methods, litellm.utils
hooks, the logging worker, the executor and several litellm globals
directly, and its tests retyped those signatures by hand, so an outdated
fake could accept a call the real code rejects. python_contract.json lists
each shim function's parameters: a Python test pins it to the real
signatures and a Rust test pins it to the Rust enum.
The lifecycle contract changes to match the Python @client wrapper:
- the driver emits CallEvent::Started before begin, so every host sees one
start time
- RequestContext carries the route-resolved api_key, so legacy pre_call and
post_call receive it, and post_call's additional_args match the Python OCR
path
- Passthrough and its re-aliasing are gone
- async deployment hooks always run, and the "no callbacks" shortcut that
skipped the logging payload is removed, as in the Python path
The OCR api_key is a SecretValue from the wire request onward, so Debug
output upstream of the callback contract cannot leak it.
host-python's RouteHost now classifies native failures once through
classify, and host ops return HostOpError. The OCR route host keeps main's
public errors by sending both through the existing Python map_failure.
OCR transformations, BaseOcrConfig with its response and connection types,
the OCR error, and the HTTP pieces (custom_httpx: http_handler, transport,
media, llm_http_handler with OcrClient and the request/response handler)
now live in litellm-llms at their Python paths. Provider code no longer
reaches into the route: it gets the caller's hooks through a route-neutral
CallHooks trait that core implements over its host, and core dispatches to
llm_http_handler::ocr with the concrete config, the way Python calls
base_llm_http_handler.ocr(provider_config=...).
Core keeps the route: entrypoint, request types, credential fallback,
provider dispatch, the machine and hook glue. ocr/mod.rs no longer
re-exports anything, provider constants moved next to their only users,
and provider tests that drive the whole route moved to core's route test
files. Twenty-two of those were exact copies of tests already there and
were dropped; every one still runs once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
litellm-types mirrors litellm/types (chat, Anthropic Messages and Responses
websocket data) and litellm-core-utils mirrors litellm/litellm_core_utils
(provider resolution, prompt factory, core helpers, call arguments). Route
request types move up to their core route module, the transform contracts
move into base_llm, and the three duplicated provider error enums become one
Error in base_llm/chat/transformation.rs, mirroring BaseLLMException.
The empty-text placeholder goes back to the value Python's factory.py uses;
the provider extraction had changed it to a single space.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extracted from #41733 without the router loop, the cache machine layer, streaming, or the
error, timeout and route-pruning work that moved to #41745
litellm-callbacks holds the contract a native call and its host share: Machine, HostOp,
CallEvent, the in-process run loop, and Passthrough, which is built only by comparing the
caller's inputs with the body the route sends, so a route can never mark a key it rewrote.
litellm-host-python (formerly python-interop) owns the CPython driver and the Execution
handle, and litellm-callbacks-legacy is the @client wrapper as the native call sees it:
function_setup, the deployment hooks, pre_call and post_call, the success and failure fan-out
and the deferred proxy release. OCR is the one route on it, and the old core and bridge
lifecycles are gone
The passthrough rule is the structural fix for the bug #41719 patched in core and #41716
reworks: an inlined remote document no longer counts as the caller's value, so the legacy
adapter never hands the caller's URL back into the body. core/tests/ocr/passthrough.rs pins
it for every route and document source, including that unchanged values stay passthrough,
and callbacks-legacy/tests/payload.rs pins the adapter side with a real pre_call callback
Python OCR integration tests that only exercised core behavior now live as Rust tests, so
tests/test_litellm_rust keeps the cases that need the full Python stack
The Rust bridge imported httpx to construct the provider error response, which
fails in the isolated wheel check where httpx is absent. Rust now raises
RustUpstreamError with a headers attribute and the Python lifecycle wraps it in
a typed UpstreamFailure carrying the httpx.Response before legacy mapping.
Test helpers gained call_native so pytest.raises blocks hold a single call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Delete litellm/ocr/input.py and the native _ocr_file_document, _ocr_upload_document
and _ocr_mime_type helpers. File documents now project to a typed OcrDocumentInput
and the core lifecycle reads local paths, encodes bytes and asks the host to read
file-like objects through a ReadDocument operation before the provider request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Move each route's bridge module under litellm/rust_bridge/<route>/ so a folder
means a Rust implementation exists while the catalog row says whether it is
used. OCR now keeps the Python implementation in litellm/ocr/main.py and the
Rust selection in litellm/ocr/rust.py, removing litellm/ocr/legacy.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Split the Rust workflow into fmt, clippy, nextest and wheel jobs so they run in parallel, replace manual actions/cache with Swatinem/rust-cache, and install a pinned checksum-verified cargo-nextest. Make two python-bridge tests self-contained so they pass when nextest runs each test in its own process.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): count tiktoken cl100k_base admission tokens in Rust
The Rust admission token counter only had the Anthropic tokenizer, so every
other model (OpenAI gpt-4 family, Azure, Gemini, Bedrock non-Claude, Mistral)
tokenized with tiktoken on the Python inference worker.
Add an exact cl100k_base counter to litellm-token-counter: the vendored rank
file (base64 token / rank lines, the bytes Python's tiktoken uses) is parsed
into a byte-level BPE model and the cl100k split pattern is a handwritten
scanner over the shared Unicode classes, so no regex engine runs per request.
Both tokenizers share the message, tool and reply-priming accounting.
The PyO3 TokenCounter gains a from_cl100k_ranks constructor; Python reads the
rank file and passes it in, the way claude_json_str already works. The bridge
selects the counter through the same predicates litellm.token_counter uses
(huggingface_tokenizer_kind, openai_tokenizer_encoding), declines o200k_base,
downloaded HuggingFace and custom tokenizers to Python, and budget reservation
counts once per distinct tokenizer a request names.
The legacy gpt-3.5-turbo-0301 message accounting (4 per message, -1 per name)
stays in Python: the selector declines it through the predicate token_counter
itself uses.
* feat(rust): count tiktoken o200k_base admission tokens in Rust (#40794)
Add a handwritten o200k_base split scanner and TokenCounter::from_o200k_ranks
next to the cl100k_base counter, sharing MergeRanks and the request
accounting. The Python bridge selects it when openai_tokenizer_encoding
names o200k_base, so gpt-4o, gpt-4.1, gpt-5, o1/o3/o4 and chatgpt-4o
requests stop tokenizing on the Python worker under LITELLM_RUST=true
Co-authored-by: yassin <yassin@berri.ai>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes
Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594)
The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of
encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission
count in the gateway pod). A hand-written scanner that yields the same
pieces, then feeds the model directly, counts the same text in 10 ms.
It only engages for tokenizers with the Anthropic shape (optional NFKC,
ByteLevel without prefix space, no post-processor) and falls back to the
full encoder when the text contains an added token. Parity with
encode_fast is tested on random texts, the pieces are compared with the
real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against
oniguruma for every code point.
NFKC runs through unicode-normalization-alignments, the crate and
Unicode tables NormalizedString::nfkc already uses, so the fast path
normalizes exactly what the full encoder would. Using the newer
unicode-normalization crate changed the count for 171 code points that
gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..).
The fast normalizer is compared with the tokenizer's for every scalar
value and on random texts.
The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(python-bridge): split non-streaming bridge modules
* refactor(python-bridge): bring shared function tracing into route layer
* feat(dev): list Python route functions and call sites
* feat(dev): list Rust route functions and call sites
* docs(dev): record OCR parity gaps across Python and Rust
* feat(dev): list executed SDK calls with runtime tracing
* feat(dev): report Python vs Rust SDK pipeline steps in one CLI
* feat(dev): side-by-side pipeline step report in compare CLI
* fix(dev): drop invalid Final annotations in compare cell loop
* feat(dev): blue python-only and yellow rust-only steps in compare CLI
* feat(dev): vertical layout with section spacing in compare CLI
* fix(dev): validate SDK trace stages across sync and async routes
* refactor(rust): align SDK route call structure with Python
* refactor(python-bridge): share sync and async route call wrappers
* refactor(dev): split compare CLI into fixtures, runtime, and report modules
* fix(ci): run SDK trace tests and satisfy test lint
