Add an aws_textract OCR provider on the Rust route, with no Python path. The
detect-document-text model returns plain lines and analyze-document renders
layout and tables as markdown. Both use Textract's synchronous API, so a
multi-page PDF or TIFF is rejected with an error that names the single-page
limit. A call with no region fails instead of falling back to Bedrock's default
SigV4 covers the request body, and host hooks can rewrite that body before it
is sent. litellm-http now has OutboundRequest, which serializes the body once,
shows those bytes to a RequestSigner and is the only thing a route can send.
Chat, audio transcription and OCR build it after their hooks ran, so a callback
that redacts the body still produces a valid Bedrock or Textract signature
ChatCompletionsAuth and AudioTranscriptionAuth are replaced by
litellm_auth::RequestAuth, and one helper in core turns it into a signed or
unsigned request. Audio transcription now signs only the AWS header set and
rejects a forwarded header that SigV4 computes, the same as chat
The OCR catalog routes aws_textract as Rust required, and the dispatch context
reads the provider from the model prefix so a provider scoped rule can match
Python resolves the Mistral key as api_key, MISTRAL_AZURE_API_KEY, then
MISTRAL_API_KEY, and the base as api_base, MISTRAL_AZURE_API_BASE, then
the public endpoint, never reading MISTRAL_API_BASE. Native OCR read
MISTRAL_API_KEY and MISTRAL_API_BASE instead, so with the Azure pair set
it sent the call to a different endpoint with a different key. Empty env
values now fall through like Python's `or` chain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Python resolves the Vertex project and location as call params, then the
litellm.vertex_project / litellm.vertex_location globals, then env, and
Azure AD token refresh from litellm.enable_azure_ad_token_refresh alone.
Native OCR skipped the globals, so a config.yaml litellm_settings value
silently fell through to the credential's project and us-central1, and a
managed identity setup without an API key failed. The bridge now reads
them through a provider_defaults settings group into OcrSettings, and
VertexConfig / AzureAuthInputs slot them in at Python's precedence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Python reads every provider credential fallback (MISTRAL_API_KEY,
AZURE_AI_API_KEY, AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, Azure AD and
Vertex env, ...) through get_secret_str, which consults the configured
key_management_system before os.environ. Native OCR read std::env
directly, so a key held only in the vault went missing and a stale env
copy silently won. OcrClient now carries an injected secret Lookup that
the connection exposes to providers and auth crates; the bridge backs it
with settings.secret -> get_secret_str, pure Rust keeps the process env.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native OCR hardcoded the default of five Python constants that come from
env vars, so an operator setting them saw no effect:
REQUEST_TIMEOUT (Rust used 600s, Python 6000s), MAX_IMAGE_URL_DOWNLOAD_SIZE_MB
(0 disables document downloads), AZURE_OPERATION_POLLING_TIMEOUT,
AZURE_DOCUMENT_INTELLIGENCE_API_VERSION and
AZURE_DOCUMENT_INTELLIGENCE_DEFAULT_DPI. OcrSettings reads them through
Lookup with Python's parsing, the bridge builds it per call and OcrClient
carries it into the connection. A zero per-call timeout now falls back to
REQUEST_TIMEOUT, matching `timeout or request_timeout`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
custom_httpx mirrored a Python module that mixes transport plumbing with
OCR orchestration. The transport half (media fetcher, transport errors,
request and header helpers) now lives in litellm-http next to the pool,
TLS, proxies and settings, and the OCR request handler moves to
base_llm/ocr/handler.rs. Drops the unused deserialize_optional_param and
stale dead_code allows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HttpClientConfig::resolve becomes From<&HttpSettings> for Resolution and client_builder becomes TryFrom<&HttpClientConfig> for reqwest::ClientBuilder, matching the rustls conversion. The verify decision moves into From<&HttpSettings> for Verify, and the proxy environment rule moves next to its flags as HttpSettings::trusts_proxy_env. The curve and cipher results are read with transpose and a default selection, which removes the tuple destructuring
litellm-http now builds the rustls config itself, so one route-neutral place covers roots, the client certificate, ALPN, ssl_ecdh_curve and ssl_security_level. A curve picks the single key exchange group. A cipher string restricts the TLS 1.2 suites it names, and entries rustls cannot express, such as @SECLEVEL=1, are logged once and skipped.
user_url_validation and user_url_allowed_hosts are applied by the media fetcher. Document downloads honor the environment proxy whenever provider calls do, keeping the per-hop address check, and stay on the pinned resolver when no proxy applies.
AIOHTTP_SO_KEEPALIVE, AIOHTTP_TCP_KEEPIDLE, AIOHTTP_TCP_KEEPINTVL, AIOHTTP_TCP_KEEPCNT and AIOHTTP_KEEPALIVE_TIMEOUT map onto the client. A client= argument and a live SSLContext are ignored
Drop the core ocr() facade so VertexAuth and the http pool stay out of litellm-core's
public API, move the http Error enum to error.rs, and inject the media DNS resolver into
HttpClientPool instead of a per-call builder hook the cache key ignored.
The bridge now reads litellm.* HTTP settings only through litellm/rust_bridge/settings.py,
pinned by python_settings.json, while env overrides stay in Rust. This adds the Python
default User-Agent, parses string ssl_verify globals like get_ssl_verify, drops per-call
ssl_verify that Python OCR never honored, and removes the unused request_timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every legacy callback call from callbacks-legacy now goes through one typed
Python shim, litellm.rust_bridge.legacy_callbacks, the only Python module
the crate reaches. Before, the crate called Logging methods, litellm.utils
hooks, the logging worker, the executor and several litellm globals
directly, and its tests retyped those signatures by hand, so an outdated
fake could accept a call the real code rejects. python_contract.json lists
each shim function's parameters: a Python test pins it to the real
signatures and a Rust test pins it to the Rust enum.
The lifecycle contract changes to match the Python @client wrapper:
- the driver emits CallEvent::Started before begin, so every host sees one
start time
- RequestContext carries the route-resolved api_key, so legacy pre_call and
post_call receive it, and post_call's additional_args match the Python OCR
path
- Passthrough and its re-aliasing are gone
- async deployment hooks always run, and the "no callbacks" shortcut that
skipped the logging payload is removed, as in the Python path
The OCR api_key is a SecretValue from the wire request onward, so Debug
output upstream of the callback contract cannot leak it.
host-python's RouteHost now classifies native failures once through
classify, and host ops return HostOpError. The OCR route host keeps main's
public errors by sending both through the existing Python map_failure.
OCR transformations, BaseOcrConfig with its response and connection types,
the OCR error, and the HTTP pieces (custom_httpx: http_handler, transport,
media, llm_http_handler with OcrClient and the request/response handler)
now live in litellm-llms at their Python paths. Provider code no longer
reaches into the route: it gets the caller's hooks through a route-neutral
CallHooks trait that core implements over its host, and core dispatches to
llm_http_handler::ocr with the concrete config, the way Python calls
base_llm_http_handler.ocr(provider_config=...).
Core keeps the route: entrypoint, request types, credential fallback,
provider dispatch, the machine and hook glue. ocr/mod.rs no longer
re-exports anything, provider constants moved next to their only users,
and provider tests that drive the whole route moved to core's route test
files. Twenty-two of those were exact copies of tests already there and
were dropped; every one still runs once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
litellm-types mirrors litellm/types (chat, Anthropic Messages and Responses
websocket data) and litellm-core-utils mirrors litellm/litellm_core_utils
(provider resolution, prompt factory, core helpers, call arguments). Route
request types move up to their core route module, the transform contracts
move into base_llm, and the three duplicated provider error enums become one
Error in base_llm/chat/transformation.rs, mirroring BaseLLMException.
The empty-text placeholder goes back to the value Python's factory.py uses;
the provider extraction had changed it to a single space.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extracted from #41733 without the router loop, the cache machine layer, streaming, or the
error, timeout and route-pruning work that moved to #41745
litellm-callbacks holds the contract a native call and its host share: Machine, HostOp,
CallEvent, the in-process run loop, and Passthrough, which is built only by comparing the
caller's inputs with the body the route sends, so a route can never mark a key it rewrote.
litellm-host-python (formerly python-interop) owns the CPython driver and the Execution
handle, and litellm-callbacks-legacy is the @client wrapper as the native call sees it:
function_setup, the deployment hooks, pre_call and post_call, the success and failure fan-out
and the deferred proxy release. OCR is the one route on it, and the old core and bridge
lifecycles are gone
The passthrough rule is the structural fix for the bug #41719 patched in core and #41716
reworks: an inlined remote document no longer counts as the caller's value, so the legacy
adapter never hands the caller's URL back into the body. core/tests/ocr/passthrough.rs pins
it for every route and document source, including that unchanged values stay passthrough,
and callbacks-legacy/tests/payload.rs pins the adapter side with a real pre_call callback
Python OCR integration tests that only exercised core behavior now live as Rust tests, so
tests/test_litellm_rust keeps the cases that need the full Python stack
Delete litellm/ocr/input.py and the native _ocr_file_document, _ocr_upload_document
and _ocr_mime_type helpers. File documents now project to a typed OcrDocumentInput
and the core lifecycle reads local paths, encodes bytes and asks the host to read
file-like objects through a ReadDocument operation before the provider request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes
Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594)
The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of
encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission
count in the gateway pod). A hand-written scanner that yields the same
pieces, then feeds the model directly, counts the same text in 10 ms.
It only engages for tokenizers with the Anthropic shape (optional NFKC,
ByteLevel without prefix space, no post-processor) and falls back to the
full encoder when the text contains an added token. Parity with
encode_fast is tested on random texts, the pieces are compared with the
real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against
oniguruma for every code point.
NFKC runs through unicode-normalization-alignments, the crate and
Unicode tables NormalizedString::nfkc already uses, so the fast path
normalizes exactly what the full encoder would. Using the newer
unicode-normalization crate changed the count for 171 code points that
gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..).
The fast normalizer is compared with the tokenizer's for every scalar
value and on random texts.
The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(litellm-rust): move provider transforms into litellm-core + crate allowlist test
* feat(litellm-rust): ai-gateway absorbs route I/O (io/) with lib+server feature split
* refactor(litellm-rust): point python-bridge at litellm-ai-gateway
* build(litellm-rust): macOS pyo3 dynamic_lookup linker flag for cdylib builds
* docs(litellm-rust): 3-crate map in README/AGENTS + refresh CLAUDE boundary
* refactor(litellm-rust): update workspace members to the three crates
