* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs(rust): plan Python interop foundation
* fix(rust): preserve Python settings coercion at the native boundary
* chore(rust): drop interop planning note
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): resolve OCR provider secrets through an async SecretSource before transformation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): project the Python secret manager into the bridge and resolve OCR secrets through it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): drop premium_user from the secret manager snapshot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust-bridge): read the private key management globals once in the settings snapshot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust): bound the bridge secret manager state cache to the active snapshot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): inline coercion unit tests
* fix(rust): preserve Python secret manager bindings
* refactor(rust-bridge): let settings projectors own their contract specs
Each settings group now declares its SettingSpec rows next to the projector
that reads them, and the manifest test derives python_settings.json from those
tables instead of a hand-copied duplicate. Field carries (group, name) instead
of a dotted path, and coercion gains the dict-item reader plus the Redis
Boolean, certificate-requirement, non-empty string, and numeric adapters that
the cache configuration projection adopts next.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust-bridge): capture the secret manager binding in one settings read
The secret_manager accessor now carries the live client and settings objects,
so the bridge classifies the binding from a single snapshot instead of
re-reading litellm globals. The unreachable native arm and the service alias
go away, the binding-to-state mapping moves next to the snapshot, and the
Python callback precomputes its key_manager name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust-bridge): execute typed settings field declarations
* refactor(rust-bridge): compare cache backends by identity behind one exact trait
cache-response gains an object-safe ExactResponseCache so every exact-match
backend sits behind one pointer; WriteBuffer flushes through it. The bridge's
NativeResponseCache shrinks from nine variants and fifteen per-backend
accessors to an exact service plus the three semantic backends, and facade
mismatch detection compares BackendIdentity values instead of matching on
each backend type. Request projections move next to NativeRequest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust-bridge): drive both Python-embedded semantic caches through one execution
Redis-semantic and Valkey-semantic operations now share one SemanticExecution
body: await the Python embedder, seed the task-local vector, run the native
backend, repeat per batch entry. Valkey drops its with_embedder path in favor
of the same seeded embedder, and each backend keeps its own embedding-failure
policy. PythonEmbedder exposes one call shape. Redis-semantic thresholds are
compared at the backend's f32 width, which un-breaks the redis-stack parity
tests that a 0.8 facade threshold failed before this branch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* wip
* feat(rust-bridge): complete response cache runtime surface
* fix(rust-bridge): preserve secret manager callback exceptions
* refactor(rust-bridge): unify route cache and secret rollout catalog
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Mirror the Redis vertical slice for S3Cache: a litellm-cache-s3 crate built on aws-sdk-s3 with path-style custom endpoints, python-identical put_object metadata (cache-control, expires, content headers), expires-aware get_object, and no-op flush/unsupported test_connection. Wire it through python-bridge config projection, facade guards, test handle, and binding dispatch, plus an in-process S3 stub and parity tests.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Settings sources beyond HTTP (media fetch, Azure Document Intelligence,
Vertex, timeouts) need the same env lookup and precedence merge, so move
them out of litellm-http into core_utils::settings. Lookup readers name the
Python idiom they mirror: get keeps a present empty value like
os.getenv(X, fallback), truthy drops it like an `or` chain, enabled only
switches on for "true". SSL_CERT_FILE now reads through truthy, matching
Python's `if ssl_cert_file and ...` check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OCR transformations, BaseOcrConfig with its response and connection types,
the OCR error, and the HTTP pieces (custom_httpx: http_handler, transport,
media, llm_http_handler with OcrClient and the request/response handler)
now live in litellm-llms at their Python paths. Provider code no longer
reaches into the route: it gets the caller's hooks through a route-neutral
CallHooks trait that core implements over its host, and core dispatches to
llm_http_handler::ocr with the concrete config, the way Python calls
base_llm_http_handler.ocr(provider_config=...).
Core keeps the route: entrypoint, request types, credential fallback,
provider dispatch, the machine and hook glue. ocr/mod.rs no longer
re-exports anything, provider constants moved next to their only users,
and provider tests that drive the whole route moved to core's route test
files. Twenty-two of those were exact copies of tests already there and
were dropped; every one still runs once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
litellm-types mirrors litellm/types (chat, Anthropic Messages and Responses
websocket data) and litellm-core-utils mirrors litellm/litellm_core_utils
(provider resolution, prompt factory, core helpers, call arguments). Route
request types move up to their core route module, the transform contracts
move into base_llm, and the three duplicated provider error enums become one
Error in base_llm/chat/transformation.rs, mirroring BaseLLMException.
The empty-text placeholder goes back to the value Python's factory.py uses;
the provider extraction had changed it to a single space.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extracted from #41733 without the router loop, the cache machine layer, streaming, or the
error, timeout and route-pruning work that moved to #41745
litellm-callbacks holds the contract a native call and its host share: Machine, HostOp,
CallEvent, the in-process run loop, and Passthrough, which is built only by comparing the
caller's inputs with the body the route sends, so a route can never mark a key it rewrote.
litellm-host-python (formerly python-interop) owns the CPython driver and the Execution
handle, and litellm-callbacks-legacy is the @client wrapper as the native call sees it:
function_setup, the deployment hooks, pre_call and post_call, the success and failure fan-out
and the deferred proxy release. OCR is the one route on it, and the old core and bridge
lifecycles are gone
The passthrough rule is the structural fix for the bug #41719 patched in core and #41716
reworks: an inlined remote document no longer counts as the caller's value, so the legacy
adapter never hands the caller's URL back into the body. core/tests/ocr/passthrough.rs pins
it for every route and document source, including that unchanged values stay passthrough,
and callbacks-legacy/tests/payload.rs pins the adapter side with a real pre_call callback
Python OCR integration tests that only exercised core behavior now live as Rust tests, so
tests/test_litellm_rust keeps the cases that need the full Python stack
Delete litellm/ocr/input.py and the native _ocr_file_document, _ocr_upload_document
and _ocr_mime_type helpers. File documents now project to a typed OcrDocumentInput
and the core lifecycle reads local paths, encodes bytes and asks the host to read
file-like objects through a ReadDocument operation before the provider request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
