* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): opt the native Messages and tokenizer suites into Rust explicitly
#42517 made the Messages, token counter and tokenizer routes Python-only, so
tests/test_litellm_rust silently exercised the Python path or failed outright.
Each suite now prepends a RUST_OPT_IN rule for its route, keeping native
coverage without changing the shipped default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust): pop one at a time in the Redis 6 lpop pipeline and drop explanatory comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Children spawned as [sys.executable, -c, ...] put the working directory first on sys.path, so under 'make test-rust-extension' a source checkout shadows the installed wheel and the child imports a litellm with no compiled extension. A shared helper spawns them with -I and asserts the child resolved the same litellm.__file__ as the parent, and a new TQ009 rule flags un-isolated sys.executable spawns.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
