* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Resolves the six conflicts main accumulated against the SDK migration.
endpoints_client.py stays deleted (main only added the safety_identifier
kwarg to it), and its two new importers move onto the SDK clients: the
Bedrock safety_identifier forwarding test in test_responses_e2e.py now
sends its /v1/responses leg through the OpenAI SDK and its chat leg
through ProxyClient.chat, and test_messages_azure_foundry_e2e.py drops
the x-litellm-rust header check together with EXPECT_RUST, which main
removed from e2e_config in a84f68b6e3. proxy_client.py keeps both the
rerank and router-settings model imports. uv.lock is main's lock plus
the anthropic e2e-dev entry, and the pricing JSONs are taken verbatim
from main since this PR never meant to change them.
Rotating the Speech-to-Text stream at 240 s no longer ends the active turn:
the turn and its billed seconds continue on the new stream, forced at 280 s.
Bound the request and event queues (64 and 256) so a slow peer applies
backpressure instead of growing memory. Route a model to the Chirp realtime
path from its cost-map entry (mode audio_transcription plus /v1/realtime)
instead of a hardcoded name. Return on every branch of the recv and
transform helpers (CodeQL mixed returns), have the shared protocol helper take
the provider's error class so the Meta tests assert MuseProtocolError again,
and pin google-cloud-speech in the ci group so unit shards import it.
Replace the bespoke dependency-install CI gate with a real migration:
require mcp>=2.2.0,<3 alongside httpx2>=2.5.0,<3 and pydantic>=2.12.0,<3
in the proxy and mcp extras, drop langchain-mcp-adapters (pins mcp<2)
from the dev group, and remove the dependency-install workflow and
tests/mcp_dependency_tests that only exercised the old pins.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Bridge OpenAI Realtime transcription sessions on vertex_ai/chirp_* models to
Google Speech-to-Text v2 StreamingRecognize over gRPC, so partial and final
transcripts stream back while audio is still being sent. Interim results become
delta events, finals become completed events carrying billed seconds, the gRPC
stream rotates at 240 s under Google's five-minute cap with billed time chained
across rotations, and audio is split into 25 KB requests.
The OpenAI transcription protocol helpers move into a shared module that Meta
Muse now uses too, google-cloud-speech ships behind a new stt-vertex-chirp extra
bundled into the proxy runtime, and the cost map lists /v1/realtime for chirp_3.
The bedrock-realtime extra pinned aws-sdk-bedrock-runtime 0.7.x, whose Config and BedrockRuntimeClient surface is gone in 0.11. The handler now resolves AsyncBedrockRuntimeConfig, builds AsyncBedrockRuntimeClient with the awscrt duplex transport, closes the client when the session ends, and tells an absent SDK apart from an installed but unsupported version. Moves the pin to >=0.10.0,<0.12.0 with the awscrt extra
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Re-migrates the e2e tests that main extended through endpoints_client
since this branch was opened (Azure Foundry, mid-conversation system
messages, Bedrock web search, google native streaming) onto the
provider SDK clients, so no endpoints_client reference remains
maturin already packages non-gitignored files under the Python source
directory of a mixed project, and the built wheel contains
litellm/rust_bridge/_native.pyi without the explicit entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
4.17 was picked on the belief that dnspython only became a core pymongo
dependency there, which is wrong: pymongo has declared dnspython>=1.16.0,<3.0.0
as a core requirement since well before that, so mongodb+srv:// URIs resolve at
4.9 too. The real floor is 4.9, the release AsyncMongoClient landed in, and 4.8
has no AsyncMongoClient at all.
Verified against live Atlas on 4.9: sync and async search, list_search_indexes,
same top hit and score as 4.17. Resolution is unchanged, pymongo 4.17.0 either
way, so this only widens what an existing environment is allowed to bring.
