* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* perf(ci): cache the Rust build the unit shards compile from scratch
Every unit shard installs the workspace, and the root package builds through
maturin, so each of the eleven jobs compiles litellm-rust/crates/python-bridge
in release mode before a single test runs. That step measured 2m40s a shard on
2026-08-21, which is more wall clock than the entire unit tier spends running
tests, and none of it was cached: the uv cache covers wheels it downloads, not
wheels it builds, and a path dependency whose source moves every commit can
never hit that cache anyway.
A composite action now exports CARGO_TARGET_DIR to a fixed workspace path and
caches it alongside the Cargo registry, keyed on Cargo.lock. Cargo rebuilds only
what changed, so a warm job pays for the bridge crate rather than its whole
dependency graph. Measured locally, that is 34s cold against 8s warm, including
after a Python-only or Rust-only edit.
The absolute path matters: uv builds the wheel from its own working directory,
so a relative target directory lands the artifacts where nothing can find them
again.
* perf(ci): cache the Rust build in the other four workflows that sync the workspace
code-quality, mcp, documentation and the schema.d.ts check each install the
workspace and so each compile the bridge from scratch, measured at 138s, 177s,
163s and 154s on 2026-08-21. The lint job pays the same and is left to #37783,
which already owns that file's setup section.
* perf(ci): cache the Rust build in the lint job too
* fix(ci): cache cargo's own target directory instead of redirecting it
uv builds the wheel in place, so cargo already writes to litellm-rust/target,
which test-rust.yml has cached all along. Redirecting CARGO_TARGET_DIR bought
nothing and cost a GITHUB_ENV write that zizmor rejects as a code-execution
path.
* chore(ci): raise the job backstop for the added setup step
The cargo cache is a fifth bounded setup step, so the base's setup ceiling goes
30m to 35m and every job budget follows: 55 to 60, and proxy-server's 95 to 100.
check_workflow_startup_safety enforces exactly this sum, and failed on the first
push without it.
* perf(ci): cache the Rust build in the four remaining workflows that sync
Six workflows were wired; ten install the workspace. The four left out still
compile the pyo3 bridge from scratch.
test-terraform-provider.yml is the one that matters per PR: its endpoint-drift
job triggers on any change under litellm/proxy/**. The other three are a
scheduled load check, a manual mutation run, and the staging-push counts
publisher, whose gate syncs the project inside scripts/type_check_gate.py
rather than in a workflow step, so nothing in the file names the build it pays
for.
PR #37550 taught the backend unit-test shards to read the pull request's own
file list, but four required jobs were never wired to that gate and ran in full
on every pull request regardless of what it touched. A UI-only pull request
still paid roughly 17 runner-minutes of Python work it could not have affected,
and a backend-only one still installed and built the dashboard.
Lint and the MCP suite now take the existing backend decision. The dashboard
build and unit tests take a new ui decision, which tracks ui/ rather than
reusing client: client deliberately runs whenever the backend changes, because
it gates CircleCI's end-to-end jobs that drive a real proxy, while the build and
the unit tests cannot see the backend at all. CI config counts as ui-relevant
too, so a pull request that rewrites the dashboard workflows still exercises
them instead of shipping unvalidated.
The gate stays inside the job rather than moving to on.paths or to a job-level
condition on the shard callers. A workflow filtered out by on.paths never starts
and never reports, so a required check waits forever, and a skipped caller job
publishes its own name instead of the nested "<shard> / Run tests" the ruleset
requires. Both were measured before settling on this shape.
Three setup steps in the shard base and in the documentation job also leaked
past the gate, so a skipped shard still spent about twelve seconds installing uv
and restoring its cache. They now carry the same condition, and the documentation
job stops cloning litellm-docs when it has nothing to validate.
detect-backend-changes diffed the event payload's base.sha against the
checked-out ref. Those are two different points in time: actions/checkout
resolves refs/pull/N/merge, and GitHub recomputes that ref whenever the base
branch advances, so the diff picked up whatever landed on staging between the
event firing and the job starting. On a recent UI-only pull request three
backend commits from staging were attributed to the branch, and every backend
shard ran in full
Ask the API which files the pull request touches instead. That is the same set
the Files changed tab shows, and it is immune to either endpoint moving. The
shell body moves into .github/scripts/detect_backend_changes.sh so it can be
exercised directly, and the fail-open paths now also cover an API failure, a
file list past the API's 3000-entry listing ceiling, and a classifier that
prints something unexpected
`prisma generate` runs `npm install prisma@<version>` whenever the
prisma-client-py binary cache directory has no CLI entrypoint, pulling ~85 MB
of query and schema engines over the network. Every workflow pointed
PRISMA_BINARY_CACHE_DIR at `${{ runner.temp }}/prisma-cache`, which GitHub
wipes and recreates per job, so that cache was empty on every job of every
run and the download was never avoidable.
The download is normally a few seconds and occasionally minutes. On one
proxy-db run it took 5m18s on a single shard against 3.8s on its eleven
siblings, which pushed the job past its 15 minute timeout and cancelled a
shard whose tests were at 99% and all passing.
Leave PRISMA_BINARY_CACHE_DIR unset so the binaries land in the
prisma-client-py default, which is already keyed by prisma and engine
version, and restore both that path and the @prisma/engines staging cache
through a shared composite action.
Job timeouts also counted setup against the test budget. `timeout-minutes`
now bounds the pytest step, with a separate allowance for checkout,
dependency install, and client generation, so slow setup shows up as a slow
job instead of a cancelled test run.
check_prisma_binary_cache.py guards all three invariants: no workflow
reintroduces the override, every job that generates the client restores the
cache, and the version the action greps out of uv.lock still resolves.
Audit of .github/workflows/ via gh run history shows the following have
either never run or have been dormant for 10+ weeks. CI coverage that
still matters is preserved on CircleCI (e.g. llm_translation_testing).
Removed workflows:
- test-litellm.yml — workflow_dispatch only, last run 2026-02-12 (cancelled);
CCI local_testing_part1/2 covers the same tests
- llm-translation-testing.yml — last run 2025-07-10; replaced by CCI
llm_translation_testing job (run_llm_translation_tests.py kept for the
make test-llm-translation target)
- run_observatory_tests.yml — last run 2026-03-03 (cancelled)
- scan_duplicate_issues.yml — last run 2026-03-02 (failure)
- publish_to_pypi.yml — never run
- read_pyproject_version.yml — fires on every push to main but its echoed
version output is not consumed by any downstream step
Removed orphan files (no callers in workflows, CCI, or Makefile):
- .github/workflows/README.md — documented only publish_to_pypi.yml
- .github/workflows/update_release.py + results_stats.csv
- .github/actions/helm-oci-chart-releaser/
The Helm chart on GHCR displays a `docker pull` command instead of
the correct `helm pull oci://` command. This is because the OCI artifact
is missing the `org.opencontainers.image.source` annotation that GHCR
uses to identify and properly display Helm charts.
Changes:
- Add OCI annotations to Chart.yaml (source + url) which Helm 3.10+
propagates to the OCI manifest on push
- Install explicit Helm v3.20.0 via azure/setup-helm@v4 for reproducible
builds and proper OCI annotation support
- Remove deprecated HELM_EXPERIMENTAL_OCI env var (OCI is GA since Helm 3.8)
