The pinned base's /etc/apk/world locks glibc-2.44=2.44-r1, so the apk upgrade in the runtime stage cannot move it. The new digest ships 2.44-r6 on amd64 and arm64
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The pinned base (built 2026-07-02) ships busybox 1.37.0-r61 and
libcrypto3/libssl3 3.6.3-r3. Grype reports 16 fixable findings against
those revisions, 8 of them High, so the image-scan gate fails once it
gets past the migration step.
The runtime stage's `apk upgrade` cannot clear them. wolfi-base writes an
exact `=version` constraint for every package it ships into
/etc/apk/world, so `apk upgrade` is a no-op even though the fixed
revisions are in the repo. Advancing them means moving the digest.
The new digest carries busybox 1.38.0-r1, libcrypto3/libssl3 3.6.3-r5
and glibc 2.43-r15, which is at or above the fix revision Wolfi's secdb
records for every finding. Verified with cosign against
chainguard-images/images release.yaml, and grype reports no fixable
findings on the rebuilt image.
CVE-2026-14456, CVE-2026-54876, CVE-2026-38752, CVE-2026-38753,
CVE-2026-38754, CVE-2026-38755
The migrations image ran `prisma migrate deploy` against a bake anchored in
$HOME with no node in the runtime stage, so prisma-client-py fell through to
nodeenv and tried to download a Node runtime on first start. In an
egress-restricted cluster that fails outright, and under an arbitrary uid the
uid-specific cache path is unreadable, so the job never applies a migration.
Move the bake to /opt/prisma with world-readable modes, install node in the
runtime stage, and pin PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH /
PRISMA_OFFLINE_MODE so the migration entrypoint runs the cached CLI directly.
This is the same treatment the root, non_root and database images already
carry.
Resolves LIT-4727
Refresh the pinned cgr.dev/chainguard/wolfi-base digest from c61ac6 to
42df77a9 (current wolfi-base:latest, a multi-arch index covering amd64
and arm64). This advances the glibc family from 2.43-r8 to 2.43-r10,
with libcrypto3 and libssl3 from 3.6.3-r2 to r3 and libgcc from
16.1.0-r2 to r4; no packages are added or removed.
The image scan reports CVE-2026-6791 against glibc 2.43-r8 (fixed in
r10). The glibc subpackages are exact-version pinned, so the
in-Dockerfile apk upgrade cannot advance them past the base's baked
revision, which is why refreshing the digest is required. Same six
Dockerfiles as #31133
Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles
from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base
digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is
the shipped image, so the runtime digest is what actually resolves the
customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles
tracked a second equally-stale digest; both are unified onto the patched one.
* fix(docker): use system Node in componentized builders + retry apk add
Two failure modes in the componentized image builds (backend, migrations,
gateway) on project-releaser, with the same root cause:
1. The builder-stage `apk add` was missing `libatomic`. `prisma generate`
triggers prisma-client-py's `nodeenv`, which downloads the latest stable
Node.js at build time. Node 26.1.0 (last passing build on 2026-05-20) did
not dynamically link `libatomic.so.1`. Node 26.2.0 (current latest) does,
and the Wolfi builder doesn't ship libatomic — so `npm install prisma@…`
fails with `node: error while loading shared libraries: libatomic.so.1`
and exit 127. Retrying or pinning the Node version is a treadmill; the
root issue is that nodeenv decides the Node version at build time.
Fix: add `nodejs npm` to the builder-stage `apk add` so prisma-client-py
uses Wolfi's own Node via its default `PRISMA_USE_GLOBAL_NODE=true`. The
legacy `docker/Dockerfile.non_root` already does this; the componentized
Dockerfiles regressed it. Setting `PRISMA_USE_GLOBAL_NODE=true` in ENV
redundantly nails the intent so a future env override can't silently
re-enable nodeenv's download.
2. Transient `apk.cgr.dev` mirror flakes during the arm64 leg of multi-arch
builds cause individual package fetches to fail mid-install (we saw
`nss-db-2.43-r7: remote server returned error (try 'apk update')` and
similar for libzstd1, libogg, binutils in this run). None of the
componentized Dockerfiles wrap `apk add` in a retry loop.
Fix: wrap every `apk add` (builder + runtime, all three files) in the
same `for i in 1 2 3; do … && break || sleep 5; done` loop that the
legacy `docker/Dockerfile.non_root` already uses.
Affected files all have the same shape — backend, migrations, gateway —
because they're three near-identical componentizations of the original
monolithic proxy Dockerfile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(docker): trim verbose comments on builder Node setup
Same fix, leaner comments. The apk-add note is 3 lines now (was 8), and the
PRISMA_USE_GLOBAL_NODE bullet matches the existing UV_* comment style.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(docker): make apk-add retry loop fail loudly on exhaustion
Greptile flagged that the retry pattern `apk add ... && break || sleep 5`
exits 0 when all three attempts fail, because `sleep 5` is the last
executed command. A persistent apk.cgr.dev outage would produce a silently
"successful" RUN layer with no packages installed, followed by cryptic
"command not found" errors in downstream RUN steps.
Fix: explicitly fail on the third miss before sleeping. Same pattern in
all six retry loops (3 files × builder + runtime).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MBP.localdomain>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
