The component runtime stages copied the builder's whole /app, which is the
entire repository plus build output: six lockfiles, helm, terraform, docs,
.env.example and the Rust build tree. Measured at runtime with an audit hook
on a booted container, the gateway reads only gateway/, litellm/ and
enterprise/ from /app, the backend only backend/, litellm/ and enterprise/,
and the migrations job only litellm/ and run.py. The pyproject.toml read is
prisma-python's optional [tool.prisma] lookup, which ours does not define
Copy just those, plus .venv, the entrypoint script and schema.prisma for the
gateway and backend. PYTHONPATH=/app and the in-Dockerfile asserts are
unchanged
Bump the pinned wolfi-base digest in every Wolfi Dockerfile to 360896ef, which
ships brotli 1.2.1-r1 and ca-certificates-bundle 20260909-r3. The previous
digest already carried libbrotli 1.2.0-r6 as an OpenSSL dependency, and
`apk add --no-cache` never upgrades a package the base already has
Move the UI image to nginx 1.31.8-alpine3.24, whose CA bundle is the 20261020
Mozilla set, and upgrade brotli-libs in its runtime stage since no nginx
digest ships 1.2.1 yet
* fix(security): bump grpcio and boto3 so their bundled CA files drop the compromised eMudhra roots
grpc ignores SSL_CERT_FILE and verifies every channel against its own bundled roots.pem. In 1.78.0 that file still trusts emSign ECC Root CA - G3 and AAA Certificate Services, so Vertex AI, Google KMS, Chirp STT and OTLP gRPC exporters (Arize) accept all 14 advisory chains. 1.84.1 is the first release whose roots.pem drops both roots.
botocore uses certifi when it is installed and falls back to its bundled cacert.pem otherwise. Through 1.43.116 that file still trusts AAA Certificate Services. Requiring boto3 1.43.117 pulls in botocore 1.43.117, the first release whose cacert.pem rejects all 14 advisory chains, so the fallback stays safe if certifi is ever absent
* fix(docker): provide OpenSSL 3 when generating Prisma engines
(cherry picked from commit 52166b51a1)
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3
Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility
Co-authored-by: shin-berri <shin-laptop@berri.ai>
(cherry picked from commit f2118fc767)
The pinned base's /etc/apk/world locks glibc-2.44=2.44-r1, so the apk upgrade in the runtime stage cannot move it. The new digest ships 2.44-r6 on amd64 and arm64
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The monolithic images install the saml extra but the split backend image
did not, so /sso/saml/* returned 501 on Helm split-image deployments.
The gateway image is unchanged since /sso/ routes are backend-only.
The pinned base (built 2026-07-02) ships busybox 1.37.0-r61 and
libcrypto3/libssl3 3.6.3-r3. Grype reports 16 fixable findings against
those revisions, 8 of them High, so the image-scan gate fails once it
gets past the migration step.
The runtime stage's `apk upgrade` cannot clear them. wolfi-base writes an
exact `=version` constraint for every package it ships into
/etc/apk/world, so `apk upgrade` is a no-op even though the fixed
revisions are in the repo. Advancing them means moving the digest.
The new digest carries busybox 1.38.0-r1, libcrypto3/libssl3 3.6.3-r5
and glibc 2.43-r15, which is at or above the fix revision Wolfi's secdb
records for every finding. Verified with cosign against
chainguard-images/images release.yaml, and grype reports no fixable
findings on the rebuilt image.
CVE-2026-14456, CVE-2026-54876, CVE-2026-38752, CVE-2026-38753,
CVE-2026-38754, CVE-2026-38755
The gateway and backend images generated the prisma client under
HOME=/home/nonroot, so the engine paths baked into the client sat inside a
directory the base image ships at mode 0700. Only uid 65532 can search it,
and prisma resolves those baked paths eagerly with an existence check that
propagates EACCES, so a container started under any other uid dies with a
PermissionError out of pathlib before the PRISMA_QUERY_ENGINE_BINARY
override is ever read. A chart that sets runAsUser, a docker run --user, or
an OpenShift namespace assigning an arbitrary uid all produce that shape,
and the gateway is the request-serving component, so the proxy does not
serve at all.
Bake to /opt/prisma instead, the fixed world-readable path the other three
images already use, and assert at build time that every baked path lands
there. chmod a+rX rather than a+r because prisma executes the engine to
check it can run on this machine. The runtime PRISMA_BINARY_CACHE_DIR pin
keeps the CLI wrapper's own resolution pointing at the bake rather than at
a /home/nonroot/.cache that no longer exists.
The componentized images exec uvicorn directly, so ddtrace-run never wraps the
interpreter. USE_DDTRACE is not inert there; the proxy lifespan still runs
patch_all and litellm's own manual spans still emit. What never gets installed
is ddtrace's ASGI TraceMiddleware: starlette builds its middleware stack lazily
on the first __call__, which is the lifespan scope, so patching from inside the
lifespan body is already too late and no root request span is ever created.
Route both entrypoints through a shared docker/component_entrypoint.sh that
mirrors the monolith's prod_entrypoint.sh contract, including the
DD_TRACE_OPENAI_ENABLED=False export that keeps ddtrace's openai integration
from double-reporting calls litellm instruments itself.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Refresh the pinned cgr.dev/chainguard/wolfi-base digest from c61ac6 to
42df77a9 (current wolfi-base:latest, a multi-arch index covering amd64
and arm64). This advances the glibc family from 2.43-r8 to 2.43-r10,
with libcrypto3 and libssl3 from 3.6.3-r2 to r3 and libgcc from
16.1.0-r2 to r4; no packages are added or removed.
The image scan reports CVE-2026-6791 against glibc 2.43-r8 (fixed in
r10). The glibc subpackages are exact-version pinned, so the
in-Dockerfile apk upgrade cannot advance them past the base's baked
revision, which is why refreshing the digest is required. Same six
Dockerfiles as #31133
Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles
from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base
digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is
the shipped image, so the runtime digest is what actually resolves the
customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles
tracked a second equally-stale digest; both are unified onto the patched one.
* fix(docker): use system Node in componentized builders + retry apk add
Two failure modes in the componentized image builds (backend, migrations,
gateway) on project-releaser, with the same root cause:
1. The builder-stage `apk add` was missing `libatomic`. `prisma generate`
triggers prisma-client-py's `nodeenv`, which downloads the latest stable
Node.js at build time. Node 26.1.0 (last passing build on 2026-05-20) did
not dynamically link `libatomic.so.1`. Node 26.2.0 (current latest) does,
and the Wolfi builder doesn't ship libatomic — so `npm install prisma@…`
fails with `node: error while loading shared libraries: libatomic.so.1`
and exit 127. Retrying or pinning the Node version is a treadmill; the
root issue is that nodeenv decides the Node version at build time.
Fix: add `nodejs npm` to the builder-stage `apk add` so prisma-client-py
uses Wolfi's own Node via its default `PRISMA_USE_GLOBAL_NODE=true`. The
legacy `docker/Dockerfile.non_root` already does this; the componentized
Dockerfiles regressed it. Setting `PRISMA_USE_GLOBAL_NODE=true` in ENV
redundantly nails the intent so a future env override can't silently
re-enable nodeenv's download.
2. Transient `apk.cgr.dev` mirror flakes during the arm64 leg of multi-arch
builds cause individual package fetches to fail mid-install (we saw
`nss-db-2.43-r7: remote server returned error (try 'apk update')` and
similar for libzstd1, libogg, binutils in this run). None of the
componentized Dockerfiles wrap `apk add` in a retry loop.
Fix: wrap every `apk add` (builder + runtime, all three files) in the
same `for i in 1 2 3; do … && break || sleep 5; done` loop that the
legacy `docker/Dockerfile.non_root` already uses.
Affected files all have the same shape — backend, migrations, gateway —
because they're three near-identical componentizations of the original
monolithic proxy Dockerfile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(docker): trim verbose comments on builder Node setup
Same fix, leaner comments. The apk-add note is 3 lines now (was 8), and the
PRISMA_USE_GLOBAL_NODE bullet matches the existing UV_* comment style.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(docker): make apk-add retry loop fail loudly on exhaustion
Greptile flagged that the retry pattern `apk add ... && break || sleep 5`
exits 0 when all three attempts fail, because `sleep 5` is the last
executed command. A persistent apk.cgr.dev outage would produce a silently
"successful" RUN layer with no packages installed, followed by cryptic
"command not found" errors in downstream RUN steps.
Fix: explicitly fail on the third miss before sleeping. Same pattern in
all six retry loops (3 files × builder + runtime).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MBP.localdomain>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
