* feat(proxy): share database connections across workers with an in-container pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): parse pgbouncer options iteratively to satisfy the recursion gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer with token db auth and retry failed pooler restarts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): build pgbouncer 1.25.2 from a pinned source archive and verify pooler replacements
The public Wolfi repository only carries pgbouncer 1.24.1-r3, which the image
scan rejects (CVE-2026-6664, CVE-2026-6665, CVE-2026-6666, CVE-2025-12819).
All three images now compile the checksummed 1.25.2 release in a builder stage.
The supervisor now waits for a replacement pooler to listen before treating it
as recovered, ends and retries one that never does, and takes the same lock for
stop() and spawn so no replacement can be started after shutdown began.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse to start pgbouncer on a loopback port another process already owns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pgbouncer ready only once its own unix socket answers, not any listener on the port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer older than 1.19, whose unix socket cannot vouch for the tcp port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(helm,terraform): expose the in-container pgbouncer pool for the componentized gateway
Add database.connectionPool to helm/litellm and gateway_connection_pool_* to
terraform/litellm/aws so the componentized gateway can receive the
LITELLM_PGBOUNCER_* env the classic image already honours. Both reject the
pool under IAM or Entra token auth at render/plan time: the pooler holds one
static database password for the life of the pod or task.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gateway): launch the componentized gateway image through a pgbouncer-aware supervisor (#40592)
The componentized gateway image started uvicorn directly, so the in-container
PgBouncer never ran for it: every worker opened its own Prisma pool to the
database. It also passed no keep-alive timeout, so behind a load balancer with
a 60s idle timeout uvicorn's 5s default closed idle connections first and the
balancer returned 502s on scale-out
gateway.launch assembles DATABASE_URL, starts PgBouncer once per pod when
LITELLM_PGBOUNCER_ENABLED is set, hands the workers the loopback URL and then
runs uvicorn on gateway.main:app with KEEPALIVE_TIMEOUT as --timeout-keep-alive.
The image builds PgBouncer 1.25.2 from a checksummed tarball, copies the
compiled Rust extension into the /app source tree it imports from (it was only
in site-packages, which PYTHONPATH=/app shadows) and asserts the native bridge
loads. The app user is added to stats_users so operators can read the PgBouncer
console with the application credentials
The supervisor returns the pooled URL instead of writing into the mapping it
was handed, a database user whose name PgBouncer would split into several
stats_users entries is refused before the config is written, and the launcher
tests drive main() with an injected serve callable
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(terraform): describe the gateway.launch pooler entrypoint in the aws module README
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run pgbouncer exit hooks only in the parent and copy the CA into the runtime dir
Gunicorn workers inherit the parent's atexit table, so a recycled worker (max_requests) stopped the shared pooler and removed its runtime dir, then hung in the inherited Popen lock. The hooks now no-op unless os.getpid() is the process that started PgBouncer
A verified TLS upstream named the operator's CA bundle directly, which is often a 0600 root-owned file that nobody (the user PgBouncer drops to) cannot read, so every server connection failed with "failed to load CA". The bundle is copied into the runtime dir next to the ini and chowned with it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(terraform): run the gateway through gateway.launch and add the gcp connection-pool variables
Cloud Run and ECS overrode the image command with uvicorn gateway.main:app, which skips the supervisor that starts the in-container PgBouncer, so LITELLM_PGBOUNCER_ENABLED was inert on both stacks. Both now exec python -m gateway.launch (under ddtrace-run when USE_DDTRACE is set), and the gcp module gains gateway_connection_pool_enabled / gateway_pool_max_db_connections / gateway_pool_max_client_conn wired to the gateway service only
The test_launch password_env fixture now restores DATABASE_URL even when it was unset: monkeypatch.delenv records nothing for an absent var, so main() left postgresql://...@db.internal in the xdist worker's environ and the key-rotation e2e test in the same proxy-infra shard stopped skipping and tried to reach db.internal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pgbouncer): keep channel_binding and gssencmode off the loopback URL
Prisma would demand TLS channel binding from a pooler that only speaks
plain TCP on 127.0.0.1. Also pass the request the marketplace test
started needing after #40518 landed on top of #40496
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The MongoDB Atlas vector store provider imports pymongo lazily from the
opt-in `mongodb` extra, but none of the shipped images installed that
extra. Any image-based deployment that configured a MongoDB vector store
failed at search time with "requires the 'pymongo' package", which the
user cannot fix without extending the image
Adds `--extra mongodb` to every uv sync in the root Dockerfile,
Dockerfile.database, Dockerfile.non_root, and the gateway component
image. The backend component does not serve /vector_stores so it is left
as is. The extra resolves from the existing uv.lock to pymongo 4.17.0
plus dnspython 2.8.0, no lock change needed
(cherry picked from commit 16fd14f537)
The pinned base (built 2026-07-02) ships busybox 1.37.0-r61 and
libcrypto3/libssl3 3.6.3-r3. Grype reports 16 fixable findings against
those revisions, 8 of them High, so the image-scan gate fails once it
gets past the migration step.
The runtime stage's `apk upgrade` cannot clear them. wolfi-base writes an
exact `=version` constraint for every package it ships into
/etc/apk/world, so `apk upgrade` is a no-op even though the fixed
revisions are in the repo. Advancing them means moving the digest.
The new digest carries busybox 1.38.0-r1, libcrypto3/libssl3 3.6.3-r5
and glibc 2.43-r15, which is at or above the fix revision Wolfi's secdb
records for every finding. Verified with cosign against
chainguard-images/images release.yaml, and grype reports no fixable
findings on the rebuilt image.
CVE-2026-14456, CVE-2026-54876, CVE-2026-38752, CVE-2026-38753,
CVE-2026-38754, CVE-2026-38755
The gateway and backend images generated the prisma client under
HOME=/home/nonroot, so the engine paths baked into the client sat inside a
directory the base image ships at mode 0700. Only uid 65532 can search it,
and prisma resolves those baked paths eagerly with an existence check that
propagates EACCES, so a container started under any other uid dies with a
PermissionError out of pathlib before the PRISMA_QUERY_ENGINE_BINARY
override is ever read. A chart that sets runAsUser, a docker run --user, or
an OpenShift namespace assigning an arbitrary uid all produce that shape,
and the gateway is the request-serving component, so the proxy does not
serve at all.
Bake to /opt/prisma instead, the fixed world-readable path the other three
images already use, and assert at build time that every baked path lands
there. chmod a+rX rather than a+r because prisma executes the engine to
check it can run on this machine. The runtime PRISMA_BINARY_CACHE_DIR pin
keeps the CLI wrapper's own resolution pointing at the bake rather than at
a /home/nonroot/.cache that no longer exists.
The componentized images exec uvicorn directly, so ddtrace-run never wraps the
interpreter. USE_DDTRACE is not inert there; the proxy lifespan still runs
patch_all and litellm's own manual spans still emit. What never gets installed
is ddtrace's ASGI TraceMiddleware: starlette builds its middleware stack lazily
on the first __call__, which is the lifespan scope, so patching from inside the
lifespan body is already too late and no root request span is ever created.
Route both entrypoints through a shared docker/component_entrypoint.sh that
mirrors the monolith's prod_entrypoint.sh contract, including the
DD_TRACE_OPENAI_ENABLED=False export that keeps ddtrace's openai integration
from double-reporting calls litellm instruments itself.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
* fix(gateway): route /a2a through the gateway component
A2A message-send runs the completion bridge, an outbound LLM call, but the
ingress only listed /v1/a2a so the serving routes at /a2a/{agent_id} fell to
the backend catch-all. Backend pods hold no provider credentials, so every
invocation died with a missing-provider-key auth error while the same call
succeeds on the gateway fleet. Adds /a2a to the ingress gateway prefixes and
the gateway route allowlist, plus a parity test so an ingress prefix that the
gateway trims can never reappear
* revert(test): drop the allowlist parity tests
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
The gateway image installed --extra proxy/proxy-runtime/extra_proxy/semantic-router
but not bedrock-realtime, so aws-sdk-bedrock-runtime was absent. Bedrock Nova Sonic
speech-to-speech uses InvokeModelWithBidirectionalStream (which boto3 cannot do) via
that package, so realtime requests failed at startup with 'Missing
aws_sdk_bedrock_runtime'. Add the extra to both uv sync stages; it is already in
uv.lock so --frozen resolves, and the package/marker (python>=3.12) matches the
python3.13 image.
Refresh the pinned cgr.dev/chainguard/wolfi-base digest from c61ac6 to
42df77a9 (current wolfi-base:latest, a multi-arch index covering amd64
and arm64). This advances the glibc family from 2.43-r8 to 2.43-r10,
with libcrypto3 and libssl3 from 3.6.3-r2 to r3 and libgcc from
16.1.0-r2 to r4; no packages are added or removed.
The image scan reports CVE-2026-6791 against glibc 2.43-r8 (fixed in
r10). The glibc subpackages are exact-version pinned, so the
in-Dockerfile apk upgrade cannot advance them past the base's baked
revision, which is why refreshing the digest is required. Same six
Dockerfiles as #31133
Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles
from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base
digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is
the shipped image, so the runtime digest is what actually resolves the
customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles
tracked a second equally-stale digest; both are unified onto the patched one.
* fix(docker): use system Node in componentized builders + retry apk add
Two failure modes in the componentized image builds (backend, migrations,
gateway) on project-releaser, with the same root cause:
1. The builder-stage `apk add` was missing `libatomic`. `prisma generate`
triggers prisma-client-py's `nodeenv`, which downloads the latest stable
Node.js at build time. Node 26.1.0 (last passing build on 2026-05-20) did
not dynamically link `libatomic.so.1`. Node 26.2.0 (current latest) does,
and the Wolfi builder doesn't ship libatomic — so `npm install prisma@…`
fails with `node: error while loading shared libraries: libatomic.so.1`
and exit 127. Retrying or pinning the Node version is a treadmill; the
root issue is that nodeenv decides the Node version at build time.
Fix: add `nodejs npm` to the builder-stage `apk add` so prisma-client-py
uses Wolfi's own Node via its default `PRISMA_USE_GLOBAL_NODE=true`. The
legacy `docker/Dockerfile.non_root` already does this; the componentized
Dockerfiles regressed it. Setting `PRISMA_USE_GLOBAL_NODE=true` in ENV
redundantly nails the intent so a future env override can't silently
re-enable nodeenv's download.
2. Transient `apk.cgr.dev` mirror flakes during the arm64 leg of multi-arch
builds cause individual package fetches to fail mid-install (we saw
`nss-db-2.43-r7: remote server returned error (try 'apk update')` and
similar for libzstd1, libogg, binutils in this run). None of the
componentized Dockerfiles wrap `apk add` in a retry loop.
Fix: wrap every `apk add` (builder + runtime, all three files) in the
same `for i in 1 2 3; do … && break || sleep 5; done` loop that the
legacy `docker/Dockerfile.non_root` already uses.
Affected files all have the same shape — backend, migrations, gateway —
because they're three near-identical componentizations of the original
monolithic proxy Dockerfile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(docker): trim verbose comments on builder Node setup
Same fix, leaner comments. The apk-add note is 3 lines now (was 8), and the
PRISMA_USE_GLOBAL_NODE bullet matches the existing UV_* comment style.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(docker): make apk-add retry loop fail loudly on exhaustion
Greptile flagged that the retry pattern `apk add ... && break || sleep 5`
exits 0 when all three attempts fail, because `sleep 5` is the last
executed command. A persistent apk.cgr.dev outage would produce a silently
"successful" RUN layer with no packages installed, followed by cryptic
"command not found" errors in downstream RUN steps.
Fix: explicitly fail on the third miss before sleeping. Same pattern in
all six retry loops (3 files × builder + runtime).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MBP.localdomain>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
