* fix(db): carry DATABASE_SSLMODE/DATABASE_SSLROOTCERT into the assembled writer and reader URLs
The componentized gateway supervisor starts the in-container PgBouncer from the
DATABASE_URL assembled out of the discrete DATABASE_* vars before config.yaml is
read, so an IAM URL had no way to request verified TLS: PgBouncer dialed the
server with server_tls_sslmode = prefer (no SNI, no verification) and public
RDS endpoints rejected the handshake. Two new env vars, exposed by the chart as
database.writer.sslMode / sslRootCert, are appended as libpq sslmode/sslrootcert
to every writer and reader URL the settings assemble (never to a pinned URL),
then translated for Prisma as before. Token refresh now also carries Prisma's
sslmode/sslcert/sslaccept over into the re-minted URL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): keep TLS params on the CLI password URL and the initial IAM reader mint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): treat DATABASE_SSLROOTCERT on its own as verify-full and cover collector and migrations TLS env
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(db): type the reader mint TLS test double and drop its mutable capture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The componentized chart rendered litellm.connectionPoolEnv into the gateway
container only, so with database.connectionPool.enabled and
gateway.collector.enabled the collector's Prisma client opened its own pool
straight to Postgres instead of going through the pod-local PgBouncer.
Render the same include in the collector container, drop the gateway.extraEnv
workaround from the collector test, and add enabled/disabled regression
assertions for the collector in both helm charts and the terraform aws and gcp
modules, which already pass the pool env to their collector containers. The
module READMEs note the IAM token-auth exception, where the collector keeps a
direct connection by design.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
On the release gate the e2e tests only see the nginx router, and the chart's
ingress sent /debug/memory/summary to the backend catch-all, so the RSS check
measured the backend pod instead of the gateway workers that serve the failing
requests. Render it as an Exact gateway path next to /test, name the host in the
summary response so workers behind one origin never collide on pid alone, and
key the harness readings by (origin, hostname, pid)
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar
py-spy on the gateway showed the post-response _PROXY_track_cost_callback,
spend-log and DBSpendUpdateWriter work running on the inference workers'
event loop, so a DB or Redis stall backed up the request path.
When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact
typed SpendEvent per success and hands it to a SpendEventProducer that ships
it over a unix socket (default) or loopback-only TCP to a sidecar started as
`python -m gateway.spend_worker`. The sidecar runs the unchanged
_ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url).
When the sidecar is unreachable, the buffer is full, or the gateway shuts
down with events still queued or in flight, the producer applies
LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The
sidecar half-closes producers on SIGTERM and drains, the producer treats
EOF as unavailable, and the gateway flushes buffered spend counters on
shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its
own process log.
Helm: both charts gain an opt-in spend-worker sidecar container sharing an
emptyDir socket dir, and the componentized chart's HPA uses a
ContainerResource CPU metric scoped to the gateway container so sidecar
CPU does not drive inference scaling.
* feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks
Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): retrieve a completed batch in the in-process spend path test
The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename the spend worker sidecar to collector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown
The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime
stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the
pod-local pgbouncer URL itself, replacing gateway/collector.py
The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it.
SpendEventProducer shields an in-progress fallback from the writer task cancellation so close()
no longer loses an event already handed to the in-process pipeline
Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs,
flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(terraform): drop the gcp job duplicated by the aws/gcp matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports
The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format pooled_database_url after the pgbouncer rebase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain
Cache hits updated the logging object after the early return, so the offloaded spend event carried
preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against
producer transports uvloop already closed so one dead connection cannot abort the drain
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): keep the gcp collector port off the metrics sidecar health port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): collector connects to Postgres directly under IAM or Entra token auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): share database connections across workers with an in-container pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): parse pgbouncer options iteratively to satisfy the recursion gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer with token db auth and retry failed pooler restarts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): build pgbouncer 1.25.2 from a pinned source archive and verify pooler replacements
The public Wolfi repository only carries pgbouncer 1.24.1-r3, which the image
scan rejects (CVE-2026-6664, CVE-2026-6665, CVE-2026-6666, CVE-2025-12819).
All three images now compile the checksummed 1.25.2 release in a builder stage.
The supervisor now waits for a replacement pooler to listen before treating it
as recovered, ends and retries one that never does, and takes the same lock for
stop() and spawn so no replacement can be started after shutdown began.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse to start pgbouncer on a loopback port another process already owns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pgbouncer ready only once its own unix socket answers, not any listener on the port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer older than 1.19, whose unix socket cannot vouch for the tcp port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(helm,terraform): expose the in-container pgbouncer pool for the componentized gateway
Add database.connectionPool to helm/litellm and gateway_connection_pool_* to
terraform/litellm/aws so the componentized gateway can receive the
LITELLM_PGBOUNCER_* env the classic image already honours. Both reject the
pool under IAM or Entra token auth at render/plan time: the pooler holds one
static database password for the life of the pod or task.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gateway): launch the componentized gateway image through a pgbouncer-aware supervisor (#40592)
The componentized gateway image started uvicorn directly, so the in-container
PgBouncer never ran for it: every worker opened its own Prisma pool to the
database. It also passed no keep-alive timeout, so behind a load balancer with
a 60s idle timeout uvicorn's 5s default closed idle connections first and the
balancer returned 502s on scale-out
gateway.launch assembles DATABASE_URL, starts PgBouncer once per pod when
LITELLM_PGBOUNCER_ENABLED is set, hands the workers the loopback URL and then
runs uvicorn on gateway.main:app with KEEPALIVE_TIMEOUT as --timeout-keep-alive.
The image builds PgBouncer 1.25.2 from a checksummed tarball, copies the
compiled Rust extension into the /app source tree it imports from (it was only
in site-packages, which PYTHONPATH=/app shadows) and asserts the native bridge
loads. The app user is added to stats_users so operators can read the PgBouncer
console with the application credentials
The supervisor returns the pooled URL instead of writing into the mapping it
was handed, a database user whose name PgBouncer would split into several
stats_users entries is refused before the config is written, and the launcher
tests drive main() with an injected serve callable
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(terraform): describe the gateway.launch pooler entrypoint in the aws module README
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run pgbouncer exit hooks only in the parent and copy the CA into the runtime dir
Gunicorn workers inherit the parent's atexit table, so a recycled worker (max_requests) stopped the shared pooler and removed its runtime dir, then hung in the inherited Popen lock. The hooks now no-op unless os.getpid() is the process that started PgBouncer
A verified TLS upstream named the operator's CA bundle directly, which is often a 0600 root-owned file that nobody (the user PgBouncer drops to) cannot read, so every server connection failed with "failed to load CA". The bundle is copied into the runtime dir next to the ini and chowned with it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(terraform): run the gateway through gateway.launch and add the gcp connection-pool variables
Cloud Run and ECS overrode the image command with uvicorn gateway.main:app, which skips the supervisor that starts the in-container PgBouncer, so LITELLM_PGBOUNCER_ENABLED was inert on both stacks. Both now exec python -m gateway.launch (under ddtrace-run when USE_DDTRACE is set), and the gcp module gains gateway_connection_pool_enabled / gateway_pool_max_db_connections / gateway_pool_max_client_conn wired to the gateway service only
The test_launch password_env fixture now restores DATABASE_URL even when it was unset: monkeypatch.delenv records nothing for an absent var, so main() left postgresql://...@db.internal in the xdist worker's environ and the key-rotation e2e test in the same proxy-infra shard stopped skipping and tried to reach db.internal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pgbouncer): keep channel_binding and gssencmode off the loopback URL
Prisma would demand TLS channel binding from a pooler that only speaks
plain TCP on 127.0.0.1. Also pass the request the marketplace test
started needing after #40518 landed on top of #40496
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(infra): scale gateway on per-pod RPM and TPM in Helm and Terraform
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): require the metrics server before rendering the gateway ServiceMonitor
The http port serves /metrics/ behind virtual-key auth, so a ServiceMonitor
pointed at it only collects 401s and the RPM/TPM HPA metrics never appear
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(infra): express gateway HPA, KEDA and ECS workload targets per second
Rename the per-pod request and token targets in both Helm charts and the
AWS module from per minute to per second, and shorten the recommended
Prometheus rate window to [1m] with no * 60 so the adapter and KEDA
signals are what the HPA compares against. ECS keeps CloudWatch's
60-second aggregation: the ALB target is 60x the per-second variable and
the token metric math divides the period Sum by 60 before dividing by
the running task count.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(deploy): expose SSE keepalive, pre-call checks and a metrics sidecar in Helm and Terraform
Typed reliability values on both Helm charts and the AWS/GCP Terraform
modules, a dedicated ClusterIP Service for the separate Prometheus port,
a /health route on the metrics server and dead-worker pruning so the
aggregate does not keep stale multiprocess samples.
Resolves LIT-7142
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(deploy): drop reliability config from Helm and Terraform, keep only the metrics sidecar
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover startup pruning of dead workers' live gauges and unsignalable pids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A dotted extraPaths entry kept its requested Exact or Prefix type under
ingress.controller=nginx, so the admission webhook rejected the render the
option exists to avoid, and the duplicate check compared the raw type against
the built-in paths' normalized one, letting a repeated /favicon.ico through.
Extra paths now go through the same controller normalization before both the
duplicate check and the render.
ingress-nginx's admission webhook (strict-validate-path-type, on by default
from v1.12.0 until v1.12.6 / v1.13.2 allowed dots again) rejects the chart's
/favicon.ico Exact and /eu.assemblyai Prefix rules, so helm install fails on
any cluster it fronts. A new ingress.controller value (alb, the default, or
nginx) renders dotted built-in paths as ImplementationSpecific under nginx,
which serves them as plain prefix locations, and drops the ALB-only /*.txt
wildcard rule there. The default render is unchanged
* fix(helm): scale the classic chart's HPA out at the documented 60 percent CPU
The litellm-helm chart shipped targetCPUUtilizationPercentage: 80, which is
unexamined helm create scaffold rather than a chosen number. It arrived packaged
with the stock minReplicas: 1, maxReplicas: 100, a commented-out
targetMemoryUtilizationPercentage: 80, and the boilerplate "such as Minikube"
comment, the same provenance as the 128Mi resource example this file just
corrected.
60 is the documented recommendation. The mechanism behind it is scale-up lag:
the chart's own startupProbe is failureThreshold: 30 times periodSeconds: 10, so
a replica can take up to 300 seconds to become ready, and a pod added at 80
percent utilization arrives minutes after saturation.
The memory target stays commented out on purpose. The prisma query engine's
resident memory is a high-water mark that ratchets to the pod's worst-ever write
and is never returned, so a memory-target HPA reads the largest write a pod ever
did rather than what it is doing now, and replicas ratchet up without scaling
back in.
hpa_tests.yaml carried its second suite after a YAML document separator, and
helm-unittest loads only the first document per file, so that suite never ran;
an assertion planted in it still passed. Fold it into the one live suite and add
coverage pinning the rendered CPU target, the absence of a memory metric by
default, and that overrides still take effect.
Bump the chart to 1.1.2, since rendered output changes for anyone running with
autoscaling enabled.
* fix(helm): bump litellm-helm to 1.1.3 after rebase onto 1.1.2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The generated masterkey Secret rendered a fresh randAlphaNum value on every
release, so any helm upgrade with masterkeySecretName and masterkey unset
rotated the master key and invalidated every client holding the old one.
Look up the existing Secret in the release namespace and reuse its value,
falling back to a random key only on first install.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The componentized chart's Ingress renders a fixed path set: the UI paths, a
hardcoded gateway prefix list mirroring gateway/routes/allowlist.py, and a `/`
Prefix catch-all to the backend. values.yaml exposes no path knob, so an
operator who needs a route the chart does not know about has to fork the
template.
That prefix list is a snapshot of the data plane at release time and drifts
from it. `/watsonx` is in GATEWAY_PATH_PREFIXES with no ingress rule and no
backend prefix, so `/watsonx/*` falls into the catch-all, reaches the backend,
and 404s. A provider passthrough prefix added upstream after the chart version
an operator runs has the same shape, and a custom
general_settings.pass_through_endpoints route has a path only the operator
knows, so the chart can never ship a rule for it at all.
ingress.extraPaths takes a list of {path, service, pathType} entries rendered
in addition to the built-in paths, never in place of them. They render after
every built-in path and before the backend catch-all, and an entry repeating a
path the chart already routes is rejected: duplicates within one rule are
resolved by position or by controller-specific tie breaking, so an entry at
`/` Prefix would swallow the whole backend management API rather than add to
it. `service` picks the component Service — gateway by default, or backend or
ui — so an operator does not have to reconstruct release-templated Service
names, and `pathType` defaults to Prefix. A non-mapping entry, an entry with
no path, an unknown service, an unknown pathType, and a duplicate each abort
the render naming the offending index rather than emitting an Ingress that
misroutes traffic.
The duplicate check needs the built-in paths as data, so the UI paths move
from six literal YAML blocks into a $uiPaths list rendered by one range, and
their explanatory comments move from the rendered manifest into template
comments. The Ingress object this produces is unchanged: the same 92 paths in
the same order with the same backends, verified by parsing both renders.
The target component still answers only the paths its own route allowlist
keeps, so this makes a route routable, not servable.
* fix(ui): boot the UI image as an arbitrary uid by anchoring nginx writes under /tmp
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): type the arbitrary-uid image test fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(helm): compose DATABASE_URL_READ_REPLICA from a reader host secret key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(helm): cover reader host composition and readReplicaUrlKey precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): suppress unused reader host env when readReplicaUrlKey is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): emit reader host only when readReplicaUrl composition is active
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Azure Database for PostgreSQL Flexible Server takes a Microsoft Entra ID access
token as the connection password, and those tokens last about an hour, so a
proxy pointed at one dies shortly after boot unless something keeps minting
fresh ones
Set AZURE_POSTGRESQL_AUTH=True (or pass --azure_postgresql_auth) alongside
DATABASE_HOST, DATABASE_USER, and DATABASE_NAME, and the proxy mints a token at
startup, assembles the connection URL around it, and refreshes it in the
background for as long as the process runs. That is the same shape
IAM_TOKEN_DB_AUTH already had for AWS RDS, so the two now share one code path:
a tagged union picks the minting strategy once, and the wrapper, the read
replica, and the refresh loop all read the choice off it instead of each
guessing from the environment. Setting both toggles is a startup error, in the
chart as well as in Python
The helm chart gets database.writer.useAzureEntraAuth and the matching reader
knob next to the existing useIAMAuth
Fixes#29661
Co-authored-by: David Balatoni <balcsida@gmail.com>
The chart shipped ghcr.io/berriai/litellm-database as its image default,
with a comment offering it as the "optimized image with database". That
distinction no longer exists: Dockerfile and docker/Dockerfile.database
differ only in comment text and one builder-stage apk package, and both
published images bake the prisma CLI, engines, schema.prisma and
prisma_migration.py into /app, so either one runs the migrations job.
Point the default at the canonical image the release notes, the cosign
verification instructions and the chart's own README already name, and
update the chart's tests and README so nothing still refers to the
legacy repository.
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.
Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
Two small pod-spec passthroughs the componentized chart was missing, both
additive and empty by default so existing renders are unchanged:
- gateway/backend/ui deployments gain a `startupProbe` knob (same
`{{- with }}` toYaml pattern as liveness/readiness), to gate liveness during
a slow cold start without a kill loop.
- gateway/backend/ui HPAs gain an `hpa.behavior` passthrough rendered verbatim
under spec.behavior (scaleUp/scaleDown policies + stabilization windows).
Tests: extend probe_tests.yaml (startupProbe absent by default / renders
verbatim) and add hpa_behavior_tests.yaml. Full chart suite: 76 tests pass.
Signed-off-by: Louis Vauterin <louis.vauterin@doctolib.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Both charts run schema migrations from a Job that is a pre-install and
pre-upgrade hook, and neither set activeDeadlineSeconds. A migration that
blocks on the database therefore never fails: backoffLimit is not reached
because the pod never terminates, so the Job stays active indefinitely and
the release waits on the hook forever. `helm upgrade` and any GitOps
controller driving it stop reconciling the whole chart until someone
deletes the Job by hand, which means unrelated changes to the gateway, the
backend and the UI silently stop shipping.
Give the field a 1800s default, guarded by `with` so setting it to null
restores the old unbounded behaviour. A migration that has exhausted its
retries is not going to succeed on the next one, so failing is strictly
better than hanging: a failed sync is visible and retryable, a hung one is
neither.
Chart.yaml is deliberately untouched. Recent template-only changes to
litellm-helm did not bump it either.
The template rendered affinity and tolerations but never nodeSelector, so a
values file that pinned the chart to a node pool got the gateway and every
subchart placed correctly while the migration Job silently fell through to
whatever the cluster's default pool was.
That is worse than an outright failure. On EKS Auto Mode the default pool hands
out 3 GiB nodes and the migration container needs roughly 3.6 GB, so the Job
was OOM-killed on a pool it was never meant to run on, while the values file
that would have placed it on a large enough node looked correct.
The new test fails against the old template with "unknown path
spec.template.spec.nodeSelector".
The litellm-helm values file shipped the stock helm create boilerplate for
resources: an empty default plus a commented 100m/128Mi example it invites
operators to uncomment. 128Mi is roughly 32x below what the proxy needs at
DB-connected steady state, and it was the only sizing figure this chart ever
showed, so operators who followed it were sized for OOMKills.
Point the example at the documented 1 CPU / 4Gi per worker instead, link the
production sizing guidance, and note why the default stays unset. The
migration job's commented block carried the same trap with a 100m/100Mi
example; drop those numbers rather than substitute proxy figures that do not
transfer to a job that migrates and exits.
The defaults are deliberately left at {} so no existing release changes shape
on upgrade; rendered output is unchanged.
The gateway and backend probes omitted timeoutSeconds, so kubelet applied its
1s default. Both containers run a single uvicorn worker (the gateway defaults
NUM_WORKERS to 1; the backend passes no --workers at all), so each pod is one
asyncio event loop and its per-request latency under closed-loop saturation
rises by queueing (~57-62ms serial vs ~6s at 100 concurrent users against one
replica). Both /health/readiness and /health/liveliness then time out on the
stage cluster while the pod is serving traffic correctly, which exposes the
deployment to losing a healthy pod from its load balancer during a burst and
to restarting a merely busy one.
Readiness now gets timeoutSeconds 10, equal to periodSeconds and above the
measured saturated latency, and keeps failureThreshold 3. kubelet drives each
probe from a time.Ticker of periodSeconds rather than sleeping between
attempts, and coalesces ticks that arrive mid-probe, so the interval between
probe starts is max(periodSeconds, probeDuration) and not their sum. Keeping
timeoutSeconds <= periodSeconds is what holds that interval at 10s, so three
consecutive failures still evict a genuinely wedged pod in ~30s.
Liveness gets the same timeout plus failureThreshold 6: /health/liveliness is
an in-memory flag check, so a timeout there only ever means event-loop
starvation, which a restart makes worse, and it now needs ~90s of sustained
unresponsiveness to fire.
The ui container keeps the default. It is nginx serving a Next.js static
export, so / is a file off disk with no application runtime that could queue
behind saturated work, and nothing measured suggests it needs more than 1s.
The componentized chart exposed no pod-hardening surface, so values that
operators of hardened clusters routinely set (podSecurityContext,
securityContext, extraContainers, podLabels, lifecycle,
terminationGracePeriodSeconds) rendered nothing at all. Helm does not error
on unknown values, so the deploy went green with none of the hardening
applied.
Adds those six knobs to gateway, backend, and ui, plus volumes,
volumeMounts, podLabels, podSecurityContext, and securityContext on the
migrations Job.
Also fixes a first-install failure: the migrations Job is a
pre-install/pre-upgrade hook, so borrowing the backend ServiceAccount name
while the chart creates that account references an account that does not
exist yet, and the Job pod is rejected as forbidden. The Job now resolves
its own name through migrationJob.serviceAccountName, falling back to the
namespace default account when the chart creates the backend one and
keeping today's shared name otherwise.
The litellm-helm proxy Deployment renders a pod-level securityContext from
.Values.podSecurityContext, but the Prisma migration Job rendered only the
container-level securityContext from .Values.securityContext. Clusters that
enforce pod-level admission policies (OPA Gatekeeper K8sPSPAllowedUsers, or a
PSP-style fsGroup MustRunAs rule) therefore admitted the Deployment and denied
the Job, which blocks install and upgrade because the Job runs as an ArgoCD
PreSync or Helm pre-install/pre-upgrade hook.
The Job now renders the same pod-level securityContext the Deployment does.
Charts that leave podSecurityContext unset render an empty securityContext,
matching what the Deployment already emitted, so default installs are unchanged.
Resolves LIT-4928
Bitnami retired the versioned tags under docker.io/bitnami and republished
the archived builds under docker.io/bitnamilegacy, so every install and
upgrade of the chart with the bundled database fails to pull
docker.io/bitnami/postgresql:16.2.0-debian-12-r6. Repoint the subchart
images at the bitnamilegacy copies of the exact builds those subchart
versions shipped with, so the on-disk data directory layout is unchanged
for existing installs.
Pin the subchart dependency ranges to the versions already in Chart.lock.
The current bitnami postgresql chart defaults to `tag: latest`, which is
PostgreSQL 18 today, so an open-ended range turns a dependency refresh
into a major-version jump on an existing volume.
Refuse to render when postgresql.image.tag is empty or `latest` while the
bundled database is deployed. Starting a different PostgreSQL major
against an existing data directory leaves the server unable to boot with
no in-place way back, which is how the reported install lost its data.
Resolves LIT-4708
* fix(gateway): route /a2a through the gateway component
A2A message-send runs the completion bridge, an outbound LLM call, but the
ingress only listed /v1/a2a so the serving routes at /a2a/{agent_id} fell to
the backend catch-all. Backend pods hold no provider credentials, so every
invocation died with a missing-provider-key auth error while the same call
succeeds on the gateway fleet. Adds /a2a to the ingress gateway prefixes and
the gateway route allowlist, plus a parity test so an ingress prefix that the
gateway trims can never reappear
* revert(test): drop the allowlist parity tests
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(helm): add per-component PodDisruptionBudget and topologySpreadConstraints to componentized chart
The componentized chart (helm/litellm) had no PodDisruptionBudget template
for the gateway, backend, or ui, so voluntary disruptions (node drains,
Karpenter consolidation) could evict every replica of a component at once.
The legacy chart shipped one out of the box. Deployments also had no way to
configure topologySpreadConstraints, blocking HA spread across AZs.
Adds a shared litellm.pdb helper rendered per component, gated on
<component>.pdb.enabled with minAvailable/maxUnavailable (minAvailable wins,
fallback maxUnavailable: 1), selectors matching each component's
selectorLabels. Adds <component>.topologySpreadConstraints rendered into
each Deployment pod spec. PDBs default to disabled since the default
hpa.minReplicas of 1 with minAvailable: 1 would block drains entirely.
Resolves LIT-4452
* fix(helm): honor explicit 0 in pdb minAvailable/maxUnavailable
A Go-template truthy check treated an explicit 0 (forbid all voluntary
disruptions via maxUnavailable: 0) as unset and silently replaced it with
the fallback maxUnavailable: 1, weakening the configured protection. Treat
a value as set when it is non-nil and non-empty-string instead.
* feat(proxy): push-based OTLP billable-request metering for enterprise deployments
Adds opt-in, license-gated metering that counts 2xx HTTP requests to LLM
inference, MCP, and A2A endpoints and exports them over mutual TLS to a global
OpenTelemetry Collector for request-based billing.
A pure ASGI middleware (BillableRequestMetricsMiddleware) classifies each
request by route and records one count per 2xx response via an injected
recorder. The recorder (BillingMetricsRecorder) owns a dedicated OTEL meter
provider and an OTLP/gRPC exporter authenticated with client certificates, kept
isolated from the global meter provider so a customer's own OTEL metrics are
untouched. The recorder is built only when a valid LITELLM_LICENSE is present
and the cert material is configured; otherwise the middleware is a transparent
pass-through.
Deployment identity rides on the mTLS client certificate rather than the
payload, so the secret license key is never sent as an attribute or header; only
the license org id travels as a resource attribute for cross-checking.
Resolves LIT-4089
* fix(proxy): align billable-request metering with the global collector
- switch the exporter to OTLP/HTTP with a TLS client certificate. The
collector front end terminates mutual TLS and validates the client cert
against our CA; server verification uses the system trust store, so the
CA env var is now an optional override for private collectors
- resolve the metrics recorder on the first request via a factory instead
of at import time, so deployments that provide the license and cert env
vars through the YAML config's environment_variables export correctly
- close the metering bypass: classify /images/edits, /images/variations,
/v1/messages, /v1/videos, video remix, /v1/ocr and Gemini generateContent
as billable, and gate LLM routes to POST so GET reads (list videos, fetch
a response) do not bill. Verified live: the collector count matches the
UI usage page successful_requests exactly, with failures excluded on both
sides
* fix(proxy): wrap enterprise billing import in try-except per code-quality gate
The check_unsafe_enterprise_import gate requires every import from an
enterprise-pathed module to be guarded. Annotate the factory with the
middleware's BillingRecorder protocol so no enterprise type import is
needed at type-check time
* chore: satisfy strict lint gates in billing modules
- builtin generics per UP006 (dict/tuple instead of typing.Dict/Tuple)
- noqa the deliberate blind catch that keeps metering from breaking startup
- sort proxy_server import blocks split by the guarded enterprise import
* fix(proxy): bill provider passthrough, search, and rag routes
Route-inventory audit against LiteLLMRoutes.llm_api_routes found more
SpendLogs-producing surfaces the classifier missed: provider passthrough
(/bedrock, /vertex-ai, /cohere and the rest of mapped_pass_through_routes),
/v1/search and vector-store search, and the rag ingest/query routes. All are
counted by the dashboard usage page, so missing them undercounts billing.
The passthrough prefix list is read from LiteLLMRoutes so new providers are
picked up without touching this module. /langfuse is excluded: it forwards
observability traffic and writes no SpendLogs row. Known limitation recorded
in the PR: /v1/realtime is a websocket flow the HTTP middleware does not see
* fix(proxy): bill MCP and A2A requests by protocol transport routes only
The billable-request classifier matched the whole /v1/mcp prefix, so
management and discovery reads such as GET /v1/mcp/tools and GET
/v1/mcp/server counted as billable MCP requests, while real MCP tool
calls on the /{server}/mcp and /toolset/{name}/mcp aliases were missed
because their route handlers rewrite the ASGI scope only after this
middleware has already classified the original path. Classify MCP by the
concrete transport surface (the /mcp streamable-HTTP and SSE sub-app plus
the single-segment server and toolset aliases) and exclude the /v1/mcp
management API. Apply the same shape to A2A, which had the identical
issue: only the /message/send invoke route bills, not /v1/a2a/discover or
the .well-known agent-card reads.
* fix(proxy): harden billable-request classification and recorder lifecycle
Exact-match Anthropic /v1/messages so OpenAI Assistants thread-message
routes no longer bill, add Google Interactions create routes, guard
recorder.record() so a broken exporter can never fail a served request,
lock lazy recorder resolution against concurrent first requests, and
disable metering on empty-string env config instead of accepting a
blank endpoint
* chore(ui): regenerate eslint metrics after staging merge
* docs(proxy): state the lower-bound billing contract in middleware comments
* fix(proxy): bill mcp-rest tool calls and bare a2a agent invokes
POST /mcp-rest/tools/call executes a tool and fires the same MCP spend
logging as the /mcp transport, and POST /a2a/{agent_id} is the JSON-RPC
invoke route whose method (message/send or message/stream) travels in
the body; both returned 2xx without being recorded
* fix(proxy): flush billable-request counts on proxy shutdown
PeriodicExportingMetricReader buffers up to one export interval of
counts; without a final flush every restart silently dropped them. The
factory registers the recorder it builds and proxy_shutdown_event pops
and flushes it, bounded by a 5s timeout so a dead collector cannot
stall shutdown
* fix(proxy): stop billing bare a2a task RPCs and close the shutdown race
POST /a2a/{agent_id} multiplexes JSON-RPC methods off the request body. Only
message/send and message/stream write a SpendLogs row; tasks/get, tasks/cancel
and the pushNotificationConfig RPCs are forwarded upstream and write none.
Classifying the bare path as billable counted those task RPCs and pushed the
metric above the dashboard's successful-request count. Since a path-only
classifier cannot read the body, the bare route no longer bills; the explicit
/message/send routes still do. Missing a bare-path invoke undercounts, which is
the only direction this metric is allowed to drift. The /mcp transport keeps
billing every method because its list path logs a SpendLogs row too.
The billing middleware also sat outside InFlightRequestsMiddleware, and it
records after the inner app returns. A request could therefore be counted as
drained while its record() had not yet run, letting proxy_shutdown_event flush
and stop the exporter underneath it. Registering it before the in-flight
tracker nests it inside, so wait_for_drain covers the record
* test(proxy): stub the OTLP exporter in the recorder-build test
test_premium_with_full_config_builds_recorder built a real MeterProvider, so
the shutdown flush resolved collector.example and opened a TLS connection from
a unit test. The exporter is now stubbed, and a getaddrinfo spy asserts nothing
resolves the collector host so the stub cannot be quietly dropped later
* fix(helm): truncate the helm.sh/chart label to 63 bytes
Kubernetes caps a label value at 63 bytes and .Chart.Version is unbounded. CI
publishes branch builds as 0.0.0-branch-<branch>-<sha>, so helm.sh/chart
rendered as a 64 byte value and the API server rejected every labeled resource
with "must be no more than 63 bytes", including the migrations Job. The
litellm-helm chart already guards this through a litellm.chart helper; this
adds the same helper here.
Swept the rest of the chart for label and name values built from unbounded
input. .Chart.Version appeared only in this label. The remaining candidates all
derive from .Release.Name, which helm itself caps at 53 characters, so they
cannot overflow; three of them are selector labels feeding immutable Deployment
matchLabels, where adding trunc would risk churn for no gain. They are left
alone deliberately.
Verified with a new helm-unittest suite, tests/chart_label_tests.yaml, which
overrides chart.version per test:
helm unittest -f 'tests/*.yaml' helm/litellm # 13 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
The truncation cases fail against the previous helper. Reproduced the original
overflow by rendering with the real branch version and measuring the label:
helm template rel helm/litellm -f helm/litellm/tests/values/required.yaml \
| grep helm.sh/chart # 64 bytes before, 63 after
* feat(proxy): accept inline PEM for the billing-metrics mTLS credentials
LITELLM_BILLING_METRICS_CLIENT_CERT, _CLIENT_KEY and _CA_CERT took a filesystem
path. ECS injects Secrets Manager values as environment content and cannot mount
them as files, so a licensed deployment there could not turn metering on.
Each variable now takes either a path or the PEM itself. Inline PEM, detected by
the "-----BEGIN" prefix, is written once when the recorder is built into a 0700
temp dir as a 0600 file, and the config points at that path. The OTLP exporter
still only ever sees paths. A write failure disables metering through the
existing failure-as-None path rather than raising, and path-valued variables are
passed through untouched, so nothing changes for deployments that mount files.
The mixed case works too: mount the CA, inject the client credentials
* feat(helm): add first-class billingMetrics values to the componentized chart
Turning enterprise billable-request metering on meant hand-rolling the env vars
and the cert volume through gateway.extraEnv and gateway.volumes. This adds a
top-level billingMetrics block, off by default, consumed only by the gateway
since that is the component serving billable traffic.
When enabled it renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths
and mounts secretName read-only at /etc/litellm/billing-mtls. caSecretName is
optional and only needed for private collectors whose server certificate is not
on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and
adds the CA env var. exportIntervalMs is passed through only when set.
Enabling without secretName or with an empty endpoint fails the render with a
named message rather than producing a gateway that silently never exports.
The generic gateway.volumes, gateway.volumeMounts and gateway.extraEnv paths are
untouched and still compose with this, so existing overlays keep working.
The chart has no values.schema.json and no README, so there is nothing further to
update. Verified with a new helm-unittest suite:
helm unittest -f 'tests/*.yaml' helm/litellm # 23 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
* feat(terraform): billing-metrics variables for the aws and gcp templates
* feat(helm): add billingMetrics values to the classic chart
The componentized chart just gained a first-class billingMetrics block; this
mirrors it in litellm-helm so enabling enterprise billable-request metering no
longer means hand-rolling the env vars and the cert volume through envVars and
volumes.
When enabled the proxy Deployment renders LITELLM_BILLING_METRICS_ENDPOINT plus
the two cert paths, and mounts secretName read-only at /etc/litellm/billing-mtls.
secretName defaults to litellm-billing-metrics-mtls, the conventional name, so
enabling the block is enough once that Secret exists. caSecretName is optional
and only needed for private collectors whose server certificate is not on the
public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the
CA env var. exportIntervalMs is passed through only when set.
The env entries render after envVars and extraEnvVars, so a user-supplied
LITELLM_BILLING_METRICS_ENDPOINT cannot silently redirect the export under
Kubernetes last-wins duplicate-env semantics; this is the same ordering the
migrations Job relies on for DISABLE_SCHEMA_UPDATE.
Enabling with an emptied secretName or endpoint fails the render with a named
message rather than producing a proxy that silently never exports.
The generic volumes, volumeMounts, envVars and extraEnvVars paths are untouched
and still compose with this, so existing overlays keep working. The chart has no
values.schema.json; README parameters and a setup section are updated.
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 68 passed (54 + 14 new)
helm lint helm/litellm-helm # 0 failed
* test(helm): pin that the migrations job never mounts the billing cert
The componentized chart's suite asserts the backend Deployment stays clear of the
billing wiring, since only the gateway serves billable traffic. The classic chart
has no backend, but it does have a second pod: the migrations Job, which renders
its own env from envVars and extraEnvVars. Nothing today wires the billing
include into it, and nothing stopped a future edit from doing so.
Asserts absence of the env, and that the Job grows no volumes or volumeMounts at
all. Both are notExists rather than notContains because the Job renders neither
key by default, so a notContains would fail on an unknown path instead of
checking the absence it looks like it is checking.
* fix(helm): meter the backend too, it serves the MCP transport
Scoping billingMetrics to the gateway was wrong. Applying each component's own
route allowlist to the proxy app shows the split is 75 billable routes on the
gateway and one on the backend: /{mcp_server_name}/mcp, the named-server MCP
transport, which writes a SpendLogs row on success. Metering only the gateway
would have silently dropped every MCP transport call from the counter, an
undercount proportional to a customer's MCP traffic.
The backend deployment now renders the same env and mounts the same read-only
cert secret. The migrations job still gets neither; it runs prisma and serves no
traffic, and a test pins that.
helm unittest -f 'tests/*.yaml' helm/litellm # 25 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
This also aligns the chart with the terraform templates, which inject the
credentials into both components.
* fix(proxy): never log billing credential values when they fail to resolve
Accepting inline PEM turned the cert env vars into secret-bearing values, but
the disable warning still echoed them. A value that is neither a readable path
nor `-----BEGIN`-prefixed PEM, for example a key with a preamble or a malformed
secret, fell through to the path branch and was written to the proxy logs
verbatim, exposing the client certificate or private key to anyone who can read
them.
The warning now names the offending environment variables and tells the operator
what a valid value looks like, without ever printing one
* Revert "fix(helm): truncate the helm.sh/chart label to 63 bytes"
This reverts commit 4f7f706a63.
Version hygiene belongs to the pipeline that mints chart versions, not to the
chart. The build workflow now caps the version slug so litellm-<version> fits
the 63 byte label budget, which removes the overflow at the source rather than
silently truncating a value operators use to identify the build.
Drops the litellm.chart helper, restores the direct helm.sh/chart printf, and
removes tests/chart_label_tests.yaml. Both chart suites stay green:
helm unittest -f 'tests/*.yaml' helm/litellm # 20 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
* feat(helm): default billingMetrics.secretName to the conventional name
The componentized chart required an explicit secretName while the classic chart
defaults to litellm-billing-metrics-mtls. Both now default to it, so the common
path is to create that Secret with tls.crt and tls.key and set enabled: true.
The required() guard stays, and with a default it now only fires when someone
explicitly blanks the override, which the tests pin from both sides
* feat(proxy): log once when billing metrics are actually enabled
build_billing_metrics_recorder returned None silently when the deployment was
not licensed, while every other disable path logged a warning. An operator
reading logs could not tell "metering active" from "metering off because this
component never saw the license", and a component can carry the cert mount and
the billing env and still meter nothing. That is the undercount direction the
metric is not allowed to drift in.
A successful build now emits one info line naming the collector endpoint and the
export interval; neither the certificate contents nor the license appear. The
unlicensed path logs at debug rather than warning, because unlicensed is the
common case and a warning there would be noise on every OSS proxy
* fix(terraform): fail the plan on a partial billing-metrics config
Each PEM secret is created only when its own variable is non-empty, so setting
billing_metrics_endpoint with a certificate but no key applied cleanly and left
the proxy logging "missing config" and never exporting. Silent non-export is the
undercount direction this metric must not drift in, and every other surface
fails fast on a half-configured metering block.
Both templates now carry a lifecycle precondition requiring the client
certificate and its key together whenever the endpoint is set. It lives on the
gateway task definition (aws) and the gateway Cloud Run service (gcp) rather
than on the secret resources, because those are themselves count-gated on the
PEM being present and would never evaluate in the failing case. Cross-variable
`validation` blocks would need terraform 1.9; versions.tf pins >= 1.6, and
preconditions work there.
ca_cert_pem stays optional, so an empty value still falls back to the system
trust store.
endpoint cert key result
"" any any metering off, no secrets created
set set set metering on
set missing either plan fails
Verified each row with `terraform console` against the condition, and reran
`terraform fmt -check` and `terraform validate` in both directories
* docs(terraform): record why the billing guard sits on the gateway resource
The precondition cannot live on the cert secret, which is count-gated on
the cert itself and so has zero instances in exactly the case the guard
must catch. That makes the guard's correctness depend on this resource
staying unconditional, which nothing else records and no test enforces
* fix(terraform): guard the backend against a partial billing config too
The precondition only sat on the gateway, but the backend receives the billing
endpoint as well, because it serves the named-server MCP transport and meters
it. A targeted apply of just the backend task or service would therefore skip
the guard entirely and provision a component holding a billing endpoint with no
credentials to use it, which is the silent never-export failure the guard exists
to prevent.
Both templates now carry the same precondition on the backend resource. The
condition and truth table are unchanged; ca_cert_pem stays optional.
terraform fmt -check and terraform validate clean in both directories
* docs(team): document mcp_rpm_limit in update_team docstring
* chore(ui): regenerate schema.d.ts for update_team docstring change
Render a general_settings.coordination_redis block into the litellm-helm
proxy config when the bundled Redis is enabled, gated on a new
redis.coordination.enabled value and skipped when the user already
supplies their own block. Sentinel deployments render sentinel_nodes and
service_name rather than a host/port pair.
Also fixes litellm.redis.serviceName, which gated its sentinel branch on
standalone architecture. The bundled Redis subchart only serves sentinel
in replication mode, and renders no master Service there, so REDIS_HOST
pointed at a Service that never existed for every sentinel user.
Documents the coordination redis in the componentized chart and in the
terraform modules, whose existing REDIS_* exports now feed it directly.
Adds helm-unittest coverage for both charts' redis wiring, which had none
The componentized chart at helm/litellm had no way to mount extra
volumes into its deployments, so custom callback or SSO handler code
could not be mounted the way the docs describe for the monolithic
chart. Adds per-component volumes and volumeMounts values for gateway,
backend, and ui, merged with the existing gateway-config volume, plus
a helm-unittest suite for the chart wired into the helm unit test
workflow
Resolves LIT-4209
* change deployment configs to include a litellm.cache for litellm-backend pod mirroring litellm-gateway pod
* omit backend annotations block when config and podAnnotations are both empty
* reuse gateway config/configmap for backend instead of separate backend config
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Tin Chi Lo <tin@Tins-MBP.localdomain>
Co-authored-by: Tin Chi Lo <tin@Tins-MacBook-Pro.local>
* feat(helm): split per-component ServiceAccounts for gateway, backend, and UI
Replace the single shared serviceAccount with three separate serviceAccounts
(gateway, backend, ui) so operators can attach different IRSA / Workload
Identity annotations per component without granting data-plane credentials
to the UI pod.
Key changes:
- values.yaml: rename serviceAccount → serviceAccounts with gateway/backend/ui
sub-keys; UI defaults to automount: false
- _helpers.tpl: replace litellm.serviceAccountName with three component-scoped
helpers (litellm.gateway/backend/ui.serviceAccountName)
- serviceaccount.yaml: create up to three separate ServiceAccount objects with
component labels and per-SA automountServiceAccountToken
- gateway/backend deployments: use their respective SA helpers
- ui deployment: use litellm.ui.serviceAccountName + explicit
automountServiceAccountToken: false on the pod spec so the projected token
is absent even when the SA itself allows it
- migrations-job: share the backend SA (both need DB write access)
Resolves LIT-3171
https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF
* fix(helm): enforce automountServiceAccountToken on all pod specs; fix leading --- in serviceaccount.yaml
- gateway/backend deployments: add explicit automountServiceAccountToken on
the pod spec so serviceAccounts.*.automount is honoured regardless of
whether the SA is chart-created or operator-supplied (previously the flag
only took effect on the SA object when create: true, creating an asymmetry
with the UI which already enforced it at pod-spec level)
- serviceaccount.yaml: use a $prev sentinel to emit --- only between
documents, preventing a leading --- when gateway SA is skipped but
backend or ui SA is created (avoids lint/GitOps warnings from strict
YAML parsers and tools like ArgoCD)
https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF
---------
Co-authored-by: Claude <noreply@anthropic.com>
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
