The componentized chart required an explicit secretName while the classic chart
defaults to litellm-billing-metrics-mtls. Both now default to it, so the common
path is to create that Secret with tls.crt and tls.key and set enabled: true.
The required() guard stays, and with a default it now only fires when someone
explicitly blanks the override, which the tests pin from both sides
This reverts commit 4f7f706a63.
Version hygiene belongs to the pipeline that mints chart versions, not to the
chart. The build workflow now caps the version slug so litellm-<version> fits
the 63 byte label budget, which removes the overflow at the source rather than
silently truncating a value operators use to identify the build.
Drops the litellm.chart helper, restores the direct helm.sh/chart printf, and
removes tests/chart_label_tests.yaml. Both chart suites stay green:
helm unittest -f 'tests/*.yaml' helm/litellm # 20 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
Scoping billingMetrics to the gateway was wrong. Applying each component's own
route allowlist to the proxy app shows the split is 75 billable routes on the
gateway and one on the backend: /{mcp_server_name}/mcp, the named-server MCP
transport, which writes a SpendLogs row on success. Metering only the gateway
would have silently dropped every MCP transport call from the counter, an
undercount proportional to a customer's MCP traffic.
The backend deployment now renders the same env and mounts the same read-only
cert secret. The migrations job still gets neither; it runs prisma and serves no
traffic, and a test pins that.
helm unittest -f 'tests/*.yaml' helm/litellm # 25 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
This also aligns the chart with the terraform templates, which inject the
credentials into both components.
The componentized chart's suite asserts the backend Deployment stays clear of the
billing wiring, since only the gateway serves billable traffic. The classic chart
has no backend, but it does have a second pod: the migrations Job, which renders
its own env from envVars and extraEnvVars. Nothing today wires the billing
include into it, and nothing stopped a future edit from doing so.
Asserts absence of the env, and that the Job grows no volumes or volumeMounts at
all. Both are notExists rather than notContains because the Job renders neither
key by default, so a notContains would fail on an unknown path instead of
checking the absence it looks like it is checking.
The componentized chart just gained a first-class billingMetrics block; this
mirrors it in litellm-helm so enabling enterprise billable-request metering no
longer means hand-rolling the env vars and the cert volume through envVars and
volumes.
When enabled the proxy Deployment renders LITELLM_BILLING_METRICS_ENDPOINT plus
the two cert paths, and mounts secretName read-only at /etc/litellm/billing-mtls.
secretName defaults to litellm-billing-metrics-mtls, the conventional name, so
enabling the block is enough once that Secret exists. caSecretName is optional
and only needed for private collectors whose server certificate is not on the
public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the
CA env var. exportIntervalMs is passed through only when set.
The env entries render after envVars and extraEnvVars, so a user-supplied
LITELLM_BILLING_METRICS_ENDPOINT cannot silently redirect the export under
Kubernetes last-wins duplicate-env semantics; this is the same ordering the
migrations Job relies on for DISABLE_SCHEMA_UPDATE.
Enabling with an emptied secretName or endpoint fails the render with a named
message rather than producing a proxy that silently never exports.
The generic volumes, volumeMounts, envVars and extraEnvVars paths are untouched
and still compose with this, so existing overlays keep working. The chart has no
values.schema.json; README parameters and a setup section are updated.
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 68 passed (54 + 14 new)
helm lint helm/litellm-helm # 0 failed
Turning enterprise billable-request metering on meant hand-rolling the env vars
and the cert volume through gateway.extraEnv and gateway.volumes. This adds a
top-level billingMetrics block, off by default, consumed only by the gateway
since that is the component serving billable traffic.
When enabled it renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths
and mounts secretName read-only at /etc/litellm/billing-mtls. caSecretName is
optional and only needed for private collectors whose server certificate is not
on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and
adds the CA env var. exportIntervalMs is passed through only when set.
Enabling without secretName or with an empty endpoint fails the render with a
named message rather than producing a gateway that silently never exports.
The generic gateway.volumes, gateway.volumeMounts and gateway.extraEnv paths are
untouched and still compose with this, so existing overlays keep working.
The chart has no values.schema.json and no README, so there is nothing further to
update. Verified with a new helm-unittest suite:
helm unittest -f 'tests/*.yaml' helm/litellm # 23 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
Kubernetes caps a label value at 63 bytes and .Chart.Version is unbounded. CI
publishes branch builds as 0.0.0-branch-<branch>-<sha>, so helm.sh/chart
rendered as a 64 byte value and the API server rejected every labeled resource
with "must be no more than 63 bytes", including the migrations Job. The
litellm-helm chart already guards this through a litellm.chart helper; this
adds the same helper here.
Swept the rest of the chart for label and name values built from unbounded
input. .Chart.Version appeared only in this label. The remaining candidates all
derive from .Release.Name, which helm itself caps at 53 characters, so they
cannot overflow; three of them are selector labels feeding immutable Deployment
matchLabels, where adding trunc would risk churn for no gain. They are left
alone deliberately.
Verified with a new helm-unittest suite, tests/chart_label_tests.yaml, which
overrides chart.version per test:
helm unittest -f 'tests/*.yaml' helm/litellm # 13 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
The truncation cases fail against the previous helper. Reproduced the original
overflow by rendering with the real branch version and measuring the label:
helm template rel helm/litellm -f helm/litellm/tests/values/required.yaml \
| grep helm.sh/chart # 64 bytes before, 63 after
The componentized chart at helm/litellm had no way to mount extra
volumes into its deployments, so custom callback or SSO handler code
could not be mounted the way the docs describe for the monolithic
chart. Adds per-component volumes and volumeMounts values for gateway,
backend, and ui, merged with the existing gateway-config volume, plus
a helm-unittest suite for the chart wired into the helm unit test
workflow
Resolves LIT-4209
* change deployment configs to include a litellm.cache for litellm-backend pod mirroring litellm-gateway pod
* omit backend annotations block when config and podAnnotations are both empty
* reuse gateway config/configmap for backend instead of separate backend config
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Tin Chi Lo <tin@Tins-MBP.localdomain>
Co-authored-by: Tin Chi Lo <tin@Tins-MacBook-Pro.local>
* feat(helm): split per-component ServiceAccounts for gateway, backend, and UI
Replace the single shared serviceAccount with three separate serviceAccounts
(gateway, backend, ui) so operators can attach different IRSA / Workload
Identity annotations per component without granting data-plane credentials
to the UI pod.
Key changes:
- values.yaml: rename serviceAccount → serviceAccounts with gateway/backend/ui
sub-keys; UI defaults to automount: false
- _helpers.tpl: replace litellm.serviceAccountName with three component-scoped
helpers (litellm.gateway/backend/ui.serviceAccountName)
- serviceaccount.yaml: create up to three separate ServiceAccount objects with
component labels and per-SA automountServiceAccountToken
- gateway/backend deployments: use their respective SA helpers
- ui deployment: use litellm.ui.serviceAccountName + explicit
automountServiceAccountToken: false on the pod spec so the projected token
is absent even when the SA itself allows it
- migrations-job: share the backend SA (both need DB write access)
Resolves LIT-3171
https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF
* fix(helm): enforce automountServiceAccountToken on all pod specs; fix leading --- in serviceaccount.yaml
- gateway/backend deployments: add explicit automountServiceAccountToken on
the pod spec so serviceAccounts.*.automount is honoured regardless of
whether the SA is chart-created or operator-supplied (previously the flag
only took effect on the SA object when create: true, creating an asymmetry
with the UI which already enforced it at pod-spec level)
- serviceaccount.yaml: use a $prev sentinel to emit --- only between
documents, preventing a leading --- when gateway SA is skipped but
backend or ui SA is created (avoids lint/GitOps warnings from strict
YAML parsers and tools like ArgoCD)
https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF
---------
Co-authored-by: Claude <noreply@anthropic.com>
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
