* feat(lens): isolate trace storage and investigation in a Rust service
* fix(lens): include Rust sources in the image build context
* feat(lens): wire service setup, scoped delivery receipts and lease attempts
* fix(lens): complete service routing and reject stale investigation results
* fix(lens): retry key propagation and validate isolated Compose setup
* fix(lens): seed through isolated ingestion and preserve upstream queue fixes
* chore: sync schema.prisma copies from root
* fix(lens): bind nullable due timestamps as text for Prisma
* chore(ui): remove stale lint suppressions
* fix(lens): address CI failures and review findings
* refactor(lens): remove retired Python worker and run evaluations in Rust
* fix(lens): reuse control connections and satisfy review checks
* test(lens): install and upgrade both Helm charts on Kubernetes
* test(lens): run connection reuse coverage as an integration test
* fix(ui): upgrade Next.js to 16.3.8 security release
* fix(lens): fence stale attempts and preserve reviewed evidence
* Revert "fix(ui): upgrade Next.js to 16.3.8 security release"
This reverts commit 2f79a51b25.
* fix(lens): stop failed investigations and stream history excerpts
* test(lens): cover model tool and result contracts
* test(lens): fix retired routes and reuse installation build artifacts
* test(lens): use portable grep in Helm installation smoke
* test(lens): wait for migrations before forwarding Helm services
* fix(lens): keep failed evidence reads retryable
* fix(lens): preserve sandbox output during process exit
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(lens): pin worker dependencies and support approved image digests
* fix(lens): include locked dependencies and release identity in build context
* fix(lens): select the dev worker package for SHA-tagged charts
`migrationJob.retries` and `migrationJob.disableSchemaUpdate` are declared
in values.yaml but referenced by no template, no test and no README row.
Setting either changes nothing about the rendered Job.
`disableSchemaUpdate` is the misleading one: its comment promises "the job
will exit with code 0", but the Job hardcodes DISABLE_SCHEMA_UPDATE=false
and renders it after envVars/extraEnvVars precisely so nothing can turn the
migration off — that ordering is what #12809 fixed. An operator who reads
values.yaml, sets the flag and watches migrations run anyway has no way to
tell the knob is inert. `migrationJob.enabled: false` is the supported way
to skip the Job, and the componentized chart in helm/litellm already ships
a migrationJob block with neither key.
`retries` is simply dead: Jobs retry through `backoffLimit`, which the
chart does render.
Removing values keys is backward compatible — Helm ignores user values that
no template consumes, so existing releases setting either key keep working.
Adds a test pinning the override: with envVars.DISABLE_SCHEMA_UPDATE="true"
the Job's last env entry is still DISABLE_SCHEMA_UPDATE=false, so the
last-wins ordering cannot regress and the key cannot quietly come back as a
chart value. Verified by mutation: flipping the hardcoded value and moving
the entry above the envVars loop each fail the suite.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(gateway): expose /api/event_logging/batch on the gateway allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gateway): route /api/event_logging to gateway pods in helm and terraform
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stop max_budget leaking between proxy_server and native_compaction tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds /azure_speech/{endpoint:path}, an authenticated pass-through for the Azure AI Speech REST APIs: short-audio recognition on <region>.stt.speech.microsoft.com and batch transcription on <region>.api.cognitive.microsoft.com. The proxy resolves the subscription key through PassthroughEndpointRouter (AZURE_SPEECH_API_KEY or an Admin UI credential), picks the host from AZURE_SPEECH_REGION or AZURE_SPEECH_API_BASE, injects Ocp-Apim-Subscription-Key, strips the caller's Authorization and subscription-key headers, forwards the raw audio body byte for byte, and records a zero-cost SpendLogs row tagged azure_speech since the price map has no Azure Speech STT entry
Resolves LIT-7939
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): carry DATABASE_SSLMODE/DATABASE_SSLROOTCERT into the assembled writer and reader URLs
The componentized gateway supervisor starts the in-container PgBouncer from the
DATABASE_URL assembled out of the discrete DATABASE_* vars before config.yaml is
read, so an IAM URL had no way to request verified TLS: PgBouncer dialed the
server with server_tls_sslmode = prefer (no SNI, no verification) and public
RDS endpoints rejected the handshake. Two new env vars, exposed by the chart as
database.writer.sslMode / sslRootCert, are appended as libpq sslmode/sslrootcert
to every writer and reader URL the settings assemble (never to a pinned URL),
then translated for Prisma as before. Token refresh now also carries Prisma's
sslmode/sslcert/sslaccept over into the re-minted URL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): keep TLS params on the CLI password URL and the initial IAM reader mint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): treat DATABASE_SSLROOTCERT on its own as verify-full and cover collector and migrations TLS env
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(db): type the reader mint TLS test double and drop its mutable capture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The componentized chart rendered litellm.connectionPoolEnv into the gateway
container only, so with database.connectionPool.enabled and
gateway.collector.enabled the collector's Prisma client opened its own pool
straight to Postgres instead of going through the pod-local PgBouncer.
Render the same include in the collector container, drop the gateway.extraEnv
workaround from the collector test, and add enabled/disabled regression
assertions for the collector in both helm charts and the terraform aws and gcp
modules, which already pass the pool env to their collector containers. The
module READMEs note the IAM token-auth exception, where the collector keeps a
direct connection by design.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
On the release gate the e2e tests only see the nginx router, and the chart's
ingress sent /debug/memory/summary to the backend catch-all, so the RSS check
measured the backend pod instead of the gateway workers that serve the failing
requests. Render it as an Exact gateway path next to /test, name the host in the
summary response so workers behind one origin never collide on pid alone, and
key the harness readings by (origin, hostname, pid)
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar
py-spy on the gateway showed the post-response _PROXY_track_cost_callback,
spend-log and DBSpendUpdateWriter work running on the inference workers'
event loop, so a DB or Redis stall backed up the request path.
When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact
typed SpendEvent per success and hands it to a SpendEventProducer that ships
it over a unix socket (default) or loopback-only TCP to a sidecar started as
`python -m gateway.spend_worker`. The sidecar runs the unchanged
_ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url).
When the sidecar is unreachable, the buffer is full, or the gateway shuts
down with events still queued or in flight, the producer applies
LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The
sidecar half-closes producers on SIGTERM and drains, the producer treats
EOF as unavailable, and the gateway flushes buffered spend counters on
shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its
own process log.
Helm: both charts gain an opt-in spend-worker sidecar container sharing an
emptyDir socket dir, and the componentized chart's HPA uses a
ContainerResource CPU metric scoped to the gateway container so sidecar
CPU does not drive inference scaling.
* feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks
Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): retrieve a completed batch in the in-process spend path test
The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename the spend worker sidecar to collector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown
The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime
stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the
pod-local pgbouncer URL itself, replacing gateway/collector.py
The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it.
SpendEventProducer shields an in-progress fallback from the writer task cancellation so close()
no longer loses an event already handed to the in-process pipeline
Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs,
flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(terraform): drop the gcp job duplicated by the aws/gcp matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports
The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format pooled_database_url after the pgbouncer rebase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain
Cache hits updated the logging object after the early return, so the offloaded spend event carried
preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against
producer transports uvloop already closed so one dead connection cannot abort the drain
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): keep the gcp collector port off the metrics sidecar health port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): collector connects to Postgres directly under IAM or Entra token auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): share database connections across workers with an in-container pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): parse pgbouncer options iteratively to satisfy the recursion gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer with token db auth and retry failed pooler restarts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): build pgbouncer 1.25.2 from a pinned source archive and verify pooler replacements
The public Wolfi repository only carries pgbouncer 1.24.1-r3, which the image
scan rejects (CVE-2026-6664, CVE-2026-6665, CVE-2026-6666, CVE-2025-12819).
All three images now compile the checksummed 1.25.2 release in a builder stage.
The supervisor now waits for a replacement pooler to listen before treating it
as recovered, ends and retries one that never does, and takes the same lock for
stop() and spawn so no replacement can be started after shutdown began.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse to start pgbouncer on a loopback port another process already owns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pgbouncer ready only once its own unix socket answers, not any listener on the port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer older than 1.19, whose unix socket cannot vouch for the tcp port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(helm,terraform): expose the in-container pgbouncer pool for the componentized gateway
Add database.connectionPool to helm/litellm and gateway_connection_pool_* to
terraform/litellm/aws so the componentized gateway can receive the
LITELLM_PGBOUNCER_* env the classic image already honours. Both reject the
pool under IAM or Entra token auth at render/plan time: the pooler holds one
static database password for the life of the pod or task.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gateway): launch the componentized gateway image through a pgbouncer-aware supervisor (#40592)
The componentized gateway image started uvicorn directly, so the in-container
PgBouncer never ran for it: every worker opened its own Prisma pool to the
database. It also passed no keep-alive timeout, so behind a load balancer with
a 60s idle timeout uvicorn's 5s default closed idle connections first and the
balancer returned 502s on scale-out
gateway.launch assembles DATABASE_URL, starts PgBouncer once per pod when
LITELLM_PGBOUNCER_ENABLED is set, hands the workers the loopback URL and then
runs uvicorn on gateway.main:app with KEEPALIVE_TIMEOUT as --timeout-keep-alive.
The image builds PgBouncer 1.25.2 from a checksummed tarball, copies the
compiled Rust extension into the /app source tree it imports from (it was only
in site-packages, which PYTHONPATH=/app shadows) and asserts the native bridge
loads. The app user is added to stats_users so operators can read the PgBouncer
console with the application credentials
The supervisor returns the pooled URL instead of writing into the mapping it
was handed, a database user whose name PgBouncer would split into several
stats_users entries is refused before the config is written, and the launcher
tests drive main() with an injected serve callable
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(terraform): describe the gateway.launch pooler entrypoint in the aws module README
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run pgbouncer exit hooks only in the parent and copy the CA into the runtime dir
Gunicorn workers inherit the parent's atexit table, so a recycled worker (max_requests) stopped the shared pooler and removed its runtime dir, then hung in the inherited Popen lock. The hooks now no-op unless os.getpid() is the process that started PgBouncer
A verified TLS upstream named the operator's CA bundle directly, which is often a 0600 root-owned file that nobody (the user PgBouncer drops to) cannot read, so every server connection failed with "failed to load CA". The bundle is copied into the runtime dir next to the ini and chowned with it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(terraform): run the gateway through gateway.launch and add the gcp connection-pool variables
Cloud Run and ECS overrode the image command with uvicorn gateway.main:app, which skips the supervisor that starts the in-container PgBouncer, so LITELLM_PGBOUNCER_ENABLED was inert on both stacks. Both now exec python -m gateway.launch (under ddtrace-run when USE_DDTRACE is set), and the gcp module gains gateway_connection_pool_enabled / gateway_pool_max_db_connections / gateway_pool_max_client_conn wired to the gateway service only
The test_launch password_env fixture now restores DATABASE_URL even when it was unset: monkeypatch.delenv records nothing for an absent var, so main() left postgresql://...@db.internal in the xdist worker's environ and the key-rotation e2e test in the same proxy-infra shard stopped skipping and tried to reach db.internal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pgbouncer): keep channel_binding and gssencmode off the loopback URL
Prisma would demand TLS channel binding from a pooler that only speaks
plain TCP on 127.0.0.1. Also pass the request the marketplace test
started needing after #40518 landed on top of #40496
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(infra): scale gateway on per-pod RPM and TPM in Helm and Terraform
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): require the metrics server before rendering the gateway ServiceMonitor
The http port serves /metrics/ behind virtual-key auth, so a ServiceMonitor
pointed at it only collects 401s and the RPM/TPM HPA metrics never appear
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(infra): express gateway HPA, KEDA and ECS workload targets per second
Rename the per-pod request and token targets in both Helm charts and the
AWS module from per minute to per second, and shorten the recommended
Prometheus rate window to [1m] with no * 60 so the adapter and KEDA
signals are what the HPA compares against. ECS keeps CloudWatch's
60-second aggregation: the ALB target is 60x the per-second variable and
the token metric math divides the period Sum by 60 before dividing by
the running task count.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(deploy): expose SSE keepalive, pre-call checks and a metrics sidecar in Helm and Terraform
Typed reliability values on both Helm charts and the AWS/GCP Terraform
modules, a dedicated ClusterIP Service for the separate Prometheus port,
a /health route on the metrics server and dead-worker pruning so the
aggregate does not keep stale multiprocess samples.
Resolves LIT-7142
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(deploy): drop reliability config from Helm and Terraform, keep only the metrics sidecar
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover startup pruning of dead workers' live gauges and unsignalable pids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A dotted extraPaths entry kept its requested Exact or Prefix type under
ingress.controller=nginx, so the admission webhook rejected the render the
option exists to avoid, and the duplicate check compared the raw type against
the built-in paths' normalized one, letting a repeated /favicon.ico through.
Extra paths now go through the same controller normalization before both the
duplicate check and the render.
ingress-nginx's admission webhook (strict-validate-path-type, on by default
from v1.12.0 until v1.12.6 / v1.13.2 allowed dots again) rejects the chart's
/favicon.ico Exact and /eu.assemblyai Prefix rules, so helm install fails on
any cluster it fronts. A new ingress.controller value (alb, the default, or
nginx) renders dotted built-in paths as ImplementationSpecific under nginx,
which serves them as plain prefix locations, and drops the ALB-only /*.txt
wildcard rule there. The default render is unchanged
* fix(helm): scale the classic chart's HPA out at the documented 60 percent CPU
The litellm-helm chart shipped targetCPUUtilizationPercentage: 80, which is
unexamined helm create scaffold rather than a chosen number. It arrived packaged
with the stock minReplicas: 1, maxReplicas: 100, a commented-out
targetMemoryUtilizationPercentage: 80, and the boilerplate "such as Minikube"
comment, the same provenance as the 128Mi resource example this file just
corrected.
60 is the documented recommendation. The mechanism behind it is scale-up lag:
the chart's own startupProbe is failureThreshold: 30 times periodSeconds: 10, so
a replica can take up to 300 seconds to become ready, and a pod added at 80
percent utilization arrives minutes after saturation.
The memory target stays commented out on purpose. The prisma query engine's
resident memory is a high-water mark that ratchets to the pod's worst-ever write
and is never returned, so a memory-target HPA reads the largest write a pod ever
did rather than what it is doing now, and replicas ratchet up without scaling
back in.
hpa_tests.yaml carried its second suite after a YAML document separator, and
helm-unittest loads only the first document per file, so that suite never ran;
an assertion planted in it still passed. Fold it into the one live suite and add
coverage pinning the rendered CPU target, the absence of a memory metric by
default, and that overrides still take effect.
Bump the chart to 1.1.2, since rendered output changes for anyone running with
autoscaling enabled.
* fix(helm): bump litellm-helm to 1.1.3 after rebase onto 1.1.2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The generated masterkey Secret rendered a fresh randAlphaNum value on every
release, so any helm upgrade with masterkeySecretName and masterkey unset
rotated the master key and invalidated every client holding the old one.
Look up the existing Secret in the release namespace and reuse its value,
falling back to a random key only on first install.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The componentized chart's Ingress renders a fixed path set: the UI paths, a
hardcoded gateway prefix list mirroring gateway/routes/allowlist.py, and a `/`
Prefix catch-all to the backend. values.yaml exposes no path knob, so an
operator who needs a route the chart does not know about has to fork the
template.
That prefix list is a snapshot of the data plane at release time and drifts
from it. `/watsonx` is in GATEWAY_PATH_PREFIXES with no ingress rule and no
backend prefix, so `/watsonx/*` falls into the catch-all, reaches the backend,
and 404s. A provider passthrough prefix added upstream after the chart version
an operator runs has the same shape, and a custom
general_settings.pass_through_endpoints route has a path only the operator
knows, so the chart can never ship a rule for it at all.
ingress.extraPaths takes a list of {path, service, pathType} entries rendered
in addition to the built-in paths, never in place of them. They render after
every built-in path and before the backend catch-all, and an entry repeating a
path the chart already routes is rejected: duplicates within one rule are
resolved by position or by controller-specific tie breaking, so an entry at
`/` Prefix would swallow the whole backend management API rather than add to
it. `service` picks the component Service — gateway by default, or backend or
ui — so an operator does not have to reconstruct release-templated Service
names, and `pathType` defaults to Prefix. A non-mapping entry, an entry with
no path, an unknown service, an unknown pathType, and a duplicate each abort
the render naming the offending index rather than emitting an Ingress that
misroutes traffic.
The duplicate check needs the built-in paths as data, so the UI paths move
from six literal YAML blocks into a $uiPaths list rendered by one range, and
their explanatory comments move from the rendered manifest into template
comments. The Ingress object this produces is unchanged: the same 92 paths in
the same order with the same backends, verified by parsing both renders.
The target component still answers only the paths its own route allowlist
keeps, so this makes a route routable, not servable.
* fix(ui): boot the UI image as an arbitrary uid by anchoring nginx writes under /tmp
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): type the arbitrary-uid image test fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(helm): compose DATABASE_URL_READ_REPLICA from a reader host secret key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(helm): cover reader host composition and readReplicaUrlKey precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): suppress unused reader host env when readReplicaUrlKey is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): emit reader host only when readReplicaUrl composition is active
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Azure Database for PostgreSQL Flexible Server takes a Microsoft Entra ID access
token as the connection password, and those tokens last about an hour, so a
proxy pointed at one dies shortly after boot unless something keeps minting
fresh ones
Set AZURE_POSTGRESQL_AUTH=True (or pass --azure_postgresql_auth) alongside
DATABASE_HOST, DATABASE_USER, and DATABASE_NAME, and the proxy mints a token at
startup, assembles the connection URL around it, and refreshes it in the
background for as long as the process runs. That is the same shape
IAM_TOKEN_DB_AUTH already had for AWS RDS, so the two now share one code path:
a tagged union picks the minting strategy once, and the wrapper, the read
replica, and the refresh loop all read the choice off it instead of each
guessing from the environment. Setting both toggles is a startup error, in the
chart as well as in Python
The helm chart gets database.writer.useAzureEntraAuth and the matching reader
knob next to the existing useIAMAuth
Fixes#29661
Co-authored-by: David Balatoni <balcsida@gmail.com>
The chart shipped ghcr.io/berriai/litellm-database as its image default,
with a comment offering it as the "optimized image with database". That
distinction no longer exists: Dockerfile and docker/Dockerfile.database
differ only in comment text and one builder-stage apk package, and both
published images bake the prisma CLI, engines, schema.prisma and
prisma_migration.py into /app, so either one runs the migrations job.
Point the default at the canonical image the release notes, the cosign
verification instructions and the chart's own README already name, and
update the chart's tests and README so nothing still refers to the
legacy repository.
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.
Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
Two small pod-spec passthroughs the componentized chart was missing, both
additive and empty by default so existing renders are unchanged:
- gateway/backend/ui deployments gain a `startupProbe` knob (same
`{{- with }}` toYaml pattern as liveness/readiness), to gate liveness during
a slow cold start without a kill loop.
- gateway/backend/ui HPAs gain an `hpa.behavior` passthrough rendered verbatim
under spec.behavior (scaleUp/scaleDown policies + stabilization windows).
Tests: extend probe_tests.yaml (startupProbe absent by default / renders
verbatim) and add hpa_behavior_tests.yaml. Full chart suite: 76 tests pass.
Signed-off-by: Louis Vauterin <louis.vauterin@doctolib.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Both charts run schema migrations from a Job that is a pre-install and
pre-upgrade hook, and neither set activeDeadlineSeconds. A migration that
blocks on the database therefore never fails: backoffLimit is not reached
because the pod never terminates, so the Job stays active indefinitely and
the release waits on the hook forever. `helm upgrade` and any GitOps
controller driving it stop reconciling the whole chart until someone
deletes the Job by hand, which means unrelated changes to the gateway, the
backend and the UI silently stop shipping.
Give the field a 1800s default, guarded by `with` so setting it to null
restores the old unbounded behaviour. A migration that has exhausted its
retries is not going to succeed on the next one, so failing is strictly
better than hanging: a failed sync is visible and retryable, a hung one is
neither.
Chart.yaml is deliberately untouched. Recent template-only changes to
litellm-helm did not bump it either.
The template rendered affinity and tolerations but never nodeSelector, so a
values file that pinned the chart to a node pool got the gateway and every
subchart placed correctly while the migration Job silently fell through to
whatever the cluster's default pool was.
That is worse than an outright failure. On EKS Auto Mode the default pool hands
out 3 GiB nodes and the migration container needs roughly 3.6 GB, so the Job
was OOM-killed on a pool it was never meant to run on, while the values file
that would have placed it on a large enough node looked correct.
The new test fails against the old template with "unknown path
spec.template.spec.nodeSelector".
The litellm-helm values file shipped the stock helm create boilerplate for
resources: an empty default plus a commented 100m/128Mi example it invites
operators to uncomment. 128Mi is roughly 32x below what the proxy needs at
DB-connected steady state, and it was the only sizing figure this chart ever
showed, so operators who followed it were sized for OOMKills.
Point the example at the documented 1 CPU / 4Gi per worker instead, link the
production sizing guidance, and note why the default stays unset. The
migration job's commented block carried the same trap with a 100m/100Mi
example; drop those numbers rather than substitute proxy figures that do not
transfer to a job that migrates and exits.
The defaults are deliberately left at {} so no existing release changes shape
on upgrade; rendered output is unchanged.
The gateway and backend probes omitted timeoutSeconds, so kubelet applied its
1s default. Both containers run a single uvicorn worker (the gateway defaults
NUM_WORKERS to 1; the backend passes no --workers at all), so each pod is one
asyncio event loop and its per-request latency under closed-loop saturation
rises by queueing (~57-62ms serial vs ~6s at 100 concurrent users against one
replica). Both /health/readiness and /health/liveliness then time out on the
stage cluster while the pod is serving traffic correctly, which exposes the
deployment to losing a healthy pod from its load balancer during a burst and
to restarting a merely busy one.
Readiness now gets timeoutSeconds 10, equal to periodSeconds and above the
measured saturated latency, and keeps failureThreshold 3. kubelet drives each
probe from a time.Ticker of periodSeconds rather than sleeping between
attempts, and coalesces ticks that arrive mid-probe, so the interval between
probe starts is max(periodSeconds, probeDuration) and not their sum. Keeping
timeoutSeconds <= periodSeconds is what holds that interval at 10s, so three
consecutive failures still evict a genuinely wedged pod in ~30s.
Liveness gets the same timeout plus failureThreshold 6: /health/liveliness is
an in-memory flag check, so a timeout there only ever means event-loop
starvation, which a restart makes worse, and it now needs ~90s of sustained
unresponsiveness to fire.
The ui container keeps the default. It is nginx serving a Next.js static
export, so / is a file off disk with no application runtime that could queue
behind saturated work, and nothing measured suggests it needs more than 1s.
The componentized chart exposed no pod-hardening surface, so values that
operators of hardened clusters routinely set (podSecurityContext,
securityContext, extraContainers, podLabels, lifecycle,
terminationGracePeriodSeconds) rendered nothing at all. Helm does not error
on unknown values, so the deploy went green with none of the hardening
applied.
Adds those six knobs to gateway, backend, and ui, plus volumes,
volumeMounts, podLabels, podSecurityContext, and securityContext on the
migrations Job.
Also fixes a first-install failure: the migrations Job is a
pre-install/pre-upgrade hook, so borrowing the backend ServiceAccount name
while the chart creates that account references an account that does not
exist yet, and the Job pod is rejected as forbidden. The Job now resolves
its own name through migrationJob.serviceAccountName, falling back to the
namespace default account when the chart creates the backend one and
keeping today's shared name otherwise.
The litellm-helm proxy Deployment renders a pod-level securityContext from
.Values.podSecurityContext, but the Prisma migration Job rendered only the
container-level securityContext from .Values.securityContext. Clusters that
enforce pod-level admission policies (OPA Gatekeeper K8sPSPAllowedUsers, or a
PSP-style fsGroup MustRunAs rule) therefore admitted the Deployment and denied
the Job, which blocks install and upgrade because the Job runs as an ArgoCD
PreSync or Helm pre-install/pre-upgrade hook.
The Job now renders the same pod-level securityContext the Deployment does.
Charts that leave podSecurityContext unset render an empty securityContext,
matching what the Deployment already emitted, so default installs are unchanged.
Resolves LIT-4928
Bitnami retired the versioned tags under docker.io/bitnami and republished
the archived builds under docker.io/bitnamilegacy, so every install and
upgrade of the chart with the bundled database fails to pull
docker.io/bitnami/postgresql:16.2.0-debian-12-r6. Repoint the subchart
images at the bitnamilegacy copies of the exact builds those subchart
versions shipped with, so the on-disk data directory layout is unchanged
for existing installs.
Pin the subchart dependency ranges to the versions already in Chart.lock.
The current bitnami postgresql chart defaults to `tag: latest`, which is
PostgreSQL 18 today, so an open-ended range turns a dependency refresh
into a major-version jump on an existing volume.
Refuse to render when postgresql.image.tag is empty or `latest` while the
bundled database is deployed. Starting a different PostgreSQL major
against an existing data directory leaves the server unable to boot with
no in-place way back, which is how the reported install lost its data.
Resolves LIT-4708
* fix(gateway): route /a2a through the gateway component
A2A message-send runs the completion bridge, an outbound LLM call, but the
ingress only listed /v1/a2a so the serving routes at /a2a/{agent_id} fell to
the backend catch-all. Backend pods hold no provider credentials, so every
invocation died with a missing-provider-key auth error while the same call
succeeds on the gateway fleet. Adds /a2a to the ingress gateway prefixes and
the gateway route allowlist, plus a parity test so an ingress prefix that the
gateway trims can never reappear
* revert(test): drop the allowlist parity tests
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(helm): add per-component PodDisruptionBudget and topologySpreadConstraints to componentized chart
The componentized chart (helm/litellm) had no PodDisruptionBudget template
for the gateway, backend, or ui, so voluntary disruptions (node drains,
Karpenter consolidation) could evict every replica of a component at once.
The legacy chart shipped one out of the box. Deployments also had no way to
configure topologySpreadConstraints, blocking HA spread across AZs.
Adds a shared litellm.pdb helper rendered per component, gated on
<component>.pdb.enabled with minAvailable/maxUnavailable (minAvailable wins,
fallback maxUnavailable: 1), selectors matching each component's
selectorLabels. Adds <component>.topologySpreadConstraints rendered into
each Deployment pod spec. PDBs default to disabled since the default
hpa.minReplicas of 1 with minAvailable: 1 would block drains entirely.
Resolves LIT-4452
* fix(helm): honor explicit 0 in pdb minAvailable/maxUnavailable
A Go-template truthy check treated an explicit 0 (forbid all voluntary
disruptions via maxUnavailable: 0) as unset and silently replaced it with
the fallback maxUnavailable: 1, weakening the configured protection. Treat
a value as set when it is non-nil and non-empty-string instead.
