mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-27 01:22:18 +00:00
ae01882535
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar py-spy on the gateway showed the post-response _PROXY_track_cost_callback, spend-log and DBSpendUpdateWriter work running on the inference workers' event loop, so a DB or Redis stall backed up the request path. When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact typed SpendEvent per success and hands it to a SpendEventProducer that ships it over a unix socket (default) or loopback-only TCP to a sidecar started as `python -m gateway.spend_worker`. The sidecar runs the unchanged _ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url). When the sidecar is unreachable, the buffer is full, or the gateway shuts down with events still queued or in flight, the producer applies LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The sidecar half-closes producers on SIGTERM and drains, the producer treats EOF as unavailable, and the gateway flushes buffered spend counters on shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its own process log. Helm: both charts gain an opt-in spend-worker sidecar container sharing an emptyDir socket dir, and the componentized chart's HPA uses a ContainerResource CPU metric scoped to the gateway container so sidecar CPU does not drive inference scaling. * feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): retrieve a completed batch in the in-process spend path test The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): rename the spend worker sidecar to collector Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the pod-local pgbouncer URL itself, replacing gateway/collector.py The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it. SpendEventProducer shields an in-progress fallback from the writer task cancellation so close() no longer loses an event already handed to the in-process pipeline Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs, flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(terraform): drop the gcp job duplicated by the aws/gcp matrix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): format pooled_database_url after the pgbouncer rebase Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain Cache hits updated the logging object after the early return, so the offloaded spend event carried preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against producer transports uvloop already closed so one dead connection cannot abort the drain Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(terraform): keep the gcp collector port off the metrics sidecar health port Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): collector connects to Postgres directly under IAM or Entra token auth Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
250 lines
9.3 KiB
YAML
250 lines
9.3 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
annotations:
|
|
{{- toYaml .Values.deploymentAnnotations | nindent 4 }}
|
|
name: {{ include "litellm.fullname" . }}
|
|
labels:
|
|
{{- include "litellm.labels" . | nindent 4 }}
|
|
{{- if .Values.deploymentLabels }}
|
|
{{- toYaml .Values.deploymentLabels | nindent 4 }}
|
|
{{- end }}
|
|
spec:
|
|
{{- if and (not .Values.keda.enabled) (not .Values.autoscaling.enabled) }}
|
|
replicas: {{ .Values.replicaCount }}
|
|
{{- end }}
|
|
{{- with .Values.strategy }}
|
|
strategy:
|
|
{{- toYaml . | nindent 4 }}
|
|
{{- end }}
|
|
selector:
|
|
matchLabels:
|
|
{{- include "litellm.selectorLabels" . | nindent 6 }}
|
|
{{- if .Values.deploymentMinReadySeconds }}
|
|
minReadySeconds: {{ .Values.deploymentMinReadySeconds }}
|
|
{{- end }}
|
|
template:
|
|
metadata:
|
|
annotations:
|
|
{{- if .Values.proxyConfigMap.create }}
|
|
checksum/config: {{ include (print $.Template.BasePath "/configmap-litellm.yaml") . | sha256sum }}
|
|
{{- end }}
|
|
{{- with .Values.podAnnotations }}
|
|
{{- tpl (toYaml .) $ | nindent 8 }}
|
|
{{- end }}
|
|
labels:
|
|
{{- include "litellm.labels" . | nindent 8 }}
|
|
{{- with .Values.podLabels }}
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
spec:
|
|
{{- with .Values.imagePullSecrets }}
|
|
imagePullSecrets:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
serviceAccountName: {{ include "litellm.serviceAccountName" . }}
|
|
securityContext:
|
|
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
|
{{- with .Values.extraInitContainers }}
|
|
initContainers:
|
|
{{- tpl (toYaml .) $ | nindent 8 }}
|
|
{{- end }}
|
|
containers:
|
|
- name: {{ include "litellm.name" . }}
|
|
securityContext:
|
|
{{- toYaml .Values.securityContext | nindent 12 }}
|
|
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
|
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
|
env:
|
|
{{- include "litellm.proxyEnv" . | nindent 12 }}
|
|
{{- include "litellm.proxyMetricsEnv" . | nindent 12 }}
|
|
{{- if .Values.collector.enabled }}
|
|
{{- include "litellm.collectorEnv" . | nindent 12 }}
|
|
{{- end }}
|
|
envFrom:
|
|
{{- range .Values.environmentSecrets }}
|
|
- secretRef:
|
|
name: {{ . }}
|
|
{{- end }}
|
|
{{- range .Values.environmentConfigMaps }}
|
|
- configMapRef:
|
|
name: {{ . }}
|
|
{{- end }}
|
|
{{- if .Values.command }}
|
|
command: {{ toYaml .Values.command | nindent 12 }}
|
|
{{- end }}
|
|
{{- if .Values.args }}
|
|
args: {{ toYaml .Values.args | nindent 12 }}
|
|
{{- else }}
|
|
args:
|
|
- --config
|
|
- /etc/litellm/config.yaml
|
|
{{ if .Values.numWorkers }}
|
|
- --num_workers
|
|
- {{ .Values.numWorkers | quote }}
|
|
{{- end }}
|
|
{{- end }}
|
|
ports:
|
|
- name: http
|
|
containerPort: {{ .Values.service.port }}
|
|
protocol: TCP
|
|
{{- if .Values.metricsServer.enabled }}
|
|
- name: metrics
|
|
containerPort: {{ .Values.metricsServer.port }}
|
|
protocol: TCP
|
|
{{- end }}
|
|
livenessProbe:
|
|
httpGet:
|
|
path: {{ .Values.livenessProbe.path | quote }}
|
|
port: "http"
|
|
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
|
|
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
|
|
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
|
|
successThreshold: {{ .Values.livenessProbe.successThreshold }}
|
|
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
|
|
readinessProbe:
|
|
httpGet:
|
|
path: {{ .Values.readinessProbe.path | quote }}
|
|
port: "http"
|
|
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
|
|
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
|
|
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
|
|
successThreshold: {{ .Values.readinessProbe.successThreshold }}
|
|
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
|
|
startupProbe:
|
|
httpGet:
|
|
path: {{ .Values.startupProbe.path | quote }}
|
|
port: "http"
|
|
initialDelaySeconds: {{ .Values.startupProbe.initialDelaySeconds }}
|
|
periodSeconds: {{ .Values.startupProbe.periodSeconds }}
|
|
timeoutSeconds: {{ .Values.startupProbe.timeoutSeconds }}
|
|
successThreshold: {{ .Values.startupProbe.successThreshold }}
|
|
failureThreshold: {{ .Values.startupProbe.failureThreshold }}
|
|
resources:
|
|
{{- toYaml .Values.resources | nindent 12 }}
|
|
volumeMounts:
|
|
- name: litellm-config
|
|
mountPath: /etc/litellm/config.yaml
|
|
subPath: config.yaml
|
|
{{ if .Values.securityContext.readOnlyRootFilesystem }}
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
- name: cache
|
|
mountPath: /.cache
|
|
- name: npm
|
|
mountPath: /.npm
|
|
{{- end }}
|
|
{{- if .Values.billingMetrics.enabled }}
|
|
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
|
|
{{- end }}
|
|
{{- if include "litellm.collector.socketDir" . }}
|
|
- name: collector-socket
|
|
mountPath: {{ include "litellm.collector.socketDir" . }}
|
|
{{- end }}
|
|
{{- with .Values.volumeMounts }}
|
|
{{- toYaml . | nindent 12 }}
|
|
{{- end }}
|
|
{{- with .Values.lifecycle }}
|
|
lifecycle:
|
|
{{- toYaml . | nindent 12 }}
|
|
{{- end }}
|
|
{{- if .Values.collector.enabled }}
|
|
- name: {{ include "litellm.name" . }}-collector
|
|
securityContext:
|
|
{{- toYaml .Values.securityContext | nindent 12 }}
|
|
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
|
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
|
command: {{ toYaml .Values.collector.command | nindent 12 }}
|
|
env:
|
|
{{- include "litellm.proxyEnv" . | nindent 12 }}
|
|
{{- include "litellm.collectorEnv" . | nindent 12 }}
|
|
- name: LITELLM_JOB_ROLE
|
|
value: collector
|
|
{{- if not (hasKey (default dict .Values.envVars) "CONFIG_FILE_PATH") }}
|
|
- name: CONFIG_FILE_PATH
|
|
value: /etc/litellm/config.yaml
|
|
{{- end }}
|
|
envFrom:
|
|
{{- range .Values.environmentSecrets }}
|
|
- secretRef:
|
|
name: {{ . }}
|
|
{{- end }}
|
|
{{- range .Values.environmentConfigMaps }}
|
|
- configMapRef:
|
|
name: {{ . }}
|
|
{{- end }}
|
|
resources:
|
|
{{- toYaml .Values.collector.resources | nindent 12 }}
|
|
volumeMounts:
|
|
- name: litellm-config
|
|
mountPath: /etc/litellm/config.yaml
|
|
subPath: config.yaml
|
|
{{- if include "litellm.collector.socketDir" . }}
|
|
- name: collector-socket
|
|
mountPath: {{ include "litellm.collector.socketDir" . }}
|
|
{{- end }}
|
|
{{ if .Values.securityContext.readOnlyRootFilesystem }}
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
- name: cache
|
|
mountPath: /.cache
|
|
- name: npm
|
|
mountPath: /.npm
|
|
{{- end }}
|
|
{{- with .Values.volumeMounts }}
|
|
{{- toYaml . | nindent 12 }}
|
|
{{- end }}
|
|
{{- end }}
|
|
{{- with .Values.extraContainers }}
|
|
{{- tpl (toYaml .) $ | nindent 8 }}
|
|
{{- end }}
|
|
volumes:
|
|
{{ if .Values.securityContext.readOnlyRootFilesystem }}
|
|
- name: tmp
|
|
emptyDir:
|
|
sizeLimit: 500Mi
|
|
- name: cache
|
|
emptyDir:
|
|
sizeLimit: 500Mi
|
|
- name: npm
|
|
emptyDir:
|
|
sizeLimit: 500Mi
|
|
{{- end }}
|
|
- name: litellm-config
|
|
configMap:
|
|
{{- if .Values.proxyConfigMap.create }}
|
|
name: {{ include "litellm.fullname" . }}-config
|
|
{{- else }}
|
|
name: {{ .Values.proxyConfigMap.name }}
|
|
{{- end }}
|
|
items:
|
|
- key: {{ .Values.proxyConfigMap.key | default "config.yaml" }}
|
|
path: "config.yaml"
|
|
{{- if .Values.billingMetrics.enabled }}
|
|
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
|
|
{{- end }}
|
|
{{- if include "litellm.collector.socketDir" . }}
|
|
- name: collector-socket
|
|
emptyDir:
|
|
sizeLimit: 1Mi
|
|
{{- end }}
|
|
{{- with .Values.volumes }}
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with .Values.nodeSelector }}
|
|
nodeSelector:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with .Values.affinity }}
|
|
affinity:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with .Values.tolerations }}
|
|
tolerations:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds | default 90 }}
|
|
{{- if .Values.topologySpreadConstraints }}
|
|
topologySpreadConstraints:
|
|
{{- toYaml .Values.topologySpreadConstraints | nindent 8 }}
|
|
{{- end }}
|