feat(proxy): offload spend tracking to a pod-local collector sidecar (#40545)

* feat(proxy): offload spend tracking to a pod-local spend worker sidecar

py-spy on the gateway showed the post-response _PROXY_track_cost_callback,
spend-log and DBSpendUpdateWriter work running on the inference workers'
event loop, so a DB or Redis stall backed up the request path.

When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact
typed SpendEvent per success and hands it to a SpendEventProducer that ships
it over a unix socket (default) or loopback-only TCP to a sidecar started as
`python -m gateway.spend_worker`. The sidecar runs the unchanged
_ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url).
When the sidecar is unreachable, the buffer is full, or the gateway shuts
down with events still queued or in flight, the producer applies
LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The
sidecar half-closes producers on SIGTERM and drains, the producer treats
EOF as unavailable, and the gateway flushes buffered spend counters on
shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its
own process log.

Helm: both charts gain an opt-in spend-worker sidecar container sharing an
emptyDir socket dir, and the componentized chart's HPA uses a
ContainerResource CPU metric scoped to the gateway container so sidecar
CPU does not drive inference scaling.

* feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks

Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): retrieve a completed batch in the in-process spend path test

The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): rename the spend worker sidecar to collector

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown

The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime
stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the
pod-local pgbouncer URL itself, replacing gateway/collector.py

The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it.
SpendEventProducer shields an in-progress fallback from the writer task cancellation so close()
no longer loses an event already handed to the in-process pipeline

Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs,
flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(terraform): drop the gcp job duplicated by the aws/gcp matrix

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports

The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): format pooled_database_url after the pgbouncer rebase

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain

Cache hits updated the logging object after the early return, so the offloaded spend event carried
preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against
producer transports uvloop already closed so one dead connection cannot abort the drain

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(terraform): keep the gcp collector port off the metrics sidecar health port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): collector connects to Postgres directly under IAM or Entra token auth

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

.github/workflows/test-terraform-modules.yml

@@ -25,13 +25,17 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  aws-module:
-    name: fmt, validate, test (aws)
+  module:
+    name: fmt, validate, test (${{ matrix.module }})
     runs-on: ubuntu-latest
     timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        module: [aws, gcp]
     defaults:
       run:
-        working-directory: terraform/litellm/aws
+        working-directory: terraform/litellm/${{ matrix.module }}
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
@@ -51,35 +55,7 @@ jobs:
       - name: validate
         run: terraform validate
 
-      # Plan-only, mock_provider-backed: no AWS credentials, no API calls.
+      # Plan-only, mock_provider-backed: no cloud credentials, no API calls.
       - name: test
         run: terraform test
 
-  gcp-module:
-    name: fmt, validate, test (gcp)
-    runs-on: ubuntu-latest
-    timeout-minutes: 15
-    defaults:
-      run:
-        working-directory: terraform/litellm/gcp
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3.1.2
-        with:
-          terraform_version: 1.13.3
-          terraform_wrapper: false
-
-      - name: fmt
-        run: terraform fmt -recursive -check -diff
-
-      - name: init
-        run: terraform init -backend=false -input=false
-
-      - name: validate
-        run: terraform validate
-
-      - name: test
-        run: terraform test

helm/litellm-helm/templates/_helpers.tpl

@@ -161,3 +161,163 @@ taken before the change, which by that point no longer exists.
 {{- fail (printf "postgresql.image.tag must be pinned to an explicit version when db.deployStandalone is true (got %q). An unpinned tag can start a different PostgreSQL major against the existing data directory, which makes the database unreadable and is not recoverable in place. Crossing a major version requires a dump and restore." $tag) -}}
 {{- end -}}
 {{- end -}}
+
+{{/*
+Environment shared by the proxy container and the opt-in collector sidecar:
+database, pgbouncer, master key, redis, user envVars. Both containers must see
+the same DATABASE_URL and REDIS_* so the sidecar reaches the pod's pgbouncer
+and the same spend transaction buffer.
+*/}}
+{{- define "litellm.proxyEnv" -}}
+- name: HOST
+  value: "{{ .Values.listen | default "0.0.0.0" }}"
+- name: PORT
+  value: {{ .Values.service.port | quote}}
+{{- if .Values.db.deployStandalone }}
+- name: DATABASE_USERNAME
+  valueFrom:
+    secretKeyRef:
+      name: {{ include "litellm.fullname" . }}-dbcredentials
+      key: username
+- name: DATABASE_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: {{ include "litellm.fullname" . }}-dbcredentials
+      key: password
+- name: DATABASE_HOST
+  value: {{ .Release.Name }}-postgresql
+- name: DATABASE_NAME
+  value: litellm
+{{- else if .Values.db.useExisting }}
+- name: DATABASE_USERNAME
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.db.secret.name }}
+      key: {{ .Values.db.secret.usernameKey }}
+- name: DATABASE_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.db.secret.name }}
+      key: {{ .Values.db.secret.passwordKey }}
+- name: DATABASE_HOST
+  {{- if .Values.db.secret.endpointKey }}
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.db.secret.name }}
+      key: {{ .Values.db.secret.endpointKey }}
+  {{- else }}
+  value: {{ .Values.db.endpoint }}
+  {{- end }}
+- name: DATABASE_NAME
+  value: {{ .Values.db.database }}
+- name: DATABASE_URL
+  value: {{ .Values.db.url | quote }}
+{{- end }}
+{{- if and .Values.db.useExisting .Values.db.readReplicaUrl .Values.db.secret.readReplicaEndpointKey (not .Values.db.secret.readReplicaUrlKey) }}
+- name: DATABASE_READER_HOST
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.db.secret.name }}
+      key: {{ .Values.db.secret.readReplicaEndpointKey }}
+{{- end }}
+{{- if and .Values.db.useExisting .Values.db.secret.readReplicaUrlKey }}
+- name: DATABASE_URL_READ_REPLICA
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.db.secret.name }}
+      key: {{ .Values.db.secret.readReplicaUrlKey }}
+{{- else if .Values.db.readReplicaUrl }}
+- name: DATABASE_URL_READ_REPLICA
+  value: {{ .Values.db.readReplicaUrl | quote }}
+{{- end }}
+{{- if .Values.db.connectionPool.enabled }}
+- name: LITELLM_PGBOUNCER_ENABLED
+  value: "true"
+- name: LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS
+  value: {{ .Values.db.connectionPool.maxDbConnections | quote }}
+- name: LITELLM_PGBOUNCER_MAX_CLIENT_CONN
+  value: {{ .Values.db.connectionPool.maxClientConn | quote }}
+{{- end }}
+- name: PROXY_MASTER_KEY
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.masterkeySecretName | default (printf "%s-masterkey" (include "litellm.fullname" .)) }}
+      key: {{ .Values.masterkeySecretKey | default "masterkey" }}
+{{- if .Values.redis.enabled }}
+- name: REDIS_HOST
+  value: {{ include "litellm.redis.serviceName" . }}
+- name: REDIS_PORT
+  value: {{ include "litellm.redis.port" . | quote }}
+- name: REDIS_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: {{ include "redis.secretName" .Subcharts.redis }}
+      key: {{include "redis.secretPasswordKey" .Subcharts.redis }}
+{{- end }}
+{{- /*
+  Inject LITELLM_LOG only when envVars does not already define it.
+*/}}
+{{- if and .Values.logLevel (not (hasKey (default dict .Values.envVars) "LITELLM_LOG")) }}
+- name: LITELLM_LOG
+  value: {{ .Values.logLevel | quote }}
+{{- end }}
+{{- if .Values.envVars }}
+{{- range $key, $val := .Values.envVars }}
+- name: {{ $key }}
+  value: {{ $val | quote }}
+{{- end }}
+{{- end }}
+{{- with .Values.extraEnvVars }}
+{{ toYaml . }}
+{{- end }}
+{{- if .Values.migrationJob.enabled }}
+# Schema updates are owned by the dedicated migrations Job; skip
+# the proxy's startup `prisma db push` so N replicas don't race
+# one DB on every rollout. Placed last (after envVars and
+# extraEnvVars) so this override can't be silently shadowed by a
+# user-supplied DISABLE_SCHEMA_UPDATE under last-wins duplicate-env
+# semantics — same pattern the migrations Job uses.
+- name: DISABLE_SCHEMA_UPDATE
+  value: "true"
+{{- end }}
+{{- end -}}
+
+{{/*
+Proxy-only metering and metrics env. The collector sidecar serves no HTTP
+traffic, so it gets neither.
+*/}}
+{{- define "litellm.proxyMetricsEnv" -}}
+{{- if .Values.billingMetrics.enabled }}
+{{ include "litellm.billingMetricsEnv" . }}
+{{- end }}
+{{- if .Values.metricsServer.enabled }}
+{{- if eq (int .Values.metricsServer.port) (int .Values.service.port) }}
+{{- fail "metricsServer.port must differ from service.port" }}
+{{- end }}
+- name: PROMETHEUS_METRICS_PORT
+  value: {{ .Values.metricsServer.port | quote }}
+{{- end }}
+{{- end -}}
+
+{{/*
+Directory of the collector's unix socket, shared between the two containers
+through an emptyDir. Empty when the sidecar is off or uses 127.0.0.1 TCP.
+*/}}
+{{- define "litellm.collector.socketDir" -}}
+{{- if and .Values.collector.enabled (hasPrefix "unix://" .Values.collector.address) -}}
+{{- dir (trimPrefix "unix://" .Values.collector.address) -}}
+{{- end -}}
+{{- end -}}
+
+{{- define "litellm.collectorEnv" -}}
+- name: LITELLM_COLLECTOR_ENABLED
+  value: "true"
+- name: LITELLM_COLLECTOR_ADDRESS
+  value: {{ .Values.collector.address | quote }}
+- name: LITELLM_COLLECTOR_BUFFER_SIZE
+  value: {{ .Values.collector.bufferSize | quote }}
+- name: LITELLM_COLLECTOR_ON_UNAVAILABLE
+  value: {{ .Values.collector.onUnavailable | quote }}
+- name: LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS
+  value: {{ .Values.collector.drainTimeoutSeconds | quote }}
+{{- end -}}

helm/litellm-helm/templates/deployment.yaml

@@ -56,126 +56,10 @@ spec:
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           env:
-            - name: HOST
-              value: "{{ .Values.listen | default "0.0.0.0" }}"
-            - name: PORT
-              value: {{ .Values.service.port | quote}}
-            {{- if .Values.db.deployStandalone }}
-            - name: DATABASE_USERNAME
-              valueFrom:
-                secretKeyRef:
-                  name: {{ include "litellm.fullname" . }}-dbcredentials
-                  key: username
-            - name: DATABASE_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: {{ include "litellm.fullname" . }}-dbcredentials
-                  key: password
-            - name: DATABASE_HOST
-              value: {{ .Release.Name }}-postgresql
-            - name: DATABASE_NAME
-              value: litellm
-            {{- else if .Values.db.useExisting }}
-            - name: DATABASE_USERNAME
-              valueFrom:
-                secretKeyRef:
-                  name: {{ .Values.db.secret.name }}
-                  key: {{ .Values.db.secret.usernameKey }}
-            - name: DATABASE_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: {{ .Values.db.secret.name }}
-                  key: {{ .Values.db.secret.passwordKey }}
-            - name: DATABASE_HOST
-              {{- if .Values.db.secret.endpointKey }}
-              valueFrom:
-                secretKeyRef:
-                  name: {{ .Values.db.secret.name }}
-                  key: {{ .Values.db.secret.endpointKey }}
-              {{- else }}
-              value: {{ .Values.db.endpoint }}
-              {{- end }}
-            - name: DATABASE_NAME
-              value: {{ .Values.db.database }}
-            - name: DATABASE_URL
-              value: {{ .Values.db.url | quote }}
-            {{- end }}
-            {{- if and .Values.db.useExisting .Values.db.readReplicaUrl .Values.db.secret.readReplicaEndpointKey (not .Values.db.secret.readReplicaUrlKey) }}
-            - name: DATABASE_READER_HOST
-              valueFrom:
-                secretKeyRef:
-                  name: {{ .Values.db.secret.name }}
-                  key: {{ .Values.db.secret.readReplicaEndpointKey }}
-            {{- end }}
-            {{- if and .Values.db.useExisting .Values.db.secret.readReplicaUrlKey }}
-            - name: DATABASE_URL_READ_REPLICA
-              valueFrom:
-                secretKeyRef:
-                  name: {{ .Values.db.secret.name }}
-                  key: {{ .Values.db.secret.readReplicaUrlKey }}
-            {{- else if .Values.db.readReplicaUrl }}
-            - name: DATABASE_URL_READ_REPLICA
-              value: {{ .Values.db.readReplicaUrl | quote }}
-            {{- end }}
-            {{- if .Values.db.connectionPool.enabled }}
-            - name: LITELLM_PGBOUNCER_ENABLED
-              value: "true"
-            - name: LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS
-              value: {{ .Values.db.connectionPool.maxDbConnections | quote }}
-            - name: LITELLM_PGBOUNCER_MAX_CLIENT_CONN
-              value: {{ .Values.db.connectionPool.maxClientConn | quote }}
-            {{- end }}
-            - name: PROXY_MASTER_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: {{ .Values.masterkeySecretName | default (printf "%s-masterkey" (include "litellm.fullname" .)) }}
-                  key: {{ .Values.masterkeySecretKey | default "masterkey" }}
-            {{- if .Values.redis.enabled }}
-            - name: REDIS_HOST
-              value: {{ include "litellm.redis.serviceName" . }}
-            - name: REDIS_PORT
-              value: {{ include "litellm.redis.port" . | quote }}
-            - name: REDIS_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: {{ include "redis.secretName" .Subcharts.redis }}
-                  key: {{include "redis.secretPasswordKey" .Subcharts.redis }}
-            {{- end }}
-            {{- /*
-              Inject LITELLM_LOG only when envVars does not already define it.
-            */}}
-            {{- if and .Values.logLevel (not (hasKey (default dict .Values.envVars) "LITELLM_LOG")) }}
-            - name: LITELLM_LOG
-              value: {{ .Values.logLevel | quote }}
-            {{- end }}
-            {{- if .Values.envVars }}
-            {{- range $key, $val := .Values.envVars }}
-            - name: {{ $key }}
-              value: {{ $val | quote }}
-            {{- end }}
-            {{- end }}
-            {{- with .Values.extraEnvVars }}
-              {{- toYaml . | nindent 12 }}
-            {{- end }}
-            {{- if .Values.billingMetrics.enabled }}
-            {{- include "litellm.billingMetricsEnv" . | nindent 12 }}
-            {{- end }}
-            {{- if .Values.metricsServer.enabled }}
-            {{- if eq (int .Values.metricsServer.port) (int .Values.service.port) }}
-            {{- fail "metricsServer.port must differ from service.port" }}
-            {{- end }}
-            - name: PROMETHEUS_METRICS_PORT
-              value: {{ .Values.metricsServer.port | quote }}
-            {{- end }}
-            {{- if .Values.migrationJob.enabled }}
-            # Schema updates are owned by the dedicated migrations Job; skip
-            # the proxy's startup `prisma db push` so N replicas don't race
-            # one DB on every rollout. Placed last (after envVars and
-            # extraEnvVars) so this override can't be silently shadowed by a
-            # user-supplied DISABLE_SCHEMA_UPDATE under last-wins duplicate-env
-            # semantics — same pattern the migrations Job uses.
-            - name: DISABLE_SCHEMA_UPDATE
-              value: "true"
+            {{- include "litellm.proxyEnv" . | nindent 12 }}
+            {{- include "litellm.proxyMetricsEnv" . | nindent 12 }}
+            {{- if .Values.collector.enabled }}
+            {{- include "litellm.collectorEnv" . | nindent 12 }}
             {{- end }}
           envFrom:
           {{- range .Values.environmentSecrets }}
@@ -253,6 +137,10 @@ spec:
           {{- if .Values.billingMetrics.enabled }}
             {{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
           {{- end }}
+          {{- if include "litellm.collector.socketDir" . }}
+            - name: collector-socket
+              mountPath: {{ include "litellm.collector.socketDir" . }}
+          {{- end }}
           {{- with .Values.volumeMounts }}
             {{- toYaml . | nindent 12 }}
           {{- end }}
@@ -260,6 +148,53 @@ spec:
           lifecycle:
             {{- toYaml . | nindent 12 }}
           {{- end }}
+      {{- if .Values.collector.enabled }}
+        - name: {{ include "litellm.name" . }}-collector
+          securityContext:
+            {{- toYaml .Values.securityContext | nindent 12 }}
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          command: {{ toYaml .Values.collector.command | nindent 12 }}
+          env:
+            {{- include "litellm.proxyEnv" . | nindent 12 }}
+            {{- include "litellm.collectorEnv" . | nindent 12 }}
+            - name: LITELLM_JOB_ROLE
+              value: collector
+            {{- if not (hasKey (default dict .Values.envVars) "CONFIG_FILE_PATH") }}
+            - name: CONFIG_FILE_PATH
+              value: /etc/litellm/config.yaml
+            {{- end }}
+          envFrom:
+          {{- range .Values.environmentSecrets }}
+            - secretRef:
+                name: {{ . }}
+          {{- end }}
+          {{- range .Values.environmentConfigMaps }}
+            - configMapRef:
+                name: {{ . }}
+          {{- end }}
+          resources:
+            {{- toYaml .Values.collector.resources | nindent 12 }}
+          volumeMounts:
+            - name: litellm-config
+              mountPath: /etc/litellm/config.yaml
+              subPath: config.yaml
+          {{- if include "litellm.collector.socketDir" . }}
+            - name: collector-socket
+              mountPath: {{ include "litellm.collector.socketDir" . }}
+          {{- end }}
+          {{ if .Values.securityContext.readOnlyRootFilesystem }}
+            - name: tmp
+              mountPath: /tmp
+            - name: cache
+              mountPath: /.cache
+            - name: npm
+              mountPath: /.npm
+          {{- end }}
+          {{- with .Values.volumeMounts }}
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+      {{- end }}
       {{- with .Values.extraContainers }}
         {{- tpl (toYaml .) $ | nindent 8 }}
       {{- end }}
@@ -288,6 +223,11 @@ spec:
         {{- if .Values.billingMetrics.enabled }}
         {{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
         {{- end }}
+        {{- if include "litellm.collector.socketDir" . }}
+        - name: collector-socket
+          emptyDir:
+            sizeLimit: 1Mi
+        {{- end }}
       {{- with .Values.volumes }}
         {{- toYaml . | nindent 8 }}
       {{- end }}

helm/litellm-helm/templates/hpa.yaml

@@ -18,6 +18,15 @@ spec:
   {{- end }}
   metrics:
     {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
+    {{- if and .Values.collector.enabled .Values.collector.scaleOnProxyContainerCpu }}
+    - type: ContainerResource
+      containerResource:
+        name: cpu
+        container: {{ include "litellm.name" . }}
+        target:
+          type: Utilization
+          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
+    {{- else }}
     - type: Resource
       resource:
         name: cpu
@@ -25,6 +34,7 @@ spec:
           type: Utilization
           averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
     {{- end }}
+    {{- end }}
     {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
     - type: Resource
       resource:

helm/litellm-helm/tests/collector_tests.yaml

@@ -0,0 +1,272 @@
+suite: test collector sidecar
+templates:
+  - deployment.yaml
+  - hpa.yaml
+  - configmap-litellm.yaml
+tests:
+  - it: should run the proxy alone with no collector env by default
+    template: deployment.yaml
+    asserts:
+      - lengthEqual:
+          path: spec.template.spec.containers
+          count: 1
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ENABLED
+            value: "true"
+      - notContains:
+          path: spec.template.spec.volumes
+          content:
+            name: collector-socket
+          any: true
+
+  - it: should add the sidecar on the same image and point both containers at the unix socket
+    template: deployment.yaml
+    set:
+      image.tag: test
+      db.connectionPool.enabled: true
+      collector.enabled: true
+      collector.resources:
+        requests:
+          cpu: 500m
+          memory: 1Gi
+        limits:
+          cpu: "1"
+          memory: 2Gi
+    asserts:
+      - lengthEqual:
+          path: spec.template.spec.containers
+          count: 2
+      - equal:
+          path: spec.template.spec.containers[1].name
+          value: litellm-collector
+      - equal:
+          path: spec.template.spec.containers[1].image
+          value: ghcr.io/berriai/litellm:test
+      - equal:
+          path: spec.template.spec.containers[1].command
+          value: [python, -m, litellm.proxy.collector]
+      - equal:
+          path: spec.template.spec.containers[1].resources.requests.cpu
+          value: 500m
+      - equal:
+          path: spec.template.spec.containers[1].resources.limits.memory
+          value: 2Gi
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ENABLED
+            value: "true"
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ADDRESS
+            value: unix:///var/run/litellm/collector.sock
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_BUFFER_SIZE
+            value: "1000"
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ON_UNAVAILABLE
+            value: fallback
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_JOB_ROLE
+            value: collector
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_JOB_ROLE
+            value: collector
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_COLLECTOR_ADDRESS
+            value: unix:///var/run/litellm/collector.sock
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: CONFIG_FILE_PATH
+            value: /etc/litellm/config.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: DATABASE_HOST
+            value: RELEASE-NAME-postgresql
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: DATABASE_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                name: RELEASE-NAME-litellm-dbcredentials
+                key: password
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_PGBOUNCER_ENABLED
+            value: "true"
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS
+            value: "20"
+      - contains:
+          path: spec.template.spec.containers[0].volumeMounts
+          content:
+            name: collector-socket
+            mountPath: /var/run/litellm
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: collector-socket
+            mountPath: /var/run/litellm
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: litellm-config
+            mountPath: /etc/litellm/config.yaml
+            subPath: config.yaml
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: collector-socket
+            emptyDir:
+              sizeLimit: 1Mi
+
+  - it: should skip the socket volume and pass the policy through on tcp transport
+    template: deployment.yaml
+    set:
+      collector.enabled: true
+      collector.address: tcp://127.0.0.1:4100
+      collector.onUnavailable: drop
+      collector.bufferSize: 50
+      envVars:
+        CONFIG_FILE_PATH: /custom/config.yaml
+    asserts:
+      - lengthEqual:
+          path: spec.template.spec.containers
+          count: 2
+      - notContains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: CONFIG_FILE_PATH
+            value: /etc/litellm/config.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: CONFIG_FILE_PATH
+            value: /custom/config.yaml
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ADDRESS
+            value: tcp://127.0.0.1:4100
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ON_UNAVAILABLE
+            value: drop
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_BUFFER_SIZE
+            value: "50"
+      - notContains:
+          path: spec.template.spec.volumes
+          content:
+            name: collector-socket
+          any: true
+
+  - it: should keep metrics and billing env on the proxy container only
+    template: deployment.yaml
+    set:
+      collector.enabled: true
+      metricsServer.enabled: true
+      metricsServer.port: 9090
+      billingMetrics.enabled: true
+      billingMetrics.endpoint: https://metering.example.com
+      billingMetrics.secretName: billing-mtls
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: PROMETHEUS_METRICS_PORT
+            value: "9090"
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_BILLING_METRICS_ENDPOINT
+            value: https://metering.example.com
+      - notContains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: PROMETHEUS_METRICS_PORT
+          any: true
+      - notContains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_BILLING_METRICS_ENDPOINT
+          any: true
+      - notContains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: billing-metrics-mtls
+          any: true
+
+  - it: should give the sidecar the same scratch mounts as the proxy on a read-only root
+    template: deployment.yaml
+    set:
+      collector.enabled: true
+      securityContext.readOnlyRootFilesystem: true
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: npm
+            mountPath: /.npm
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: cache
+            mountPath: /.cache
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: tmp
+            mountPath: /tmp
+
+  - it: should keep the pod-wide cpu metric unless asked to scale on the proxy container
+    template: hpa.yaml
+    set:
+      autoscaling.enabled: true
+      collector.enabled: true
+    asserts:
+      - equal: { path: "spec.metrics[0].type", value: Resource }
+      - equal: { path: "spec.metrics[0].resource.name", value: cpu }
+
+  - it: should scale on the proxy container's cpu only when opted in
+    template: hpa.yaml
+    set:
+      autoscaling.enabled: true
+      collector.enabled: true
+      collector.scaleOnProxyContainerCpu: true
+    asserts:
+      - equal: { path: "spec.metrics[0].type", value: ContainerResource }
+      - equal: { path: "spec.metrics[0].containerResource.name", value: cpu }
+      - equal: { path: "spec.metrics[0].containerResource.container", value: litellm }
+      - equal: { path: "spec.metrics[0].containerResource.target.averageUtilization", value: 60 }
+      - isNull: { path: "spec.metrics[0].resource" }
+
+  - it: should not switch to the container metric while the sidecar is off
+    template: hpa.yaml
+    set:
+      autoscaling.enabled: true
+      collector.scaleOnProxyContainerCpu: true
+    asserts:
+      - equal: { path: "spec.metrics[0].type", value: Resource }

helm/litellm-helm/values.yaml

@@ -190,6 +190,48 @@ metricsServer:
   enabled: false
   port: 4001
 
+# Opt-in sidecar that runs the post-response spend pipeline (cost calculation,
+# spend logs, spend counters, budget reservation reconciliation) so the proxy's
+# uvicorn workers only serialise a compact typed event and go back to serving
+# inference. Same image and tag as the proxy, second container in the same pod,
+# fed over loopback (a unix socket on a shared emptyDir, or 127.0.0.1 TCP). It
+# reuses the pod's in-container pgbouncer (db.connectionPool) and the same Redis
+# spend transaction buffer, so the per-pod DB connection budget is unchanged.
+# Delivery is at-most-once inside the pod: events already handed to the sidecar
+# are lost if it crashes before writing them; events the workers could not hand
+# over follow onUnavailable. Both containers drain on SIGTERM within
+# terminationGracePeriodSeconds
+collector:
+  enabled: false
+  # unix:///<dir>/<file>.sock (the <dir> becomes a shared emptyDir) or tcp://127.0.0.1:<port>
+  address: unix:///var/run/litellm/collector.sock
+  # Events each uvicorn worker holds in memory while the sidecar is slow or restarting
+  bufferSize: 1000
+  # fallback: run the pipeline in the worker when the sidecar is unreachable or the
+  # buffer is full (spend stays exact, that request costs proxy CPU again)
+  # drop: count and discard the event instead (spend under-reports)
+  onUnavailable: fallback
+  # How long the workers keep pushing buffered events on shutdown, and how long the
+  # sidecar keeps serving its open connections after SIGTERM
+  drainTimeoutSeconds: 10
+  command:
+    - python
+    - -m
+    - litellm.proxy.collector
+  # Sized independently of the proxy container; the pipeline is CPU bound
+  resources: {}
+  # requests:
+  #   cpu: 500m
+  #   memory: 1Gi
+  # limits:
+  #   cpu: "1"
+  #   memory: 2Gi
+  # When autoscaling.enabled, swap the pod-wide cpu Resource metric for an
+  # autoscaling/v2 ContainerResource metric on the proxy container only, so the
+  # sidecar's CPU never scales inference replicas. Needs Kubernetes 1.30+ (or the
+  # HPAContainerMetrics feature gate on 1.27 to 1.29)
+  scaleOnProxyContainerCpu: false
+
 resources:
   {}
   # Unset by default so the chart installs on small clusters such as Minikube, and so an

helm/litellm/templates/_helpers.tpl

@@ -457,3 +457,34 @@ ImplementationSpecific
 {{- end -}}
 
 {{- define "litellm.gateway.prometheusMultiprocDir" -}}/tmp/litellm_prometheus_multiproc{{- end -}}
+
+{{/*
+Directory of the collector's unix socket, shared by the gateway and
+collector containers through an emptyDir. Empty when the sidecar is off
+or gateway.collector.address is a tcp://127.0.0.1:<port> address.
+*/}}
+{{- define "litellm.gateway.collectorSocketDir" -}}
+{{- if and .Values.gateway.collector.enabled (hasPrefix "unix://" .Values.gateway.collector.address) -}}
+{{- dir (trimPrefix "unix://" .Values.gateway.collector.address) -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+LITELLM_COLLECTOR_* env shared by the producer (gateway container) and the
+consumer (collector container), so both agree on the transport and the
+shutdown drain window.
+*/}}
+{{- define "litellm.gateway.collectorEnv" -}}
+{{- with .Values.gateway.collector }}
+- name: LITELLM_COLLECTOR_ENABLED
+  value: "true"
+- name: LITELLM_COLLECTOR_ADDRESS
+  value: {{ .address | quote }}
+- name: LITELLM_COLLECTOR_BUFFER_SIZE
+  value: {{ .bufferSize | quote }}
+- name: LITELLM_COLLECTOR_ON_UNAVAILABLE
+  value: {{ .onUnavailable | quote }}
+- name: LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS
+  value: {{ .drainTimeoutSeconds | quote }}
+{{- end }}
+{{- end -}}

helm/litellm/templates/gateway/deployment.yaml

@@ -74,8 +74,11 @@ spec:
             - name: PROMETHEUS_MULTIPROC_DIR
               value: {{ include "litellm.gateway.prometheusMultiprocDir" . }}
             {{- end }}
+            {{- if .Values.gateway.collector.enabled }}
+            {{- include "litellm.gateway.collectorEnv" . | nindent 12 }}
+            {{- end }}
           {{- include "litellm.envFrom" .Values.gateway | nindent 10 }}
-          {{- if or .Values.gateway.config.create .Values.gateway.volumeMounts .Values.billingMetrics.enabled .Values.gateway.metricsServer.enabled }}
+          {{- if or .Values.gateway.config.create .Values.gateway.volumeMounts .Values.billingMetrics.enabled .Values.gateway.metricsServer.enabled (include "litellm.gateway.collectorSocketDir" .) }}
           volumeMounts:
             {{- if .Values.gateway.config.create }}
             - name: gateway-config
@@ -86,6 +89,10 @@ spec:
             - name: prometheus-multiproc
               mountPath: {{ include "litellm.gateway.prometheusMultiprocDir" . }}
             {{- end }}
+            {{- if include "litellm.gateway.collectorSocketDir" . }}
+            - name: collector-socket
+              mountPath: {{ include "litellm.gateway.collectorSocketDir" . }}
+            {{- end }}
             {{- if .Values.billingMetrics.enabled }}
             {{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
             {{- end }}
@@ -145,10 +152,50 @@ spec:
           resources:
             {{- toYaml .Values.gateway.metricsServer.resources | nindent 12 }}
         {{- end }}
+        {{- if .Values.gateway.collector.enabled }}
+        - name: collector
+          image: "{{ .Values.gateway.image.repository }}:{{ .Values.gateway.image.tag | default .Chart.AppVersion }}"
+          imagePullPolicy: {{ .Values.gateway.image.pullPolicy }}
+          {{- with .Values.gateway.securityContext }}
+          securityContext:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          command:
+            - python
+            - -m
+            - litellm.proxy.collector
+          env:
+            {{- include "litellm.serverEnv" (dict "root" $ "component" .Values.gateway) | nindent 12 }}
+            {{- if .Values.gateway.config.create }}
+            - name: CONFIG_FILE_PATH
+              value: /app/config/config.yaml
+            {{- end }}
+            {{- include "litellm.gateway.collectorEnv" . | nindent 12 }}
+            - name: LITELLM_JOB_ROLE
+              value: collector
+          {{- include "litellm.envFrom" .Values.gateway | nindent 10 }}
+          {{- if or .Values.gateway.config.create .Values.gateway.volumeMounts (include "litellm.gateway.collectorSocketDir" .) }}
+          volumeMounts:
+            {{- if .Values.gateway.config.create }}
+            - name: gateway-config
+              mountPath: /app/config/config.yaml
+              subPath: config.yaml
+            {{- end }}
+            {{- if include "litellm.gateway.collectorSocketDir" . }}
+            - name: collector-socket
+              mountPath: {{ include "litellm.gateway.collectorSocketDir" . }}
+            {{- end }}
+            {{- with .Values.gateway.volumeMounts }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
+          {{- end }}
+          resources:
+            {{- toYaml .Values.gateway.collector.resources | nindent 12 }}
+        {{- end }}
       {{- with .Values.gateway.extraContainers }}
         {{- tpl (toYaml .) $ | nindent 8 }}
       {{- end }}
-      {{- if or .Values.gateway.config.create .Values.gateway.volumes .Values.billingMetrics.enabled .Values.gateway.metricsServer.enabled }}
+      {{- if or .Values.gateway.config.create .Values.gateway.volumes .Values.billingMetrics.enabled .Values.gateway.metricsServer.enabled (include "litellm.gateway.collectorSocketDir" .) }}
       volumes:
         {{- if .Values.gateway.config.create }}
         - name: gateway-config
@@ -159,6 +206,11 @@ spec:
         - name: prometheus-multiproc
           emptyDir: {}
         {{- end }}
+        {{- if include "litellm.gateway.collectorSocketDir" . }}
+        - name: collector-socket
+          emptyDir:
+            sizeLimit: 1Mi
+        {{- end }}
         {{- if .Values.billingMetrics.enabled }}
         {{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
         {{- end }}

helm/litellm/templates/gateway/hpa.yaml

@@ -15,6 +15,15 @@ spec:
   maxReplicas: {{ .Values.gateway.hpa.maxReplicas }}
   metrics:
     {{- if .Values.gateway.hpa.targetCPUUtilizationPercentage }}
+    {{- if and .Values.gateway.collector.enabled .Values.gateway.collector.scaleOnGatewayContainerCpu }}
+    - type: ContainerResource
+      containerResource:
+        name: cpu
+        container: gateway
+        target:
+          type: Utilization
+          averageUtilization: {{ .Values.gateway.hpa.targetCPUUtilizationPercentage }}
+    {{- else }}
     - type: Resource
       resource:
         name: cpu
@@ -22,6 +31,7 @@ spec:
           type: Utilization
           averageUtilization: {{ .Values.gateway.hpa.targetCPUUtilizationPercentage }}
     {{- end }}
+    {{- end }}
     {{- if .Values.gateway.hpa.targetMemoryUtilizationPercentage }}
     - type: Resource
       resource:

helm/litellm/tests/collector_tests.yaml

@@ -0,0 +1,204 @@
+suite: test gateway collector sidecar
+templates:
+  - gateway/configmap.yaml
+  - gateway/deployment.yaml
+  - gateway/hpa.yaml
+values:
+  - ./values/required.yaml
+tests:
+  - it: adds no sidecar, env, volume or container metric when the collector is off
+    asserts:
+      - lengthEqual:
+          path: spec.template.spec.containers
+          count: 1
+        template: gateway/deployment.yaml
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ENABLED
+            value: "true"
+        template: gateway/deployment.yaml
+      - notContains:
+          path: spec.template.spec.volumes
+          content:
+            name: collector-socket
+          any: true
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.metrics[0].type
+          value: Resource
+        template: gateway/hpa.yaml
+
+  - it: runs the collector as a sidecar sharing env, config and a unix socket emptyDir, and scales on the gateway container only
+    set:
+      gateway.collector.enabled: true
+      gateway.collector.bufferSize: 250
+      gateway.collector.onUnavailable: drop
+      gateway.image.tag: v1.102.0
+      gateway.numWorkers: 4
+      gateway.extraEnv:
+        - name: LITELLM_PGBOUNCER_ENABLED
+          value: "true"
+      gateway.envSecrets:
+        - litellm-license
+      gateway.volumes:
+        - name: redis-ca
+          secret:
+            secretName: redis-ca
+      gateway.volumeMounts:
+        - name: redis-ca
+          mountPath: /etc/litellm/redis-ca
+          readOnly: true
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ADDRESS
+            value: unix:///var/run/litellm/collector.sock
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_BUFFER_SIZE
+            value: "250"
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: LITELLM_COLLECTOR_ON_UNAVAILABLE
+            value: drop
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[0].volumeMounts
+          content:
+            name: collector-socket
+            mountPath: /var/run/litellm
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.template.spec.containers[1].name
+          value: collector
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.template.spec.containers[1].image
+          value: ghcr.io/berriai/litellm-gateway:v1.102.0
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.template.spec.containers[1].command
+          value:
+            - python
+            - -m
+            - litellm.proxy.collector
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_JOB_ROLE
+            value: collector
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: CONFIG_FILE_PATH
+            value: /app/config/config.yaml
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_PGBOUNCER_ENABLED
+            value: "true"
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: DATABASE_HOST
+            value: postgres.example.com
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_COLLECTOR_ADDRESS
+            value: unix:///var/run/litellm/collector.sock
+        template: gateway/deployment.yaml
+      - notContains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: NUM_WORKERS
+          any: true
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.template.spec.containers[1].envFrom
+          value:
+            - secretRef:
+                name: litellm-license
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: gateway-config
+            mountPath: /app/config/config.yaml
+            subPath: config.yaml
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: collector-socket
+            mountPath: /var/run/litellm
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: redis-ca
+            mountPath: /etc/litellm/redis-ca
+            readOnly: true
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.template.spec.containers[1].resources.limits.cpu
+          value: "1"
+        template: gateway/deployment.yaml
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: collector-socket
+            emptyDir:
+              sizeLimit: 1Mi
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.metrics[0]
+          value:
+            type: ContainerResource
+            containerResource:
+              name: cpu
+              container: gateway
+              target:
+                type: Utilization
+                averageUtilization: 70
+        template: gateway/hpa.yaml
+
+  - it: uses loopback tcp without a socket volume and keeps the pod-wide cpu metric when asked
+    set:
+      gateway.collector.enabled: true
+      gateway.collector.address: tcp://127.0.0.1:4010
+      gateway.collector.scaleOnGatewayContainerCpu: false
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[1].env
+          content:
+            name: LITELLM_COLLECTOR_ADDRESS
+            value: tcp://127.0.0.1:4010
+        template: gateway/deployment.yaml
+      - notContains:
+          path: spec.template.spec.volumes
+          content:
+            name: collector-socket
+          any: true
+        template: gateway/deployment.yaml
+      - notContains:
+          path: spec.template.spec.containers[1].volumeMounts
+          content:
+            name: collector-socket
+          any: true
+        template: gateway/deployment.yaml
+      - equal:
+          path: spec.metrics[0].type
+          value: Resource
+        template: gateway/hpa.yaml

helm/litellm/values.yaml

@@ -314,6 +314,42 @@ gateway:
     labels: {}
     interval: 15s
     scrapeTimeout: 10s
+  # Opt-in `collector` sidecar (same image, `python -m litellm.proxy.collector`)
+  # that runs the post-response spend pipeline (cost calculation, spend logs,
+  # spend counters, budget reservation reconciliation) so the uvicorn workers
+  # only serialise a compact event over loopback and go back to serving
+  # requests. It shares the pod's env, proxy config, in-container pgbouncer and
+  # Redis spend buffer, so the per-pod DB connection budget is unchanged.
+  # Delivery is at-most-once inside the pod: events already handed over are
+  # lost if the sidecar dies before writing them; events the workers cannot
+  # hand over follow `onUnavailable`.
+  collector:
+    enabled: false
+    # unix:///<dir>/<file>.sock (the <dir> becomes a shared emptyDir) or
+    # tcp://127.0.0.1:<port>
+    address: unix:///var/run/litellm/collector.sock
+    # Events each uvicorn worker holds in memory while the sidecar is slow or
+    # restarting.
+    bufferSize: 1000
+    # fallback: run the pipeline in the worker when the sidecar is unreachable
+    # or the buffer is full (spend stays exact, that request costs gateway CPU
+    # again). drop: count and discard the event instead (spend under-reports).
+    onUnavailable: fallback
+    # How long the workers keep pushing buffered events on shutdown, and how
+    # long the sidecar keeps serving open connections after SIGTERM.
+    drainTimeoutSeconds: 10
+    # Sized independently of the gateway container; the pipeline is CPU bound.
+    resources:
+      requests:
+        cpu: 500m
+        memory: 1Gi
+      limits:
+        cpu: "1"
+        memory: 2Gi
+    # With hpa.targetCPUUtilizationPercentage set, scale on an autoscaling/v2
+    # ContainerResource metric of the `gateway` container only, so the
+    # sidecar's CPU never drives inference replicas. Needs Kubernetes 1.30+.
+    scaleOnGatewayContainerCpu: true
   image:
     repository: ghcr.io/berriai/litellm-gateway
     tag: ""              # defaults to .Chart.AppVersion

litellm/caching/caching_handler.py

@@ -1217,7 +1217,9 @@ class LLMCachingHandler:
         }
 
         if litellm.cache is not None:
-            litellm_params["preset_cache_key"] = litellm.cache._get_preset_cache_key_from_kwargs(**kwargs)
+            litellm_params["preset_cache_key"] = (
+                self.preset_cache_key or litellm.cache._get_preset_cache_key_from_kwargs(**kwargs)
+            )
         else:
             litellm_params["preset_cache_key"] = None
 

litellm/proxy/collector.py

@@ -0,0 +1,220 @@
+"""Collector sidecar: consume spend events from the pod's inference workers and run the cost pipeline.
+
+Runs the proxy startup lifespan (config, Prisma, Redis transaction buffer, scheduled spend flushes)
+without serving HTTP, then listens on ``LITELLM_COLLECTOR_ADDRESS`` for newline-delimited spend
+events. Each event goes through the unchanged ``_ProxyDBLogger._PROXY_track_cost_callback``, so
+spend logs, spend counters, budget reservation reconciliation and cache updates happen exactly as
+they would in-process, just in this container. Events are handled in order per producer connection
+(one per uvicorn worker); a slow pipeline fills the socket buffer and the producer's bounded queue,
+which is the backpressure that triggers its fallback or drop policy. ``SIGTERM`` stops accepting
+connections, half-closes every producer connection so the producers switch to their unavailable
+policy, finishes the events already sent, then runs the proxy shutdown (which flushes the buffered
+spend transactions).
+
+``DATABASE_URL`` is assembled from the same ``DATABASE_*`` inputs as the proxy container, and when
+``LITELLM_PGBOUNCER_ENABLED`` is set it points at the PgBouncer that container already runs on the
+pod's loopback, so the sidecar must see the same env as the proxy. Under ``IAM_TOKEN_DB_AUTH`` or
+``AZURE_POSTGRESQL_AUTH`` that PgBouncer only accepts the token the proxy container minted, so the
+sidecar goes to Postgres directly and mints its own. Works from any image that has ``litellm``
+installed:
+
+    python -m litellm.proxy.collector [--address unix:///path.sock]
+"""
+
+import asyncio
+import logging
+import os
+import signal
+import sys
+from collections.abc import Awaitable, Callable, Mapping, Sequence
+from pathlib import Path
+from typing import Final
+
+from litellm._logging import verbose_logger, verbose_proxy_logger, verbose_router_logger
+from litellm.proxy.db.db_url_settings import DatabaseURLSettings
+from litellm.proxy.db.pgbouncer import (
+    PgBouncerError,
+    PgBouncerSettings,
+    export_pooled_database_url,
+    pooled_database_url,
+)
+from litellm.proxy.spend_tracking.spend_event_producer import (
+    COLLECTOR_JOB_ROLE,
+    AddressError,
+    CollectorAddress,
+    CollectorSettings,
+    TcpAddress,
+    UnixAddress,
+    parse_collector_address,
+)
+
+MAX_EVENT_BYTES: Final = 64 * 1024 * 1024
+
+
+class SpendEventConsumer:
+    """Accepts producer connections and runs ``handler`` on every line each one sends, in order."""
+
+    def __init__(self, handler: Callable[[bytes], Awaitable[None]]) -> None:
+        self._handler = handler
+        self._open_connections: set[asyncio.StreamWriter] = set()  # mutable-ok: live producer connections
+        self._idle = asyncio.Event()
+        self._idle.set()
+        self._received = 0
+        self._handled = 0
+        self._failed = 0
+
+    @property
+    def received(self) -> int:
+        return self._received
+
+    @property
+    def handled(self) -> int:
+        return self._handled
+
+    @property
+    def failed(self) -> int:
+        return self._failed
+
+    async def serve(self, address: CollectorAddress) -> asyncio.Server:
+        match address:
+            case UnixAddress(path=path):
+                socket_path: Final = Path(path)
+                socket_path.parent.mkdir(parents=True, exist_ok=True)
+                socket_path.unlink(missing_ok=True)
+                return await asyncio.start_unix_server(self._on_connection, path=path, limit=MAX_EVENT_BYTES)
+            case TcpAddress(host=host, port=port):
+                return await asyncio.start_server(self._on_connection, host=host, port=port, limit=MAX_EVENT_BYTES)
+
+    async def _on_connection(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
+        self._open_connections.add(writer)
+        self._idle.clear()
+        try:
+            while line := await reader.readline():
+                if not line.endswith(b"\n"):
+                    verbose_proxy_logger.error("collector: discarding truncated spend event (%d bytes)", len(line))
+                    break
+                self._received += 1
+                await self._handle(line)
+        except (ConnectionError, asyncio.IncompleteReadError, asyncio.LimitOverrunError) as error:
+            verbose_proxy_logger.warning("collector: producer connection ended abnormally: %s", error)
+        finally:
+            writer.close()
+            self._open_connections.discard(writer)
+            if not self._open_connections:
+                self._idle.set()
+
+    async def _handle(self, line: bytes) -> None:
+        try:
+            await self._handler(line)
+            self._handled += 1
+        except Exception:  # noqa: BLE001  # the cost pipeline raises anything; one bad event must not stop the sidecar
+            self._failed += 1
+            verbose_proxy_logger.exception("collector: spend event failed")
+
+    async def drain(self, timeout: float) -> int:
+        """Half-close every producer connection, then keep reading until each producer hangs up or ``timeout``.
+
+        Returns how many producer connections were still open when the timeout hit.
+        """
+        for writer in tuple(self._open_connections):
+            if writer.is_closing() or not writer.can_write_eof():
+                continue
+            try:
+                writer.write_eof()
+            except (OSError, RuntimeError) as error:
+                verbose_proxy_logger.debug("collector: producer already gone before half-close: %s", error)
+        try:
+            await asyncio.wait_for(self._idle.wait(), timeout)
+        except TimeoutError:
+            pass
+        return len(self._open_connections)
+
+
+def _install_stop_signals(loop: asyncio.AbstractEventLoop, stop: asyncio.Event) -> None:
+    for signum in (signal.SIGTERM, signal.SIGINT):
+        loop.add_signal_handler(signum, stop.set)
+
+
+async def run_collector(address: CollectorAddress, drain_timeout: float) -> None:
+    from fastapi import FastAPI
+
+    from litellm.proxy.hooks.proxy_track_cost_callback import run_spend_event
+    from litellm.proxy.proxy_server import proxy_startup_event
+
+    stop: Final = asyncio.Event()
+    _install_stop_signals(asyncio.get_running_loop(), stop)
+    consumer: Final = SpendEventConsumer(handler=run_spend_event)
+    async with proxy_startup_event(FastAPI()):
+        server: Final = await consumer.serve(address)
+        verbose_proxy_logger.info("collector: listening on %s", address)
+        await stop.wait()
+        server.close()
+        still_open: Final = await consumer.drain(drain_timeout)
+        verbose_proxy_logger.info(
+            "collector: stopping. received=%d handled=%d failed=%d connections_cut=%d",
+            consumer.received,
+            consumer.handled,
+            consumer.failed,
+            still_open,
+        )
+
+
+def address_argument(argv: Sequence[str], default: str) -> str | AddressError:
+    match tuple(argv):
+        case ():
+            return default
+        case ("--address", value):
+            return value
+        case _:
+            return AddressError(f"usage: python -m litellm.proxy.collector [--address ADDRESS], got {tuple(argv)}")
+
+
+def apply_log_level(litellm_log: str | None) -> None:
+    """Mirror the proxy's ``LITELLM_LOG`` handling: the sidecar has no CLI flags to turn logging on."""
+    level: Final = logging.getLevelNamesMapping().get((litellm_log or "").upper())
+    if level is None:
+        return
+    for logger in (verbose_logger, verbose_router_logger, verbose_proxy_logger):
+        logger.setLevel(level)
+
+
+def pod_pgbouncer_database_url(
+    pgbouncer: PgBouncerSettings, environ: Mapping[str, str], *, token_auth: bool
+) -> str | PgBouncerError | None:
+    """The proxy container's PgBouncer URL for ``environ["DATABASE_URL"]``, or None to connect to Postgres directly.
+
+    Direct is the answer when PgBouncer is off, and also under token auth: that PgBouncer's auth file
+    only holds the token its own container minted, which this container cannot present.
+    """
+    if not pgbouncer.enabled or token_auth:
+        return None
+    upstream_url: Final = environ.get("DATABASE_URL")
+    if upstream_url is None:
+        return PgBouncerError("LITELLM_PGBOUNCER_ENABLED is set but no DATABASE_URL could be assembled")
+    return pooled_database_url(upstream_url, pgbouncer)
+
+
+def main(argv: Sequence[str]) -> None:
+    os.environ.setdefault("LITELLM_JOB_ROLE", COLLECTOR_JOB_ROLE)
+    apply_log_level(os.environ.get("LITELLM_LOG"))
+    database: Final = DatabaseURLSettings.from_env()
+    database.apply_to_env()
+    pooled: Final = pod_pgbouncer_database_url(
+        PgBouncerSettings(),
+        os.environ,
+        token_auth=database.iam_token_db_auth or database.azure_postgresql_auth,
+    )
+    if isinstance(pooled, PgBouncerError):
+        sys.exit(f"LiteLLM collector: cannot use the pod's pgbouncer: {pooled.reason}")
+    if pooled is not None:
+        export_pooled_database_url(pooled)
+    settings: Final = CollectorSettings()
+    raw_address: Final = address_argument(argv, default=settings.address)
+    address: Final = raw_address if isinstance(raw_address, AddressError) else parse_collector_address(raw_address)
+    if isinstance(address, AddressError):
+        sys.exit(f"LiteLLM collector: {address.reason}")
+    asyncio.run(run_collector(address, drain_timeout=settings.drain_timeout_seconds))
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

litellm/proxy/db/pgbouncer.py

@@ -280,6 +280,22 @@ def plan_pgbouncer(
     )
 
 
+def pooled_database_url(upstream_url: str, settings: PgBouncerSettings) -> str | PgBouncerError:
+    """The loopback URL of a PgBouncer another container in the pod already runs for ``upstream_url``.
+
+    Only the container that started PgBouncer knows the pool user's password, so
+    this logs in as the upstream user, whom the auth file lists as well.
+    """
+    plan: Final = plan_pgbouncer(upstream_url, settings, runtime_dir=Path("/nonexistent"), run_as_user=None)
+    if isinstance(plan, PgBouncerError):
+        return plan
+    password: Final = urllib.parse.urlsplit(upstream_url).password or ""
+    credentials: Final = f"{urllib.parse.quote(plan.upstream_user, safe='')}:{password}"
+    return urllib.parse.urlunsplit(
+        urllib.parse.urlsplit(plan.pooled_url)._replace(netloc=f"{credentials}@{PGBOUNCER_LISTEN_ADDR}:{settings.port}")
+    )
+
+
 def _write_private(path: Path, content: str, run_as_user: str | None) -> None:
     with open(os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), "w", encoding="utf-8") as handle:
         handle.write(content)

litellm/proxy/hooks/proxy_track_cost_callback.py

@@ -27,6 +27,16 @@ from litellm.proxy.db.db_spend_update_writer import (
     get_llm_router,
 )
 from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup
+from litellm.proxy.spend_tracking.spend_event import (
+    ObjectMapping,
+    SpendEventBuildError,
+    SpendEventDecodeError,
+    build_spend_event,
+    decode_spend_event,
+    is_offloadable_success,
+    spend_event_callback_args,
+)
+from litellm.proxy.spend_tracking.spend_event_producer import SpendEventProducer
 from litellm.proxy.spend_tracking.spend_log_error_logger import (
     should_suppress_spend_log_tracebacks,
     spend_log_error,
@@ -34,6 +44,7 @@ from litellm.proxy.spend_tracking.spend_log_error_logger import (
 from litellm.proxy.spend_tracking.spend_tracking_utils import (
     _sanitize_error_information_for_spend_logs,
     get_request_model_access_groups,
+    should_store_prompts_and_responses_in_spend_logs,
 )
 from litellm.proxy.utils import ProxyUpdateSpend
 from litellm.types.utils import (
@@ -71,8 +82,43 @@ _CAPTURED_IDENTITY_CALL_TYPES: Final[frozenset[str]] = frozenset(
 
 
 class _ProxyDBLogger(CustomLogger):
-    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
-        await self._PROXY_track_cost_callback(kwargs, response_obj, start_time, end_time)
+    def __init__(
+        self,
+        spend_event_producer: SpendEventProducer | None = None,
+        *,
+        turn_off_message_logging: bool = False,
+        message_logging: bool = True,
+    ) -> None:
+        super().__init__(turn_off_message_logging=turn_off_message_logging, message_logging=message_logging)
+        self.spend_event_producer = spend_event_producer
+
+    async def async_log_success_event(
+        self, kwargs: ObjectMapping, response_obj: object, start_time: datetime, end_time: datetime
+    ) -> None:
+        if self.spend_event_producer is None or not is_offloadable_success(response_obj):
+            await self._PROXY_track_cost_callback(kwargs, response_obj, start_time, end_time)
+            return
+        event: Final = build_spend_event(
+            kwargs,
+            response_obj,
+            start_time,
+            end_time,
+            store_bodies=should_store_prompts_and_responses_in_spend_logs(),
+        )
+        if isinstance(event, SpendEventBuildError):
+            verbose_proxy_logger.warning("collector: tracking cost in-process, event not buildable: %s", event.reason)
+            await self._PROXY_track_cost_callback(kwargs, response_obj, start_time, end_time)
+            return
+        await self.spend_event_producer.publish(event)
+
+    async def run_spend_event(self, line: bytes) -> None:
+        """Run the unchanged cost pipeline on a serialized spend event (sidecar consumer and in-process fallback)."""
+        event: Final = decode_spend_event(line)
+        if isinstance(event, SpendEventDecodeError):
+            verbose_proxy_logger.error("collector: discarding undecodable spend event: %s", event.reason)
+            return
+        args: Final = spend_event_callback_args(event)
+        await self._PROXY_track_cost_callback(args.kwargs, args.response_obj, args.start_time, args.end_time)
 
     async def async_post_call_failure_hook(
         self,
@@ -503,6 +549,10 @@ def _write_spend_metadata_to_kwargs(kwargs: dict, metadata: dict) -> None:
                     bucket[key] = value
 
 
+async def run_spend_event(line: bytes) -> None:
+    await _ProxyDBLogger().run_spend_event(line)
+
+
 def _is_unbilled_interaction_response(completion_response: object) -> bool:
     from litellm.interactions.background_cost_polling import missing_usage_is_expected
     from litellm.types.interactions import InteractionsAPIResponse

litellm/proxy/proxy_server.py

@@ -469,7 +469,7 @@ from litellm.proxy.hooks.model_max_budget_limiter import (
 from litellm.proxy.hooks.prompt_injection_detection import (
     _OPTIONAL_PromptInjectionDetection,
 )
-from litellm.proxy.hooks.proxy_track_cost_callback import _ProxyDBLogger
+from litellm.proxy.hooks.proxy_track_cost_callback import _ProxyDBLogger, run_spend_event
 from litellm.proxy.image_endpoints.endpoints import router as image_router
 from litellm.proxy.list_api.common import (
     PROBLEM_TYPE_BASE,
@@ -592,6 +592,11 @@ from litellm.proxy.plugin_routes import (
 from litellm.proxy.plugin_routes import (
     router as plugin_router,
 )
+from litellm.proxy.spend_tracking.spend_event_producer import (
+    CollectorSettings,
+    SpendEventProducer,
+    build_spend_event_producer,
+)
 from litellm.types.proxy.management_endpoints.management_v1 import ProblemDetail
 
 try:
@@ -943,6 +948,17 @@ def cleanup_router_config_variables():
     heuristic_v1_tuning_baselines = None
 
 
+async def flush_spend_counters_on_shutdown() -> None:
+    if prisma_client is None:
+        return
+    try:
+        await proxy_logging_obj.db_spend_update_writer.db_update_spend_transaction_handler(
+            prisma_client=prisma_client, n_retry_times=3, proxy_logging_obj=proxy_logging_obj
+        )
+    except Exception as e:  # noqa: BLE001  # shutdown must continue even if the commit fails
+        verbose_proxy_logger.exception("Error flushing spend counters on shutdown: %s", e)
+
+
 async def _flush_spend_logs_queue_on_shutdown() -> None:
     if prisma_client is None:
         return
@@ -1378,6 +1394,10 @@ async def proxy_startup_event(app: FastAPI) -> AsyncGenerator[None, None]:
         except Exception as e:
             verbose_proxy_logger.error("Error stopping DB health watchdog task: %s", e)
 
+    await _drain_spend_event_producer_on_shutdown()
+
+    await flush_spend_counters_on_shutdown()
+
     await _flush_spend_logs_queue_on_shutdown()
 
     await proxy_config.stop_config_sync_subscriber()
@@ -2457,16 +2477,27 @@ def load_from_azure_key_vault(use_azure_key_vault: bool = False):
         )
 
 
+spend_event_producer: SpendEventProducer | None = None
+
+
 def cost_tracking():
-    global prisma_client
+    global prisma_client, spend_event_producer
     if prisma_client is not None:
         from litellm.integrations.shadow_eval_logger import ShadowEvalLogger
 
-        litellm.logging_callback_manager.add_litellm_callback(_ProxyDBLogger())
-        litellm.logging_callback_manager.add_litellm_async_success_callback(_ProxyDBLogger())
+        spend_event_producer = build_spend_event_producer(CollectorSettings(), fallback=run_spend_event)
+        litellm.logging_callback_manager.add_litellm_callback(_ProxyDBLogger(spend_event_producer))
+        litellm.logging_callback_manager.add_litellm_async_success_callback(_ProxyDBLogger(spend_event_producer))
         litellm.logging_callback_manager.add_litellm_callback(ShadowEvalLogger())
 
 
+async def _drain_spend_event_producer_on_shutdown() -> None:
+    if spend_event_producer is None:
+        return
+    await spend_event_producer.close(drain_timeout=CollectorSettings().drain_timeout_seconds)
+    verbose_proxy_logger.info("collector: producer drained on shutdown. stats=%s", spend_event_producer.stats())
+
+
 # Bounds authoritative DB re-reads when enforcing a budget against a
 # stale-low spend counter: at most one DB read per counter per window.
 SPEND_DB_FLOOR_CACHE_TTL_SECONDS: Final = 5

litellm/proxy/spend_tracking/spend_event.py

@@ -0,0 +1,418 @@
+"""Compact, typed success event handed from an inference worker to the collector.
+
+``build_spend_event`` runs on the inference worker right after ``Logging.async_success_handler``
+has built the ``standard_logging_object`` (so the cost is already known). It validates the success
+callback's ``kwargs`` into the projection ``_PROXY_track_cost_callback`` and
+``DBSpendUpdateWriter.update_database`` actually read: identities and metadata, timings, usage, the
+standard logging payload without its prompt/response bodies, and the tool names. The request
+messages, the raw ``proxy_server_request`` body and the full response travel only when spend logs
+are configured to store prompts and responses. The cache key is the preset key the caching layer
+already computed, never a fresh hash over the request body.
+
+``spend_event_callback_args`` rebuilds the ``(kwargs, response_obj, start_time, end_time)`` tuple
+the existing cost pipeline consumes, so the sidecar runs the unchanged pipeline against the event.
+"""
+
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+from datetime import datetime
+from types import MappingProxyType
+from typing import Final, Literal, TypeAlias
+
+from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
+from typing_extensions import NotRequired, ReadOnly, TypedDict
+
+import litellm
+from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.db.spend_log_tool_index import response_tool_call_names
+from litellm.types.interactions import InteractionsAPIResponse
+from litellm.types.utils import LiteLLMBatch, Usage
+
+SPEND_EVENT_VERSION: Final = 1
+CACHE_OFF_KEY: Final = "Cache OFF"
+
+ObjectMapping: TypeAlias = Mapping[str, object]
+
+_UNSERIALIZABLE_METADATA_KEYS: Final = frozenset({"user_api_key_auth", "litellm_parent_otel_span"})
+_STANDARD_LOGGING_BODY_KEYS: Final = frozenset({"messages", "response"})
+_STANDARD_LOGGING_DROPPED_KEYS: Final = frozenset({"model_parameters"})
+_NOT_OFFLOADED_RESPONSE_TYPES: Final = (LiteLLMBatch, InteractionsAPIResponse)
+
+
+class _LitellmParams(TypedDict, total=False):
+    api_base: ReadOnly[str | None]
+    custom_llm_provider: ReadOnly[str | None]
+    litellm_call_id: ReadOnly[str | None]
+    user_api_key_end_user_id: ReadOnly[str | None]
+    metadata: ReadOnly[ObjectMapping | None]
+    litellm_metadata: ReadOnly[ObjectMapping | None]
+    proxy_server_request: ReadOnly[ObjectMapping | None]
+    preset_cache_key: ReadOnly[str | None]
+
+
+class _DynamicParams(TypedDict, total=False):
+    turn_off_message_logging: ReadOnly[bool | None]
+
+
+class _RequestBody(TypedDict, total=False):
+    tools: ReadOnly[Sequence[ObjectMapping] | None]
+
+
+class _PassthroughPayload(TypedDict, total=False):
+    request_body: ReadOnly[_RequestBody | None]
+
+
+class _ToolCallFunction(TypedDict):
+    name: ReadOnly[str]
+    arguments: ReadOnly[str]
+
+
+class _ToolCall(TypedDict):
+    id: ReadOnly[str | None]
+    type: ReadOnly[Literal["function"]]
+    function: ReadOnly[_ToolCallFunction]
+
+
+class _ToolCallMessage(TypedDict):
+    role: ReadOnly[Literal["assistant"]]
+    content: ReadOnly[None]
+    tool_calls: ReadOnly[Sequence[_ToolCall]]
+
+
+class _ToolCallChoice(TypedDict):
+    index: ReadOnly[int]
+    finish_reason: ReadOnly[Literal["tool_calls"]]
+    message: ReadOnly[_ToolCallMessage]
+
+
+class CompactResponse(TypedDict, total=False):
+    """What the spend pipeline reads off a response: its id, usage and which tools it called."""
+
+    id: ReadOnly[object]
+    model: ReadOnly[object]
+    usage: ReadOnly[object]
+    usage_info: ReadOnly[object]
+    status: ReadOnly[object]
+    background: ReadOnly[object]
+    choices: ReadOnly[Sequence[_ToolCallChoice]]
+
+
+class _SuccessKwargs(TypedDict, total=False):
+    """The success callback's ``kwargs`` (``Logging.model_call_details``), validated and projected."""
+
+    litellm_call_id: ReadOnly[str | None]
+    call_type: ReadOnly[str | None]
+    model: ReadOnly[str | None]
+    custom_llm_provider: ReadOnly[str | None]
+    stream: ReadOnly[bool | None]
+    complete_streaming_response: ReadOnly[object]
+    cache_hit: ReadOnly[bool | None]
+    response_cost: ReadOnly[float | None]
+    completion_start_time: ReadOnly[datetime | None]
+    agent_id: ReadOnly[str | None]
+    litellm_trace_id: ReadOnly[str | None]
+    litellm_params: ReadOnly[_LitellmParams]
+    standard_logging_object: ReadOnly[ObjectMapping | None]
+    standard_callback_dynamic_params: ReadOnly[_DynamicParams | None]
+    combined_usage_object: ReadOnly[Usage | None]
+    realtime_tools: ReadOnly[Sequence[object] | None]
+    realtime_tool_calls: ReadOnly[Sequence[object] | None]
+    tools: ReadOnly[Sequence[ObjectMapping] | None]
+    passthrough_logging_payload: ReadOnly[_PassthroughPayload | None]
+
+
+class _FunctionToolFunction(TypedDict):
+    name: ReadOnly[str]
+
+
+class _FunctionTool(TypedDict):
+    type: ReadOnly[Literal["function"]]
+    function: ReadOnly[_FunctionToolFunction]
+
+
+class SpendCallbackKwargs(TypedDict):
+    """The ``kwargs`` handed to ``_PROXY_track_cost_callback`` on the sidecar."""
+
+    litellm_call_id: ReadOnly[str | None]
+    call_type: ReadOnly[str | None]
+    model: ReadOnly[str | None]
+    custom_llm_provider: ReadOnly[str | None]
+    stream: ReadOnly[bool | None]
+    cache_hit: ReadOnly[bool | None]
+    response_cost: ReadOnly[float | None]
+    completion_start_time: ReadOnly[datetime | None]
+    agent_id: ReadOnly[str | None]
+    litellm_trace_id: ReadOnly[str | None]
+    litellm_params: ReadOnly[_LitellmParams]
+    standard_logging_object: ReadOnly[ObjectMapping | None]
+    standard_callback_dynamic_params: ReadOnly[_DynamicParams | None]
+    combined_usage_object: ReadOnly[Usage | None]
+    realtime_tools: ReadOnly[Sequence[object] | None]
+    realtime_tool_calls: ReadOnly[Sequence[object] | None]
+    tools: ReadOnly[Sequence[_FunctionTool] | None]
+    complete_streaming_response: NotRequired[ReadOnly[CompactResponse | None]]
+
+
+_NO_LITELLM_PARAMS: Final[_LitellmParams] = {}
+_SUCCESS_KWARGS: Final = TypeAdapter(_SuccessKwargs)
+_OBJECT_MAPPING: Final = TypeAdapter(ObjectMapping)
+_COMPACT_RESPONSE: Final = TypeAdapter(CompactResponse)
+
+
+class SpendEvent(BaseModel):
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    version: Literal[1]
+    litellm_call_id: str | None
+    call_type: str | None
+    model: str | None
+    custom_llm_provider: str | None
+    stream: bool | None
+    complete_streaming_response: bool
+    cache_hit: bool | None
+    response_cost: float | None
+    start_time: datetime
+    end_time: datetime
+    completion_start_time: datetime | None
+    agent_id: str | None
+    litellm_trace_id: str | None
+    litellm_params: _LitellmParams
+    standard_logging_object: ObjectMapping | None
+    standard_callback_dynamic_params: _DynamicParams | None
+    response: CompactResponse | None
+    combined_usage: ObjectMapping | None
+    realtime_tools: Sequence[object] | None
+    realtime_tool_calls: Sequence[object] | None
+    request_tool_names: tuple[str, ...]
+
+
+@dataclass(frozen=True, slots=True)
+class SpendEventCallbackArgs:
+    kwargs: SpendCallbackKwargs
+    response_obj: CompactResponse | None
+    start_time: datetime
+    end_time: datetime
+
+
+@dataclass(frozen=True, slots=True)
+class SpendEventBuildError:
+    reason: str
+
+
+@dataclass(frozen=True, slots=True)
+class SpendEventDecodeError:
+    reason: str
+
+
+def is_offloadable_success(response_obj: object) -> bool:
+    """Batch retrieves and interaction polls branch on the concrete response class, so they stay in-process."""
+    return not isinstance(response_obj, _NOT_OFFLOADED_RESPONSE_TYPES)
+
+
+def _json_fallback(value: object) -> str:
+    return str(value)
+
+
+def _mapping_or_none(value: object) -> ObjectMapping | None:
+    try:
+        return _OBJECT_MAPPING.validate_python(value)
+    except ValidationError:
+        return None
+
+
+def _drop_keys(mapping: ObjectMapping, keys: frozenset[str]) -> ObjectMapping:
+    return MappingProxyType({key: value for key, value in mapping.items() if key not in keys})
+
+
+def _budget_reservation(metadata: ObjectMapping) -> ObjectMapping | None:
+    """The admission-time reservation, wherever the request setup left it, so the sidecar can reconcile it."""
+    direct: Final = _mapping_or_none(metadata.get("user_api_key_budget_reservation"))
+    if direct is not None:
+        return direct
+    auth: Final = metadata.get("user_api_key_auth")
+    if isinstance(auth, UserAPIKeyAuth):
+        return auth.budget_reservation
+    auth_mapping: Final = _mapping_or_none(auth)
+    return _mapping_or_none(auth_mapping.get("budget_reservation")) if auth_mapping is not None else None
+
+
+def _metadata_for_event(
+    metadata: ObjectMapping | None, budget_reservation: ObjectMapping | None
+) -> ObjectMapping | None:
+    if metadata is None:
+        return None
+    kept: Final = _drop_keys(metadata, _UNSERIALIZABLE_METADATA_KEYS)
+    if budget_reservation is None:
+        return kept
+    return MappingProxyType({**kept, "user_api_key_budget_reservation": budget_reservation})
+
+
+def _litellm_params_for_event(
+    litellm_params: _LitellmParams, cache_key: str | None, store_bodies: bool
+) -> _LitellmParams:
+    metadata: Final = litellm_params.get("metadata")
+    litellm_metadata: Final = litellm_params.get("litellm_metadata")
+    budget_reservation: Final = next(
+        (
+            reservation
+            for source in (litellm_metadata, metadata)
+            if source is not None and (reservation := _budget_reservation(source)) is not None
+        ),
+        None,
+    )
+    projected: Final[_LitellmParams] = {
+        "api_base": litellm_params.get("api_base"),
+        "custom_llm_provider": litellm_params.get("custom_llm_provider"),
+        "litellm_call_id": litellm_params.get("litellm_call_id"),
+        "user_api_key_end_user_id": litellm_params.get("user_api_key_end_user_id"),
+        "metadata": _metadata_for_event(metadata, budget_reservation),
+        "litellm_metadata": _metadata_for_event(litellm_metadata, budget_reservation),
+        "proxy_server_request": litellm_params.get("proxy_server_request") if store_bodies else None,
+        "preset_cache_key": cache_key,
+    }
+    return projected
+
+
+def _standard_logging_for_event(sl_object: ObjectMapping | None, store_bodies: bool) -> ObjectMapping | None:
+    if sl_object is None:
+        return None
+    dropped: Final = (
+        _STANDARD_LOGGING_DROPPED_KEYS if store_bodies else _STANDARD_LOGGING_DROPPED_KEYS | _STANDARD_LOGGING_BODY_KEYS
+    )
+    return _drop_keys(sl_object, dropped)
+
+
+def _tool_call(name: str) -> _ToolCall:
+    tool_call: Final[_ToolCall] = {"id": None, "type": "function", "function": {"name": name, "arguments": "{}"}}
+    return tool_call
+
+
+def _tool_call_choice(names: Sequence[str]) -> _ToolCallChoice:
+    choice: Final[_ToolCallChoice] = {
+        "index": 0,
+        "finish_reason": "tool_calls",
+        "message": {"role": "assistant", "content": None, "tool_calls": tuple(_tool_call(name) for name in names)},
+    }
+    return choice
+
+
+def _compact_response(response_obj: object) -> CompactResponse | None:
+    """Usage, identity and tool calls of the response, in chat-completions shape, without the content."""
+    dumped: Final = response_obj.model_dump() if isinstance(response_obj, BaseModel) else _mapping_or_none(response_obj)
+    if dumped is None:
+        return None
+    scalars: Final = _COMPACT_RESPONSE.validate_python(_drop_keys(dumped, frozenset({"choices"})))
+    tool_call_names: Final = response_tool_call_names(response_obj)
+    if not tool_call_names:
+        return scalars
+    with_tool_calls: Final[CompactResponse] = {**scalars, "choices": (_tool_call_choice(tool_call_names),)}
+    return with_tool_calls
+
+
+def _tool_name(tool: ObjectMapping) -> str | None:
+    """Chat tools nest the name under ``function``; Anthropic and Responses API tools keep it at the top."""
+    function: Final = _mapping_or_none(tool.get("function"))
+    name: Final = function.get("name") if function is not None else tool.get("name")
+    return name.strip() if isinstance(name, str) and name.strip() else None
+
+
+def _request_tool_names(kwargs: _SuccessKwargs) -> tuple[str, ...]:
+    passthrough: Final = kwargs.get("passthrough_logging_payload")
+    request_body: Final = passthrough.get("request_body") if passthrough is not None else None
+    passthrough_tools: Final = request_body.get("tools") if request_body is not None else None
+    return tuple(
+        name
+        for source in (kwargs.get("tools"), passthrough_tools)
+        if source is not None
+        for tool in source
+        if (name := _tool_name(tool)) is not None
+    )
+
+
+def preset_spend_log_cache_key(litellm_params: _LitellmParams) -> str | None:
+    """The key the caching layer already stored in ``litellm_params``, or ``Cache OFF``; never hashes the body."""
+    if litellm.cache is None:
+        return CACHE_OFF_KEY
+    return litellm_params.get("preset_cache_key")
+
+
+def _function_tool(name: str) -> _FunctionTool:
+    tool: Final[_FunctionTool] = {"type": "function", "function": {"name": name}}
+    return tool
+
+
+def build_spend_event(
+    raw_kwargs: ObjectMapping, response_obj: object, start_time: datetime, end_time: datetime, store_bodies: bool
+) -> bytes | SpendEventBuildError:
+    """Validate the success callback's kwargs and serialize the event once, as a single JSON line."""
+    try:
+        kwargs: Final = _SUCCESS_KWARGS.validate_python(raw_kwargs)
+    except ValidationError as error:
+        return SpendEventBuildError(reason=str(error))
+    litellm_params: Final = kwargs.get("litellm_params", _NO_LITELLM_PARAMS)
+    sl_object: Final = kwargs.get("standard_logging_object")
+    cache_key: Final = preset_spend_log_cache_key(litellm_params)
+    response_cost: Final = sl_object.get("response_cost") if sl_object is not None else kwargs.get("response_cost")
+    combined_usage: Final = kwargs.get("combined_usage_object")
+    event: Final = SpendEvent(
+        version=SPEND_EVENT_VERSION,
+        litellm_call_id=kwargs.get("litellm_call_id"),
+        call_type=kwargs.get("call_type"),
+        model=kwargs.get("model"),
+        custom_llm_provider=kwargs.get("custom_llm_provider"),
+        stream=kwargs.get("stream"),
+        complete_streaming_response="complete_streaming_response" in kwargs,
+        cache_hit=kwargs.get("cache_hit"),
+        response_cost=response_cost if isinstance(response_cost, (int, float)) else None,
+        start_time=start_time,
+        end_time=end_time,
+        completion_start_time=kwargs.get("completion_start_time"),
+        agent_id=kwargs.get("agent_id"),
+        litellm_trace_id=kwargs.get("litellm_trace_id"),
+        litellm_params=_litellm_params_for_event(litellm_params, cache_key, store_bodies),
+        standard_logging_object=_standard_logging_for_event(sl_object, store_bodies),
+        standard_callback_dynamic_params=kwargs.get("standard_callback_dynamic_params"),
+        response=_compact_response(response_obj),
+        combined_usage=combined_usage.model_dump() if combined_usage is not None else None,
+        realtime_tools=kwargs.get("realtime_tools"),
+        realtime_tool_calls=kwargs.get("realtime_tool_calls"),
+        request_tool_names=_request_tool_names(kwargs),
+    )
+    return event.model_dump_json(fallback=_json_fallback).encode() + b"\n"
+
+
+def decode_spend_event(line: bytes) -> SpendEvent | SpendEventDecodeError:
+    try:
+        return SpendEvent.model_validate_json(line)
+    except ValidationError as error:
+        return SpendEventDecodeError(reason=str(error))
+
+
+def spend_event_callback_args(event: SpendEvent) -> SpendEventCallbackArgs:
+    """The ``(kwargs, response_obj, start_time, end_time)`` the in-process cost callback receives."""
+    tools: Final = tuple(_function_tool(name) for name in event.request_tool_names)
+    kwargs: Final[SpendCallbackKwargs] = {
+        "litellm_call_id": event.litellm_call_id,
+        "call_type": event.call_type,
+        "model": event.model,
+        "custom_llm_provider": event.custom_llm_provider,
+        "stream": event.stream,
+        "cache_hit": event.cache_hit,
+        "response_cost": event.response_cost,
+        "completion_start_time": event.completion_start_time,
+        "agent_id": event.agent_id,
+        "litellm_trace_id": event.litellm_trace_id,
+        "litellm_params": event.litellm_params,
+        "standard_logging_object": event.standard_logging_object,
+        "standard_callback_dynamic_params": event.standard_callback_dynamic_params,
+        "combined_usage_object": Usage.model_validate(event.combined_usage)
+        if event.combined_usage is not None
+        else None,
+        "realtime_tools": event.realtime_tools,
+        "realtime_tool_calls": event.realtime_tool_calls,
+        "tools": tools or None,
+    }
+    if not event.complete_streaming_response:
+        return SpendEventCallbackArgs(kwargs, event.response, event.start_time, event.end_time)
+    streaming_kwargs: Final[SpendCallbackKwargs] = {**kwargs, "complete_streaming_response": event.response}
+    return SpendEventCallbackArgs(streaming_kwargs, event.response, event.start_time, event.end_time)

litellm/proxy/spend_tracking/spend_event_producer.py

@@ -0,0 +1,338 @@
+"""Fire-and-forget push of serialized spend events from an inference worker to the pod-local sidecar.
+
+``LITELLM_COLLECTOR_ENABLED=true`` turns the push on in the gateway; the sidecar process sets
+``LITELLM_JOB_ROLE=collector`` and always runs the pipeline in-process. Events queue in a bounded
+in-memory buffer that a single writer task flushes over a unix socket or loopback TCP connection.
+When the sidecar is unreachable, the buffer is full, or the connection breaks mid-write, each affected
+event follows ``LITELLM_COLLECTOR_ON_UNAVAILABLE``: ``fallback`` runs the existing cost pipeline in
+the worker, ``drop`` counts it and moves on. Transitions are logged with the counters, so a sidecar
+outage is visible without scraping anything.
+
+Delivery is at-most-once: a sidecar crash loses the events the kernel already took from its socket.
+A sidecar that stops gracefully half-closes each connection first (EOF towards the producer) and
+keeps reading until the producer hangs up, so the producer switches to the unavailable policy without
+losing the events in flight. A write that fails part-way follows the unavailable policy without double
+counting: ``drain()`` only fails while part of the line is still buffered in this process, so the
+sidecar can at most have read a truncated line, which it discards. When the gateway itself stops with
+the writer stuck mid-send, only an event whose bytes are still in the producer's write buffer follows
+the unavailable policy; the connection is aborted first so the sidecar discards the truncated line
+instead of also counting it. Events from one uvicorn worker are handled in the order it produced them;
+events from different workers interleave, exactly like the in-process callbacks do today.
+"""
+
+import asyncio
+import ipaddress
+import time
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass
+from typing import Final, Literal, TypeAlias
+from urllib.parse import urlsplit
+
+from pydantic import AliasChoices, Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+from litellm._logging import verbose_proxy_logger
+
+COLLECTOR_ENV_PREFIX: Final = "LITELLM_COLLECTOR_"
+COLLECTOR_JOB_ROLE: Final = "collector"
+DEFAULT_COLLECTOR_ADDRESS: Final = "unix:///var/run/litellm/collector.sock"
+RECONNECT_BACKOFF_SECONDS: Final = 1.0
+DROP_LOG_EVERY: Final = 1000
+
+UnavailablePolicy: TypeAlias = Literal["fallback", "drop"]
+PublishOutcome: TypeAlias = Literal["queued", "fallback", "dropped"]
+
+
+class CollectorSettings(BaseSettings):
+    """``LITELLM_COLLECTOR_*`` env vars, shared by the gateway producer and the sidecar consumer."""
+
+    model_config = SettingsConfigDict(
+        env_prefix=COLLECTOR_ENV_PREFIX, case_sensitive=False, extra="ignore", frozen=True, populate_by_name=True
+    )
+
+    enabled: bool = False
+    address: str = DEFAULT_COLLECTOR_ADDRESS
+    buffer_size: int = Field(default=1000, ge=1)
+    on_unavailable: UnavailablePolicy = "fallback"
+    drain_timeout_seconds: float = Field(default=10.0, gt=0)
+    connect_timeout_seconds: float = Field(default=1.0, gt=0)
+    job_role: str | None = Field(default=None, validation_alias=AliasChoices("LITELLM_JOB_ROLE"))
+
+    @property
+    def produces(self) -> bool:
+        return self.enabled and self.job_role != COLLECTOR_JOB_ROLE
+
+
+@dataclass(frozen=True, slots=True)
+class UnixAddress:
+    path: str
+
+
+@dataclass(frozen=True, slots=True)
+class TcpAddress:
+    host: str
+    port: int
+
+
+@dataclass(frozen=True, slots=True)
+class AddressError:
+    reason: str
+
+
+CollectorAddress: TypeAlias = UnixAddress | TcpAddress
+
+
+def _is_loopback(host: str) -> bool:
+    try:
+        return ipaddress.ip_address(host).is_loopback
+    except ValueError:
+        return host == "localhost"
+
+
+def parse_collector_address(address: str) -> CollectorAddress | AddressError:
+    """``unix:///path/to.sock`` or ``tcp://127.0.0.1:port``; the socket carries unauthenticated spend events."""
+    parsed: Final = urlsplit(address)
+    if parsed.scheme == "unix" and parsed.path:
+        return UnixAddress(path=parsed.path)
+    if parsed.scheme == "tcp" and parsed.hostname and parsed.port is not None:
+        if not _is_loopback(parsed.hostname):
+            return AddressError(reason=f"tcp collector address must be a loopback host, got {address!r}")
+        return TcpAddress(host=parsed.hostname, port=parsed.port)
+    return AddressError(reason=f"expected unix:///path or tcp://127.0.0.1:port, got {address!r}")
+
+
+async def open_collector_connection(
+    address: CollectorAddress, timeout: float
+) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]:
+    match address:
+        case UnixAddress(path=path):
+            return await asyncio.wait_for(asyncio.open_unix_connection(path), timeout)
+        case TcpAddress(host=host, port=port):
+            return await asyncio.wait_for(asyncio.open_connection(host, port), timeout)
+
+
+def build_spend_event_producer(
+    settings: CollectorSettings, fallback: Callable[[bytes], Awaitable[None]]
+) -> "SpendEventProducer | None":
+    """The gateway producer for these settings, or ``None`` when the pipeline stays in-process."""
+    if not settings.produces:
+        return None
+    address: Final = parse_collector_address(settings.address)
+    if isinstance(address, AddressError):
+        verbose_proxy_logger.error("collector: %s; running the spend pipeline in-process", address.reason)
+        return None
+    verbose_proxy_logger.info(
+        "collector: offloading spend tracking to %s (buffer=%d, on_unavailable=%s)",
+        settings.address,
+        settings.buffer_size,
+        settings.on_unavailable,
+    )
+    return SpendEventProducer(
+        address=address,
+        on_unavailable=settings.on_unavailable,
+        buffer_size=settings.buffer_size,
+        connect_timeout=settings.connect_timeout_seconds,
+        fallback=fallback,
+    )
+
+
+@dataclass(frozen=True, slots=True)
+class _Connection:
+    reader: asyncio.StreamReader
+    writer: asyncio.StreamWriter
+
+    @property
+    def alive(self) -> bool:
+        return not self.writer.is_closing() and not self.reader.at_eof()
+
+
+@dataclass(frozen=True, slots=True)
+class SpendEventProducerStats:
+    queued: int
+    sent: int
+    fallback: int
+    dropped: int
+    connected: bool
+
+
+class SpendEventProducer:
+    """Bounded buffer plus one writer task per process; see the module docstring for the contract."""
+
+    def __init__(
+        self,
+        address: CollectorAddress,
+        on_unavailable: UnavailablePolicy,
+        buffer_size: int,
+        connect_timeout: float,
+        fallback: Callable[[bytes], Awaitable[None]],
+        clock: Callable[[], float] = time.monotonic,
+        open_connection: Callable[
+            [CollectorAddress, float], Awaitable[tuple[asyncio.StreamReader, asyncio.StreamWriter]]
+        ] = open_collector_connection,
+    ) -> None:
+        self._address = address
+        self._on_unavailable = on_unavailable
+        self._buffer_size = buffer_size
+        self._connect_timeout = connect_timeout
+        self._fallback = fallback
+        self._clock = clock
+        self._open_connection = open_connection
+        self._queue: asyncio.Queue[bytes] | None = None
+        self._writer_task: asyncio.Task[None] | None = None
+        self._connection: _Connection | None = None
+        self._in_flight: bytes | None = None
+        self._closing = False
+        self._next_connect_at = 0.0
+        self._queued = 0
+        self._sent = 0
+        self._fallback_count = 0
+        self._dropped = 0
+
+    def stats(self) -> SpendEventProducerStats:
+        return SpendEventProducerStats(
+            queued=self._queued,
+            sent=self._sent,
+            fallback=self._fallback_count,
+            dropped=self._dropped,
+            connected=self._connection is not None,
+        )
+
+    async def publish(self, line: bytes) -> PublishOutcome:
+        """Hand one serialized event to the writer task, or apply the unavailable policy right away."""
+        if self._closing or self._clock() < self._next_connect_at:
+            return await self._unavailable(line, "sidecar unreachable")
+        queue: Final = self._ensure_writer()
+        try:
+            queue.put_nowait(line)
+        except asyncio.QueueFull:
+            return await self._unavailable(line, "buffer full")
+        self._queued += 1
+        return "queued"
+
+    async def close(self, drain_timeout: float) -> None:
+        """Flush the buffer for up to ``drain_timeout`` seconds, then apply the unavailable policy to the rest."""
+        self._closing = True
+        queue: Final = self._queue
+        task: Final = self._writer_task
+        if queue is None or task is None:
+            return
+        try:
+            await asyncio.wait_for(queue.join(), drain_timeout)
+        except asyncio.TimeoutError:
+            verbose_proxy_logger.warning(
+                "collector: %s events still buffered after %.1fs drain timeout", queue.qsize(), drain_timeout
+            )
+        task.cancel()
+        try:
+            await task
+        except asyncio.CancelledError:
+            pass
+        unsent: Final = self._take_unsent()
+        await self._disconnect()
+        if unsent is not None:
+            await self._unavailable(unsent, "shutdown")
+        while not queue.empty():
+            await self._unavailable(queue.get_nowait(), "shutdown")
+
+    def _take_unsent(self) -> bytes | None:
+        """The in-flight event if any of its bytes never left this process, aborting the half-written connection."""
+        in_flight: Final = self._in_flight
+        self._in_flight = None
+        connection: Final = self._connection
+        if in_flight is None:
+            return None
+        if connection is None:
+            return in_flight
+        if connection.writer.transport.get_write_buffer_size() == 0:
+            return None
+        connection.writer.transport.abort()
+        return in_flight
+
+    def _ensure_writer(self) -> asyncio.Queue[bytes]:
+        if self._queue is None:
+            self._queue = asyncio.Queue(maxsize=self._buffer_size)
+        if self._writer_task is None or self._writer_task.done():
+            self._writer_task = asyncio.get_running_loop().create_task(self._run_writer(self._queue))
+        return self._queue
+
+    async def _run_writer(self, queue: asyncio.Queue[bytes]) -> None:
+        while True:
+            line = await queue.get()
+            try:
+                await self._send(line)
+            finally:
+                queue.task_done()
+
+    async def _send(self, line: bytes) -> None:
+        self._in_flight = line
+        connection: Final = await self._connect()
+        if connection is None:
+            self._in_flight = None
+            await self._unavailable(line, "sidecar unreachable")
+            return
+        try:
+            connection.writer.write(line)
+            await connection.writer.drain()
+        except (ConnectionError, OSError, RuntimeError) as error:  # uvloop: RuntimeError on a closed transport
+            self._in_flight = None
+            await self._disconnect()
+            self._next_connect_at = self._clock() + RECONNECT_BACKOFF_SECONDS
+            await self._unavailable(line, f"write failed: {error}")
+            return
+        self._in_flight = None
+        self._sent += 1
+
+    async def _connect(self) -> _Connection | None:
+        if self._connection is not None and self._connection.alive:
+            return self._connection
+        await self._disconnect()
+        if self._clock() < self._next_connect_at:
+            return None
+        try:
+            reader, writer = await self._open_connection(self._address, self._connect_timeout)
+        except (ConnectionError, OSError, asyncio.TimeoutError) as error:
+            self._next_connect_at = self._clock() + RECONNECT_BACKOFF_SECONDS
+            verbose_proxy_logger.warning(
+                "collector: cannot reach %s (%s); applying %s policy for %.0fs. stats=%s",
+                self._address,
+                error,
+                self._on_unavailable,
+                RECONNECT_BACKOFF_SECONDS,
+                self.stats(),
+            )
+            return None
+        self._connection = _Connection(reader=reader, writer=writer)
+        verbose_proxy_logger.info("collector: connected to %s. stats=%s", self._address, self.stats())
+        return self._connection
+
+    async def _disconnect(self) -> None:
+        connection: Final = self._connection
+        self._connection = None
+        if connection is None:
+            return
+        connection.writer.close()
+        try:
+            await connection.writer.wait_closed()
+        except (ConnectionError, OSError):
+            pass
+
+    async def _unavailable(self, line: bytes, reason: str) -> PublishOutcome:
+        if self._on_unavailable == "fallback":
+            self._fallback_count += 1
+            fallback: Final = asyncio.ensure_future(self._run_fallback(line, reason))
+            try:
+                await asyncio.shield(fallback)
+            except asyncio.CancelledError:
+                await fallback
+                raise
+            return "fallback"
+        self._dropped += 1
+        if self._dropped % DROP_LOG_EVERY == 1:
+            verbose_proxy_logger.warning("collector: dropping spend event (%s). stats=%s", reason, self.stats())
+        return "dropped"
+
+    async def _run_fallback(self, line: bytes, reason: str) -> None:
+        try:
+            await self._fallback(line)
+        except Exception:  # noqa: BLE001  # one failing event must not kill the writer task
+            verbose_proxy_logger.exception("collector: in-process fallback failed (%s)", reason)

litellm/proxy/spend_tracking/spend_tracking_utils.py

@@ -554,10 +554,12 @@ def get_logging_payload(kwargs, response_obj, start_time, end_time) -> SpendLogs
                 additional_usage_values["cache_creation_input_tokens"] = cache_write_tokens
     clean_metadata["additional_usage_values"] = additional_usage_values
 
-    if litellm.cache is not None:
-        cache_key = litellm.cache.get_cache_key(**kwargs)
-    else:
+    if litellm.cache is None:
         cache_key = "Cache OFF"
+    elif litellm_params.get("preset_cache_key") is not None:
+        cache_key = litellm_params["preset_cache_key"]
+    else:
+        cache_key = litellm.cache.get_cache_key(**kwargs)
     if cache_hit is True:
         import time
 
@@ -864,7 +866,7 @@ def _get_messages_for_spend_logs_payload(
     standard_logging_payload: StandardLoggingPayload | None,
     metadata: dict | None = None,
 ) -> str:
-    if _should_store_prompts_and_responses_in_spend_logs():
+    if should_store_prompts_and_responses_in_spend_logs():
         if standard_logging_payload is not None:
             call_type: Final = standard_logging_payload.get("call_type", "")
             if call_type == "_arealtime":
@@ -1114,7 +1116,7 @@ def _sanitize_guardrail_information_for_spend_logs(
     here to match OTEL's defensive read pattern; otherwise iteration would
     yield the dict's keys and crash the whole spend-log write.
     """
-    if guardrail_information is None or _should_store_prompts_and_responses_in_spend_logs():
+    if guardrail_information is None or should_store_prompts_and_responses_in_spend_logs():
         return guardrail_information
     entries: Final = [guardrail_information] if isinstance(guardrail_information, dict) else guardrail_information
     return [_redact_prompt_fields_in_guardrail_entry(entry) for entry in entries if isinstance(entry, dict)]
@@ -1186,7 +1188,7 @@ def _sanitize_error_information_for_spend_logs(
 
     sanitized = cast(dict, {**error_information})
 
-    if not _should_store_prompts_and_responses_in_spend_logs():
+    if not should_store_prompts_and_responses_in_spend_logs():
         for field in ("error_message", "traceback"):
             value = sanitized.get(field)
             if isinstance(value, str):
@@ -1263,11 +1265,11 @@ def _get_proxy_server_request_for_spend_logs_payload(
     kwargs: dict | None = None,
 ) -> str:
     """
-    Only store if _should_store_prompts_and_responses_in_spend_logs() is True
+    Only store if should_store_prompts_and_responses_in_spend_logs() is True
 
     If turn_off_message_logging is enabled, redact messages in the request body.
     """
-    if _should_store_prompts_and_responses_in_spend_logs():
+    if should_store_prompts_and_responses_in_spend_logs():
         _proxy_server_request: Final = cast(dict | None, litellm_params.get("proxy_server_request", EMPTY_MAPPING))
         if _proxy_server_request is not None:
             _request_body = _proxy_server_request.get("body", EMPTY_MAPPING) or EMPTY_MAPPING
@@ -1317,7 +1319,7 @@ def _get_vector_store_request_for_spend_logs_payload(
     """
     If user does not want to store prompts and responses, then remove the content from the vector store request metadata
     """
-    if _should_store_prompts_and_responses_in_spend_logs():
+    if should_store_prompts_and_responses_in_spend_logs():
         return vector_store_request_metadata
 
     # if user does not want to store prompts and responses, then remove the content from the vector store request metadata
@@ -1341,7 +1343,7 @@ def _get_response_for_spend_logs_payload(
 ) -> str:
     if payload is None:
         return "{}"
-    if _should_store_prompts_and_responses_in_spend_logs():
+    if should_store_prompts_and_responses_in_spend_logs():
         response_obj: object = payload.get("response")
         if response_obj is None:
             return "{}"
@@ -1389,7 +1391,7 @@ def _get_response_for_spend_logs_payload(
     return "{}"
 
 
-def _should_store_prompts_and_responses_in_spend_logs() -> bool:
+def should_store_prompts_and_responses_in_spend_logs() -> bool:
     from litellm.proxy.proxy_server import general_settings
     from litellm.secret_managers.main import get_secret_bool
 

litellm/utils.py

@@ -1843,6 +1843,9 @@ def client(original_function):
                 elif _caching_handler_response.embedding_all_elements_cache_hit is True:
                     return _caching_handler_response.final_embedding_cached_response
 
+            if _llm_caching_handler.preset_cache_key is not None:
+                logging_obj.litellm_params["preset_cache_key"] = _llm_caching_handler.preset_cache_key
+
             # CHECK MAX TOKENS
             if (
                 kwargs.get("max_tokens", None) is not None

terraform/litellm/README.md

@@ -183,6 +183,7 @@ only where the underlying cloud forces it.
 | Extra secret-backed env          | `gateway_extra_secrets`, `backend_extra_secrets` (ARNs) | `gateway_extra_secrets`, `backend_extra_secrets` (resource IDs) |
 | Uvicorn `--workers` on gateway   | `gateway_num_workers`                                   | `gateway_num_workers`                                     |
 | OpenTelemetry v2 (opt-in)        | `otel_endpoint`, `otel_exporter`, `otel_environment_name`, `otel_capture_message_content`, `otel_headers_secret_arn` | `otel_endpoint`, `otel_exporter`, `otel_environment_name`, `otel_capture_message_content`, `otel_headers_secret` |
+| Collector sidecar (opt-in)    | `collector_enabled`, `collector_port`, `collector_cpu`, `collector_memory`, `collector_buffer_size`, `collector_on_unavailable`, `collector_drain_timeout_seconds` | same names; `collector_cpu` / `collector_memory` take Cloud Run strings |
 
 Each module stamps its own stack-identity tag (`litellm:stack` on AWS,
 `litellm-stack` on GCP — GCP label keys forbid colons) plus

terraform/litellm/aws/README.md

@@ -345,6 +345,39 @@ ten tasks handle 4,200,000,000 tokens in a minute, `tokens / 60` is
 `ceil(10 * 7000000 / 6000000) = 12`. Container Insights must be enabled on the
 cluster for `RunningTaskCount` to exist
 
+### Collector sidecar
+
+`collector_enabled = true` adds a second container to the gateway task
+that runs `python -m litellm.proxy.collector` from the gateway image, and sets
+`LITELLM_COLLECTOR_ENABLED=true` on the gateway so its uvicorn workers
+ship spend events (SpendLogs writes, key/team/user spend updates, budget
+alerts) to the sidecar instead of running that pipeline in the request
+path. This is the Terraform counterpart of helm's `gateway.collector`.
+The default (`false`) leaves the task definition exactly as before.
+
+Fargate tasks share one network namespace, so the sidecar listens on
+loopback TCP (`tcp://127.0.0.1:${collector_port}`, default 4010) instead
+of the Unix socket helm uses; the proxy rejects any non-loopback address.
+The sidecar gets the same database, Redis, master-key, license, proxy
+config, and `gateway_extra_env` / `gateway_extra_secrets` values as the
+gateway container, runs with `LITELLM_JOB_ROLE=collector`, and is
+non-essential with an ECS restart policy, so a sidecar crash restarts it in
+place while the gateway falls back to in-process spend tracking.
+
+```hcl
+collector_enabled = true
+# collector_cpu               = 512    # carved out of gateway_cpu
+# collector_memory            = 2048   # MiB, carved out of gateway_memory
+# collector_buffer_size       = 1000
+# collector_on_unavailable    = "fallback"  # or "drop"
+# collector_drain_timeout_seconds = 10
+```
+
+Both sidecar reservations must leave room for the gateway container inside
+`gateway_cpu` / `gateway_memory` (the plan fails otherwise). Service
+autoscaling keeps tracking the whole task's CPU and memory, sidecar
+included
+
 ## Tenant deployment
 
 Every resource the stack creates is named `${tenant}-litellm-${env}` (or

terraform/litellm/aws/ecs.tf

@@ -278,6 +278,62 @@ locals {
       "${local.proxy_config_fetch_cmd} && ${local.backend_launch_cmd}"
     ]
   } : {}
+
+  collector_address = "tcp://127.0.0.1:${var.collector_port}"
+  collector_env = var.collector_enabled ? [
+    { name = "LITELLM_COLLECTOR_ENABLED", value = "true" },
+    { name = "LITELLM_COLLECTOR_ADDRESS", value = local.collector_address },
+    { name = "LITELLM_COLLECTOR_BUFFER_SIZE", value = tostring(var.collector_buffer_size) },
+    { name = "LITELLM_COLLECTOR_ON_UNAVAILABLE", value = var.collector_on_unavailable },
+    { name = "LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS", value = tostring(var.collector_drain_timeout_seconds) },
+  ] : []
+
+  gateway_environment = concat(
+    local.shared_env,
+    local.gateway_otel_env,
+    local.billing_metrics_env,
+    local.gateway_extra_env_list,
+    local.proxy_config_env,
+    local.metrics_env,
+    local.gateway_pool_env,
+    local.collector_env,
+  )
+
+  collector_launch_cmd = "exec python -m litellm.proxy.collector"
+  collector_command = [
+    local.proxy_config_enabled ? "${local.proxy_config_fetch_cmd} && ${local.collector_launch_cmd}" : local.collector_launch_cmd
+  ]
+
+  collector_container = var.collector_enabled ? [{
+    name      = "collector"
+    image     = var.gateway_image
+    essential = false
+    cpu       = var.collector_cpu
+    memory    = var.collector_memory
+
+    restartPolicy = { enabled = true }
+
+    entryPoint = ["sh", "-c"]
+    command    = local.collector_command
+    environment = concat(
+      local.shared_env,
+      local.gateway_extra_env_list,
+      local.proxy_config_env,
+      local.gateway_pool_env,
+      local.collector_env,
+      [{ name = "LITELLM_JOB_ROLE", value = "collector" }],
+    )
+    secrets = concat(local.shared_secrets, local.gateway_extra_secrets_list)
+
+    logConfiguration = {
+      logDriver = "awslogs"
+      options = {
+        awslogs-group         = aws_cloudwatch_log_group.gateway.name
+        awslogs-region        = var.region
+        awslogs-stream-prefix = "collector"
+      }
+    }
+  }] : []
 }
 
 # ---------- Gateway ----------
@@ -309,6 +365,16 @@ resource "aws_ecs_task_definition" "gateway" {
       condition     = !var.gateway_connection_pool_enabled || local.database_enabled
       error_message = "gateway_connection_pool_enabled needs a database: set create_database = true or pass database_url."
     }
+
+    precondition {
+      condition     = !var.collector_enabled || (var.collector_cpu < var.gateway_cpu && var.collector_memory < var.gateway_memory)
+      error_message = "collector_cpu and collector_memory are carved out of gateway_cpu / gateway_memory and must leave room for the gateway container."
+    }
+
+    precondition {
+      condition     = !var.collector_enabled || var.gateway_metrics_port == null || var.collector_port != var.gateway_metrics_port
+      error_message = "collector_port and gateway_metrics_port must differ: both sidecars bind loopback in the same task."
+    }
   }
 
   family                   = "${local.name}-gateway"
@@ -327,17 +393,9 @@ resource "aws_ecs_task_definition" "gateway" {
         essential = true
 
         portMappings = [{ containerPort = 4000, protocol = "tcp" }]
-        environment = concat(
-          local.shared_env,
-          local.gateway_otel_env,
-          local.billing_metrics_env,
-          local.gateway_extra_env_list,
-          local.proxy_config_env,
-          local.metrics_env,
-          local.gateway_pool_env,
-        )
-        secrets     = concat(local.shared_secrets, local.gateway_extra_secrets_list)
-        mountPoints = local.metrics_mount_points
+        environment  = local.gateway_environment
+        secrets      = concat(local.shared_secrets, local.gateway_extra_secrets_list)
+        mountPoints  = local.metrics_mount_points
 
         # Container-level healthCheck intentionally omitted — the wolfi
         # runtime image doesn't ship curl/wget. The ALB target group polls
@@ -354,7 +412,7 @@ resource "aws_ecs_task_definition" "gateway" {
       },
       local.gateway_proxy_overrides,
     )
-  ], local.gateway_metrics_container))
+  ], local.gateway_metrics_container, local.collector_container))
 
   dynamic "volume" {
     for_each = local.metrics_enabled ? [1] : []

terraform/litellm/aws/tests/collector.tftest.hcl

@@ -0,0 +1,144 @@
+# Plan-only coverage for the opt-in collector sidecar in the gateway task.
+# The rendered container_definitions JSON is unknown at plan time (it embeds
+# Aurora/ElastiCache endpoints and secret ARNs), so the assertions target the
+# locals it is built from. Run from terraform/litellm/aws with `terraform test`.
+
+mock_provider "aws" {
+  mock_data "aws_iam_policy_document" {
+    defaults = {
+      json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}"
+    }
+  }
+}
+mock_provider "random" {}
+
+variables {
+  region              = "us-east-1"
+  tenant              = "acme"
+  env                 = "test"
+  allow_plaintext_alb = true
+  azs                 = ["us-east-1a", "us-east-1b"]
+}
+
+run "disabled_by_default_leaves_the_task_untouched" {
+  command = plan
+
+  assert {
+    condition     = length(local.collector_container) == 0
+    error_message = "The gateway task must stay single-container unless collector_enabled is set."
+  }
+
+  assert {
+    condition     = !anytrue([for e in local.gateway_environment : startswith(e.name, "LITELLM_COLLECTOR_")])
+    error_message = "No LITELLM_COLLECTOR_* env may reach the gateway while the sidecar is disabled."
+  }
+}
+
+run "enabled_adds_a_sidecar_that_shares_the_gateway_transport" {
+  command = plan
+
+  variables {
+    collector_enabled        = true
+    collector_port           = 4321
+    collector_buffer_size    = 250
+    collector_on_unavailable = "drop"
+    gateway_extra_env        = { OPENAI_API_BASE = "https://example.invalid" }
+    gateway_extra_secrets    = { OPENAI_API_KEY = "arn:aws:secretsmanager:us-east-1:111122223333:secret:openai-AbCdEf" }
+  }
+
+  assert {
+    condition     = length(local.collector_container) == 1 && local.collector_container[0].name == "collector"
+    error_message = "Enabling the sidecar must add exactly one collector container."
+  }
+
+  assert {
+    condition = alltrue([
+      for env in [local.gateway_environment, local.collector_container[0].environment] : (
+        { for e in env : e.name => e.value }["LITELLM_COLLECTOR_ENABLED"] == "true" &&
+        { for e in env : e.name => e.value }["LITELLM_COLLECTOR_ADDRESS"] == "tcp://127.0.0.1:4321" &&
+        { for e in env : e.name => e.value }["LITELLM_COLLECTOR_BUFFER_SIZE"] == "250" &&
+        { for e in env : e.name => e.value }["LITELLM_COLLECTOR_ON_UNAVAILABLE"] == "drop" &&
+        { for e in env : e.name => e.value }["LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS"] == "10"
+      )
+    ])
+    error_message = "Gateway and sidecar must agree on the loopback address and the collector knobs."
+  }
+
+  assert {
+    condition = (
+      local.collector_container[0].image == var.gateway_image &&
+      local.collector_container[0].entryPoint == ["sh", "-c"] &&
+      local.collector_container[0].command == ["exec python -m litellm.proxy.collector"] &&
+      local.collector_container[0].essential == false &&
+      local.collector_container[0].restartPolicy.enabled == true &&
+      { for e in local.collector_container[0].environment : e.name => e.value }["LITELLM_JOB_ROLE"] == "collector"
+    )
+    error_message = "The sidecar must run litellm.proxy.collector from the gateway image as a restartable, non-essential collector."
+  }
+
+  assert {
+    condition = (
+      { for e in local.collector_container[0].environment : e.name => e.value }["OPENAI_API_BASE"] == "https://example.invalid" &&
+      contains([for e in local.collector_container[0].environment : e.name], "DATABASE_HOST") &&
+      contains([for e in local.collector_container[0].environment : e.name], "REDIS_HOST") &&
+      contains([for s in local.collector_container[0].secrets : s.name], "LITELLM_MASTER_KEY") &&
+      contains([for s in local.collector_container[0].secrets : s.name], "OPENAI_API_KEY")
+    )
+    error_message = "The sidecar must receive the gateway's database, Redis, and shared secrets plus gateway_extra_env / gateway_extra_secrets."
+  }
+
+  assert {
+    condition     = !contains(keys(local.collector_container[0]), "portMappings")
+    error_message = "The sidecar must not expose a port to the task's load balancer."
+  }
+
+  assert {
+    condition     = local.collector_container[0].cpu == 512 && local.collector_container[0].memory == 2048
+    error_message = "The sidecar defaults must mirror helm's collector resources (500m / 2Gi)."
+  }
+}
+
+run "proxy_config_is_fetched_by_the_sidecar_too" {
+  command = plan
+
+  variables {
+    collector_enabled = true
+    proxy_config      = { model_list = [] }
+  }
+
+  assert {
+    condition = (
+      startswith(local.collector_container[0].command[0], local.proxy_config_fetch_cmd) &&
+      endswith(local.collector_container[0].command[0], "exec python -m litellm.proxy.collector") &&
+      contains([for e in local.collector_container[0].environment : e.name], "CONFIG_FILE_PATH")
+    )
+    error_message = "The sidecar must pull the proxy config from S3 before starting, like the gateway does."
+  }
+}
+
+run "sidecar_must_leave_room_for_the_gateway" {
+  command = plan
+
+  variables {
+    collector_enabled = true
+    collector_cpu     = 1024
+  }
+
+  expect_failures = [
+    aws_ecs_task_definition.gateway,
+  ]
+}
+
+run "sidecars_must_not_share_a_loopback_port" {
+  command = plan
+
+  variables {
+    collector_enabled    = true
+    collector_port       = 4001
+    gateway_metrics_port = 4001
+  }
+
+  expect_failures = [
+    aws_ecs_task_definition.gateway,
+  ]
+}

terraform/litellm/aws/variables.tf

@@ -808,3 +808,73 @@ variable "billing_metrics_ca_cert_pem" {
   default     = ""
   sensitive   = true
 }
+
+# ---------- Collector sidecar ----------
+#
+# Opt-in offload of spend tracking from the gateway's uvicorn workers to a
+# `python -m litellm.proxy.collector` sidecar in the same Fargate task (helm's
+# `gateway.collector`). Fargate awsvpc tasks share one network namespace,
+# so the sidecar listens on loopback TCP. Disabled (the default) adds nothing
+# to the task definition.
+
+variable "collector_enabled" {
+  description = "Run the collector sidecar next to the gateway container and have the gateway ship spend events to it (sets LITELLM_COLLECTOR_ENABLED=true on both). Autoscaling still targets the whole task's CPU/memory, sidecar included."
+  type        = bool
+  default     = false
+}
+
+variable "collector_port" {
+  description = "Loopback TCP port the sidecar listens on (LITELLM_COLLECTOR_ADDRESS=tcp://127.0.0.1:<port>)."
+  type        = number
+  default     = 4010
+
+  validation {
+    condition     = var.collector_port >= 1024 && var.collector_port <= 65535 && var.collector_port != 4000
+    error_message = "collector_port must be in 1024-65535 and not 4000."
+  }
+}
+
+variable "collector_cpu" {
+  description = "CPU units reserved for the sidecar container, carved out of gateway_cpu. Matches helm's collector.resources.requests.cpu (500m)."
+  type        = number
+  default     = 512
+}
+
+variable "collector_memory" {
+  description = "Hard memory limit (MiB) for the sidecar container, carved out of gateway_memory. Matches helm's collector.resources.limits.memory (2Gi)."
+  type        = number
+  default     = 2048
+}
+
+variable "collector_buffer_size" {
+  description = "Per-worker in-memory queue of spend events waiting to be shipped to the sidecar (LITELLM_COLLECTOR_BUFFER_SIZE)."
+  type        = number
+  default     = 1000
+
+  validation {
+    condition     = var.collector_buffer_size >= 1
+    error_message = "collector_buffer_size must be >= 1."
+  }
+}
+
+variable "collector_on_unavailable" {
+  description = "What the gateway does with spend events when the sidecar is unreachable or the buffer is full (LITELLM_COLLECTOR_ON_UNAVAILABLE): `fallback` runs the pipeline in-process, `drop` discards them."
+  type        = string
+  default     = "fallback"
+
+  validation {
+    condition     = contains(["fallback", "drop"], var.collector_on_unavailable)
+    error_message = "collector_on_unavailable must be one of: fallback, drop."
+  }
+}
+
+variable "collector_drain_timeout_seconds" {
+  description = "Seconds a gateway worker waits on shutdown for its buffered spend events to reach the sidecar (LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS)."
+  type        = number
+  default     = 10
+
+  validation {
+    condition     = var.collector_drain_timeout_seconds > 0
+    error_message = "collector_drain_timeout_seconds must be > 0."
+  }
+}

terraform/litellm/gcp/README.md

@@ -321,6 +321,43 @@ launcher reads these variables, starts the pooler once per instance before
 uvicorn forks the workers and hands them its loopback `DATABASE_URL`. It also
 honours `KEEPALIVE_TIMEOUT` from `gateway_extra_env` the way the image does
 
+### Collector sidecar
+
+`collector_enabled = true` adds a `spend-collector` container to the gateway
+Cloud Run service that runs `python -m litellm.proxy.collector` from the gateway
+image, and sets `LITELLM_COLLECTOR_ENABLED=true` on the gateway so its
+uvicorn workers ship spend events (SpendLogs writes, key/team/user spend
+updates, budget alerts) to the sidecar instead of running that pipeline in
+the request path. This is the Terraform counterpart of helm's
+`gateway.collector`. The default (`false`) leaves the service exactly as
+before. It is independent of the metrics sidecars above, whose GMP scraper
+already owns the `collector` container name.
+
+Containers in one Cloud Run instance share localhost, so the sidecar listens
+on loopback TCP (`tcp://127.0.0.1:${collector_port}`, default 4010)
+instead of the Unix socket helm uses; the proxy rejects any non-loopback
+address. The sidecar runs the same Redis CA + `DATABASE_URL` bootstrap as
+the gateway container, gets the same database, Redis, master-key, license,
+proxy config, and `gateway_extra_env` / `gateway_extra_secrets` values, and
+runs with `LITELLM_JOB_ROLE=collector`. When it is unreachable the
+gateway falls back to in-process spend tracking.
+
+```hcl
+collector_enabled = true
+# collector_cpu               = "1000m"  # added on top of gateway_cpu
+# collector_memory            = "2Gi"    # added on top of gateway_memory
+# collector_buffer_size       = 1000
+# collector_on_unavailable    = "fallback"  # or "drop"
+# collector_drain_timeout_seconds = 10
+```
+
+Cloud Run allocates CPU per instance while requests are in flight, and the
+sidecar shares that allocation. Spend events are shipped right after each
+response, so this works with request-based billing, but keep
+`gateway_min_instances >= 1` if spend must keep draining while an instance
+is otherwise idle. Variable names match the AWS stack; only the resource
+units differ (Cloud Run strings vs Fargate units)
+
 ## Tenant deployment
 
 Every resource the stack creates is named `${tenant}-litellm-${env}` (or

terraform/litellm/gcp/cloudrun.tf

@@ -177,6 +177,34 @@ locals {
     [local.backend_launch_cmd],
   ))
 
+  collector_address = "tcp://127.0.0.1:${var.collector_port}"
+  collector_env_kv = var.collector_enabled ? [
+    { name = "LITELLM_COLLECTOR_ENABLED", value = "true" },
+    { name = "LITELLM_COLLECTOR_ADDRESS", value = local.collector_address },
+    { name = "LITELLM_COLLECTOR_BUFFER_SIZE", value = tostring(var.collector_buffer_size) },
+    { name = "LITELLM_COLLECTOR_ON_UNAVAILABLE", value = var.collector_on_unavailable },
+    { name = "LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS", value = tostring(var.collector_drain_timeout_seconds) },
+  ] : []
+
+  gateway_env_kv      = concat(local.shared_env_kv, local.gateway_otel_env_kv, local.billing_metrics_env_kv, local.gateway_extra_env_kv, local.proxy_config_env, local.metrics_env_kv, local.gateway_pool_env, local.collector_env_kv)
+  gateway_env_secrets = concat(local.shared_env_secrets, local.otel_env_secrets, local.billing_metrics_env_secrets, local.gateway_extra_secret_kv)
+
+  collector_env_kv_all = concat(
+    local.shared_env_kv,
+    local.gateway_extra_env_kv,
+    local.proxy_config_env,
+    local.gateway_pool_env,
+    local.collector_env_kv,
+    [{ name = "LITELLM_JOB_ROLE", value = "collector" }],
+  )
+  collector_env_secrets = concat(local.shared_env_secrets, local.gateway_extra_secret_kv)
+
+  collector_args = join(" && ", concat(
+    local.redis_ca_fragment,
+    local.database_url_fragment,
+    ["exec python -m litellm.proxy.collector"],
+  ))
+
   # Env shipped to the migrations Job. The migrations image runs run.py
   # which assembles DATABASE_URL from these discrete vars itself, so we
   # only need writer-side DB env (no read replica, no proxy_config, no
@@ -203,6 +231,13 @@ resource "google_cloud_run_v2_service" "gateway" {
   labels              = local.labels
   deletion_protection = false
 
+  lifecycle {
+    precondition {
+      condition     = !var.collector_enabled || var.gateway_metrics_port == null || var.collector_port != var.gateway_metrics_port
+      error_message = "collector_port and gateway_metrics_port must differ: both sidecars bind loopback in the same instance."
+    }
+  }
+
   template {
     service_account                  = google_service_account.runtime.email
     max_instance_request_concurrency = var.gateway_max_instance_request_concurrency
@@ -235,7 +270,7 @@ resource "google_cloud_run_v2_service" "gateway" {
       }
 
       dynamic "env" {
-        for_each = concat(local.shared_env_kv, local.gateway_otel_env_kv, local.billing_metrics_env_kv, local.gateway_extra_env_kv, local.proxy_config_env, local.metrics_env_kv, local.gateway_pool_env)
+        for_each = local.gateway_env_kv
         content {
           name  = env.value.name
           value = env.value.value
@@ -243,7 +278,7 @@ resource "google_cloud_run_v2_service" "gateway" {
       }
 
       dynamic "env" {
-        for_each = concat(local.shared_env_secrets, local.otel_env_secrets, local.billing_metrics_env_secrets, local.gateway_extra_secret_kv)
+        for_each = local.gateway_env_secrets
         content {
           name = env.value.name
           value_source {
@@ -357,6 +392,52 @@ resource "google_cloud_run_v2_service" "gateway" {
       }
     }
 
+    dynamic "containers" {
+      for_each = var.collector_enabled ? [1] : []
+      content {
+        name    = "spend-collector"
+        image   = local.gateway_image
+        command = ["sh", "-c"]
+        args    = [local.collector_args]
+
+        resources {
+          limits = {
+            cpu    = var.collector_cpu
+            memory = var.collector_memory
+          }
+        }
+
+        dynamic "env" {
+          for_each = local.collector_env_kv_all
+          content {
+            name  = env.value.name
+            value = env.value.value
+          }
+        }
+
+        dynamic "env" {
+          for_each = local.collector_env_secrets
+          content {
+            name = env.value.name
+            value_source {
+              secret_key_ref {
+                secret  = env.value.secret
+                version = env.value.version
+              }
+            }
+          }
+        }
+
+        dynamic "volume_mounts" {
+          for_each = local.proxy_config_enabled ? [1] : []
+          content {
+            name       = local.proxy_config_volume
+            mount_path = local.proxy_config_mount_path
+          }
+        }
+      }
+    }
+
     dynamic "volumes" {
       for_each = local.proxy_config_enabled ? [1] : []
       content {

terraform/litellm/gcp/tests/collector.tftest.hcl

@@ -0,0 +1,165 @@
+# Plan-only coverage for the opt-in collector sidecar on the gateway Cloud
+# Run service. `mock_provider` keeps this offline: no GCP credentials, no API
+# calls. Run from terraform/litellm/gcp with `terraform test`.
+
+mock_provider "google" {}
+mock_provider "google-beta" {}
+mock_provider "random" {}
+
+variables {
+  project_id         = "acme-test"
+  region             = "us-central1"
+  tenant             = "acme"
+  env                = "test"
+  allow_plaintext_lb = true
+}
+
+run "disabled_by_default_leaves_the_service_untouched" {
+  command = plan
+
+  assert {
+    condition     = [for c in google_cloud_run_v2_service.gateway[0].template[0].containers : c.name] == ["gateway"]
+    error_message = "The gateway service must stay single-container unless collector_enabled is set."
+  }
+
+  assert {
+    condition = !anytrue([
+      for e in google_cloud_run_v2_service.gateway[0].template[0].containers[0].env : startswith(e.name, "LITELLM_COLLECTOR_")
+    ])
+    error_message = "No LITELLM_COLLECTOR_* env may reach the gateway while the sidecar is disabled."
+  }
+}
+
+run "enabled_adds_a_sidecar_that_shares_the_gateway_transport" {
+  command = plan
+
+  variables {
+    collector_enabled        = true
+    collector_port           = 4321
+    collector_buffer_size    = 250
+    collector_on_unavailable = "drop"
+    collector_cpu            = "500m"
+    collector_memory         = "1Gi"
+    gateway_extra_env        = { OPENAI_API_BASE = "https://example.invalid" }
+    gateway_extra_secrets    = { OPENAI_API_KEY = "projects/acme-test/secrets/openai-api-key" }
+  }
+
+  assert {
+    condition     = [for c in google_cloud_run_v2_service.gateway[0].template[0].containers : c.name] == ["gateway", "spend-collector"]
+    error_message = "Enabling the sidecar must append a spend-collector container after the gateway container."
+  }
+
+  assert {
+    condition = alltrue([
+      for c in google_cloud_run_v2_service.gateway[0].template[0].containers : (
+        { for e in c.env : e.name => e.value }["LITELLM_COLLECTOR_ENABLED"] == "true" &&
+        { for e in c.env : e.name => e.value }["LITELLM_COLLECTOR_ADDRESS"] == "tcp://127.0.0.1:4321" &&
+        { for e in c.env : e.name => e.value }["LITELLM_COLLECTOR_BUFFER_SIZE"] == "250" &&
+        { for e in c.env : e.name => e.value }["LITELLM_COLLECTOR_ON_UNAVAILABLE"] == "drop" &&
+        { for e in c.env : e.name => e.value }["LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS"] == "10"
+      )
+    ])
+    error_message = "Gateway and sidecar must agree on the loopback address and the collector knobs."
+  }
+
+  assert {
+    condition = (
+      google_cloud_run_v2_service.gateway[0].template[0].containers[1].image == local.gateway_image &&
+      google_cloud_run_v2_service.gateway[0].template[0].containers[1].command == tolist(["sh", "-c"]) &&
+      endswith(google_cloud_run_v2_service.gateway[0].template[0].containers[1].args[0], " && exec python -m litellm.proxy.collector") &&
+      strcontains(google_cloud_run_v2_service.gateway[0].template[0].containers[1].args[0], "export DATABASE_URL=") &&
+      strcontains(google_cloud_run_v2_service.gateway[0].template[0].containers[1].args[0], "REDIS_SSL_CA_CERTS") &&
+      { for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name => e.value }["LITELLM_JOB_ROLE"] == "collector"
+    )
+    error_message = "The sidecar must run litellm.proxy.collector from the gateway image with the same Redis CA + DATABASE_URL bootstrap as the gateway."
+  }
+
+  assert {
+    condition = (
+      length(google_cloud_run_v2_service.gateway[0].template[0].containers[1].ports) == 0 &&
+      google_cloud_run_v2_service.gateway[0].template[0].containers[1].resources[0].limits.cpu == "500m" &&
+      google_cloud_run_v2_service.gateway[0].template[0].containers[1].resources[0].limits.memory == "1Gi"
+    )
+    error_message = "The sidecar must not claim the ingress port and must carry its own resource limits."
+  }
+
+  assert {
+    condition = (
+      { for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name => e.value }["OPENAI_API_BASE"] == "https://example.invalid" &&
+      contains([for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name], "DATABASE_HOST") &&
+      contains([for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name], "REDIS_HOST") &&
+      contains([for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name if length(e.value_source) > 0], "LITELLM_MASTER_KEY") &&
+      contains([for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name if length(e.value_source) > 0], "DATABASE_PASSWORD") &&
+      contains([for e in google_cloud_run_v2_service.gateway[0].template[0].containers[1].env : e.name if length(e.value_source) > 0], "OPENAI_API_KEY")
+    )
+    error_message = "The sidecar must receive the gateway's database, Redis, and Secret Manager env plus gateway_extra_env / gateway_extra_secrets."
+  }
+}
+
+run "coexists_with_the_metrics_sidecars" {
+  command = plan
+
+  variables {
+    collector_enabled    = true
+    gateway_metrics_port = 4001
+  }
+
+  assert {
+    condition     = [for c in google_cloud_run_v2_service.gateway[0].template[0].containers : c.name] == ["gateway", "metrics", "collector", "spend-collector"]
+    error_message = "The spend collector must keep its own container name next to the GMP metrics collector."
+  }
+
+  assert {
+    condition = (
+      { for e in google_cloud_run_v2_service.gateway[0].template[0].containers[0].env : e.name => e.value }["PROMETHEUS_MULTIPROC_DIR"] == local.metrics_multiproc_dir &&
+      { for e in google_cloud_run_v2_service.gateway[0].template[0].containers[0].env : e.name => e.value }["LITELLM_COLLECTOR_ENABLED"] == "true"
+    )
+    error_message = "The gateway container must keep both the metrics and the collector env when both sidecars are on."
+  }
+}
+
+run "sidecars_must_not_share_a_loopback_port" {
+  command = plan
+
+  variables {
+    collector_enabled    = true
+    collector_port       = 4001
+    gateway_metrics_port = 4001
+  }
+
+  expect_failures = [
+    google_cloud_run_v2_service.gateway,
+  ]
+}
+
+run "collector_cannot_take_the_metrics_sidecar_health_port" {
+  command = plan
+
+  variables {
+    collector_enabled = true
+    collector_port    = 13133
+  }
+
+  expect_failures = [
+    var.collector_port,
+  ]
+}
+
+run "proxy_config_is_mounted_into_the_sidecar_too" {
+  command = plan
+
+  variables {
+    collector_enabled = true
+    proxy_config      = { model_list = [] }
+  }
+
+  assert {
+    condition = alltrue([
+      for c in google_cloud_run_v2_service.gateway[0].template[0].containers : (
+        [for m in c.volume_mounts : m.name] == [local.proxy_config_volume] &&
+        contains([for e in c.env : e.name], "CONFIG_FILE_PATH")
+      )
+    ])
+    error_message = "Both containers must mount the proxy-config GCS volume and point CONFIG_FILE_PATH at it."
+  }
+}

terraform/litellm/gcp/variables.tf

@@ -656,3 +656,73 @@ variable "billing_metrics_ca_cert_pem" {
   default     = ""
   sensitive   = true
 }
+
+# ---------- Collector sidecar ----------
+#
+# Opt-in offload of spend tracking from the gateway's uvicorn workers to a
+# `python -m litellm.proxy.collector` sidecar container in the same Cloud Run
+# instance (helm's `gateway.collector`, mirrors the AWS stack). Containers
+# in one instance share localhost, so the sidecar listens on loopback TCP.
+# Disabled (the default) adds nothing to the service.
+
+variable "collector_enabled" {
+  description = "Run the collector sidecar next to the gateway container and have the gateway ship spend events to it (sets LITELLM_COLLECTOR_ENABLED=true on both). The sidecar shares the instance's request-based CPU allocation, so pair it with a non-zero gateway_min_instances if spend must keep flowing between requests."
+  type        = bool
+  default     = false
+}
+
+variable "collector_port" {
+  description = "Loopback TCP port the sidecar listens on (LITELLM_COLLECTOR_ADDRESS=tcp://127.0.0.1:<port>)."
+  type        = number
+  default     = 4010
+
+  validation {
+    condition     = var.collector_port >= 1024 && var.collector_port <= 65535 && !contains([4000, 13133], var.collector_port)
+    error_message = "collector_port must be in 1024-65535 and not 4000 (the gateway port) or 13133 (the metrics sidecar health port)."
+  }
+}
+
+variable "collector_cpu" {
+  description = "Cloud Run CPU limit for the sidecar container, on top of gateway_cpu. Matches helm's collector.resources.limits.cpu."
+  type        = string
+  default     = "1000m"
+}
+
+variable "collector_memory" {
+  description = "Cloud Run memory limit for the sidecar container, on top of gateway_memory. Matches helm's collector.resources.limits.memory."
+  type        = string
+  default     = "2Gi"
+}
+
+variable "collector_buffer_size" {
+  description = "Per-worker in-memory queue of spend events waiting to be shipped to the sidecar (LITELLM_COLLECTOR_BUFFER_SIZE)."
+  type        = number
+  default     = 1000
+
+  validation {
+    condition     = var.collector_buffer_size >= 1
+    error_message = "collector_buffer_size must be >= 1."
+  }
+}
+
+variable "collector_on_unavailable" {
+  description = "What the gateway does with spend events when the sidecar is unreachable or the buffer is full (LITELLM_COLLECTOR_ON_UNAVAILABLE): `fallback` runs the pipeline in-process, `drop` discards them."
+  type        = string
+  default     = "fallback"
+
+  validation {
+    condition     = contains(["fallback", "drop"], var.collector_on_unavailable)
+    error_message = "collector_on_unavailable must be one of: fallback, drop."
+  }
+}
+
+variable "collector_drain_timeout_seconds" {
+  description = "Seconds a gateway worker waits on shutdown for its buffered spend events to reach the sidecar (LITELLM_COLLECTOR_DRAIN_TIMEOUT_SECONDS)."
+  type        = number
+  default     = 10
+
+  validation {
+    condition     = var.collector_drain_timeout_seconds > 0
+    error_message = "collector_drain_timeout_seconds must be > 0."
+  }
+}

tests/test_gateway/test_launch.py

@@ -3,6 +3,7 @@ import socket
 import sys
 import textwrap
 import urllib.parse
+from collections.abc import Iterator
 from pathlib import Path
 from typing import Final, cast
 from unittest.mock import MagicMock, patch
@@ -66,7 +67,7 @@ def _query(url: str) -> dict[str, str]:
 
 
 @pytest.fixture
-def password_env(monkeypatch: pytest.MonkeyPatch) -> dict[str, str]:
+def password_env(monkeypatch: pytest.MonkeyPatch) -> Iterator[dict[str, str]]:
     for var in (
         "DATABASE_URL",
         "IAM_TOKEN_DB_AUTH",
@@ -78,7 +79,8 @@ def password_env(monkeypatch: pytest.MonkeyPatch) -> dict[str, str]:
         monkeypatch.delenv(var)
     for var, value in DB_ENV.items():
         monkeypatch.setenv(var, value)
-    return dict(DB_ENV)
+    yield dict(DB_ENV)
+    os.environ.pop("DATABASE_URL", None)
 
 
 def _minted_iam_token(token: str):

tests/test_litellm/caching/test_caching_handler.py

@@ -658,3 +658,38 @@ def test_async_cache_write_completes_when_asyncio_run_closes_the_loop(monkeypatc
     asyncio.run(_short_lived_script())
 
     assert len(writes) == 1
+
+
+@pytest.mark.asyncio
+async def test_cache_hit_records_the_looked_up_key_as_the_preset_cache_key(monkeypatch):
+    """The spend log for a cache hit must reuse the key the lookup already computed instead of hashing again."""
+    import litellm
+    from litellm.caching.caching import Cache
+    from litellm.types.utils import CallTypes
+
+    async def acompletion(**kwargs):
+        return None
+
+    monkeypatch.setattr(litellm, "cache", Cache(type="local"))
+    kwargs = {"model": "gpt-5.4", "messages": [{"role": "user", "content": "hello"}], "caching": True}
+    await litellm.cache.async_add_cache(
+        litellm.ModelResponse(choices=[{"message": {"role": "assistant", "content": "hi"}}]), **kwargs
+    )
+    handler = LLMCachingHandler(original_function=acompletion, request_kwargs=kwargs, start_time=datetime.now())
+    logging_obj = _build_logging_obj(CallTypes.acompletion.value, stream=False)
+    logging_obj.async_success_handler = AsyncMock()
+
+    hit = await handler._async_get_cache(
+        model="gpt-5.4",
+        original_function=acompletion,
+        logging_obj=logging_obj,
+        start_time=datetime.now(),
+        call_type=CallTypes.acompletion.value,
+        kwargs=kwargs,
+        args=(),
+    )
+
+    assert hit is not None and hit.cached_result is not None
+    assert handler.preset_cache_key is not None
+    assert logging_obj.litellm_params["preset_cache_key"] == handler.preset_cache_key
+    assert hit.cached_result._hidden_params["cache_key"] == handler.preset_cache_key

tests/test_litellm/proxy/hooks/test_proxy_track_cost_callback.py

@@ -5,14 +5,20 @@ from unittest.mock import AsyncMock, MagicMock, patch
 import pytest
 
 from litellm.litellm_core_utils.internal_call_metadata import MODEL_ACCESS_GROUP_METADATA_KEY
-from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy._types import SpendLogsPayload, UserAPIKeyAuth
+from litellm.proxy.collector import SpendEventConsumer
+from litellm.proxy.db.spend_log_tool_index import response_tool_call_names
 from litellm.proxy.hooks.proxy_track_cost_callback import (
     _get_budget_reservation_from_metadata,
     _ProxyDBLogger,
     _should_track_cost_callback,
     _update_database_and_spend_counters,
+    run_spend_event,
 )
-from litellm.types.utils import CallTypes, Usage
+from litellm.proxy.spend_tracking.spend_event import SpendEventDecodeError, build_spend_event, decode_spend_event
+from litellm.proxy.spend_tracking.spend_event_producer import SpendEventProducer, UnixAddress
+from litellm.proxy.spend_tracking.spend_tracking_utils import get_logging_payload
+from litellm.types.utils import CallTypes, LiteLLMBatch, ModelResponse, Usage
 
 
 @pytest.mark.asyncio
@@ -2096,3 +2102,248 @@ async def test_spend_counters_keep_every_granted_group_when_the_deployment_is_un
     )
 
     assert charged == ("premium", "tier0")
+
+
+def _offload_kwargs() -> dict:
+    big_prompt = "x" * 10_000
+    reservation = {"reserved_cost": 0.5, "entries": [{"counter_key": "key:hash-1", "reserved_cost": 0.5}]}
+    return {
+        "litellm_call_id": "call-1",
+        "call_type": "acompletion",
+        "model": "gpt-4o",
+        "custom_llm_provider": "openai",
+        "stream": False,
+        "cache_hit": None,
+        "response_cost": 0.0125,
+        "completion_start_time": datetime(2026, 1, 1, 0, 0, 1),
+        "messages": [{"role": "user", "content": big_prompt}],
+        "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object"}}}],
+        "litellm_params": {
+            "api_base": "https://api.openai.com",
+            "preset_cache_key": None,
+            "proxy_server_request": {"body": {"messages": [{"role": "user", "content": big_prompt}]}},
+            "metadata": {
+                "user_api_key": "hash-1",
+                "user_api_key_hash": "hash-1",
+                "user_api_key_alias": "alias-1",
+                "user_api_key_user_id": "user-1",
+                "user_api_key_team_id": "team-1",
+                "user_api_key_org_id": "org-1",
+                "user_api_key_end_user_id": "end-user-1",
+                "user_api_key_auth": UserAPIKeyAuth(api_key="hash-1", budget_reservation=reservation),
+                "model_group": "gpt-4o",
+                "model_info": {"id": "deployment-1"},
+                "tags": ["tag-a"],
+            },
+        },
+        "standard_logging_object": {
+            "id": "chatcmpl-1",
+            "trace_id": "trace-1",
+            "response_cost": 0.0125,
+            "model": "gpt-4o-2024-08-06",
+            "model_id": "deployment-1",
+            "model_group": "gpt-4o",
+            "api_base": "https://api.openai.com",
+            "custom_llm_provider": "openai",
+            "prompt_tokens": 5000,
+            "completion_tokens": 4000,
+            "total_tokens": 9000,
+            "request_tags": ["tag-a"],
+            "request_model_access_groups": ["premium"],
+            "messages": [{"role": "user", "content": big_prompt}],
+            "response": {"choices": [{"message": {"content": "y" * 10_000}}]},
+            "model_parameters": {"temperature": 0.1},
+            "metadata": {
+                "user_api_key_hash": "hash-1",
+                "user_api_key_end_user_id": "end-user-1",
+                "usage_object": {"prompt_tokens": 5000, "completion_tokens": 4000, "total_tokens": 9000},
+            },
+            "hidden_params": {"litellm_overhead_time_ms": 3},
+            "model_map_information": {},
+            "cost_breakdown": {"input_cost": 0.0125, "output_cost": 0.0},
+        },
+    }
+
+
+def _offload_response() -> ModelResponse:
+    return ModelResponse(
+        id="chatcmpl-1",
+        model="gpt-4o-2024-08-06",
+        choices=[
+            {
+                "index": 0,
+                "message": {
+                    "role": "assistant",
+                    "content": None,
+                    "tool_calls": [
+                        {"id": "call-1", "type": "function", "function": {"name": "get_weather", "arguments": "{}"}}
+                    ],
+                },
+                "finish_reason": "tool_calls",
+            }
+        ],
+        usage=Usage(prompt_tokens=5000, completion_tokens=4000, total_tokens=9000),
+    )
+
+
+class _RecordingHandler:
+    def __init__(self) -> None:
+        self.lines: list[bytes] = []  # mutable-ok: test double records the events the sidecar received
+
+    async def __call__(self, line: bytes) -> None:
+        self.lines.append(line)
+
+
+async def _no_fallback(line: bytes) -> None:
+    raise AssertionError("the sidecar was reachable, nothing should fall back")
+
+
+@pytest.mark.asyncio
+async def test_async_log_success_event_hands_the_sidecar_a_compact_event_and_skips_the_pipeline(tmp_path):
+    handler = _RecordingHandler()
+    consumer = SpendEventConsumer(handler)
+    address = UnixAddress(path=str(tmp_path / "spend.sock"))
+    server = await consumer.serve(address)
+    producer = SpendEventProducer(
+        address=address, on_unavailable="fallback", buffer_size=10, connect_timeout=1.0, fallback=_no_fallback
+    )
+    logger = _ProxyDBLogger(producer)
+
+    with (
+        patch(  # test-quality-ok: the callback imports this from proxy_server inside its body, so there is no injection seam
+            "litellm.proxy.proxy_server.proxy_logging_obj"
+        ) as mock_proxy_logging,
+        patch(  # test-quality-ok: same function-body import, no injection seam
+            "litellm.proxy.proxy_server.increment_spend_counters", new_callable=AsyncMock
+        ) as counters,
+    ):
+        mock_proxy_logging.db_spend_update_writer.update_database = AsyncMock()
+        await logger.async_log_success_event(_offload_kwargs(), _offload_response(), datetime.now(), datetime.now())
+        await producer.close(drain_timeout=5.0)
+        server.close()
+        assert await consumer.drain(timeout=5.0) == 0
+
+    mock_proxy_logging.db_spend_update_writer.update_database.assert_not_awaited()
+    counters.assert_not_awaited()
+    assert producer.stats().sent == 1
+    assert len(handler.lines) == 1
+    assert len(handler.lines[0]) < 4_000
+    event = decode_spend_event(handler.lines[0])
+    assert not isinstance(event, SpendEventDecodeError)
+    assert event.litellm_params["metadata"]["user_api_key_team_id"] == "team-1"
+    assert event.response_cost == 0.0125
+
+
+@pytest.mark.asyncio
+async def test_async_log_success_event_keeps_batch_retrieves_in_process():
+    producer = SpendEventProducer(
+        address=UnixAddress(path="/nonexistent/spend.sock"),
+        on_unavailable="drop",
+        buffer_size=10,
+        connect_timeout=1.0,
+        fallback=_no_fallback,
+    )
+    logger = _ProxyDBLogger(producer)
+    kwargs = {**_offload_kwargs(), "call_type": CallTypes.aretrieve_batch.value}
+    completed_batch = LiteLLMBatch(
+        id="batch_abc",
+        completion_window="24h",
+        created_at=1,
+        endpoint="/v1/chat/completions",
+        input_file_id="file-in",
+        output_file_id="file-out",
+        object="batch",
+        status="completed",
+    )
+
+    with (
+        patch(  # test-quality-ok: the callback imports this from proxy_server inside its body, so there is no injection seam
+            "litellm.proxy.proxy_server.proxy_logging_obj"
+        ) as mock_proxy_logging,
+        patch(  # test-quality-ok: same function-body import, no injection seam
+            "litellm.proxy.proxy_server.increment_spend_counters", new_callable=AsyncMock
+        ),
+        patch(  # test-quality-ok: same function-body import, no injection seam
+            "litellm.proxy.proxy_server.update_cache", new_callable=AsyncMock
+        ),
+    ):
+        mock_proxy_logging.db_spend_update_writer.update_database = AsyncMock(return_value=True)
+        mock_proxy_logging.slack_alerting_instance.customer_spend_alert = AsyncMock()
+        await logger.async_log_success_event(kwargs, completed_batch, datetime.now(), datetime.now())
+
+    mock_proxy_logging.db_spend_update_writer.update_database.assert_awaited_once()
+    assert producer.stats().queued == 0
+
+
+async def _spend_row_written_by(run) -> tuple[SpendLogsPayload, dict, tuple[str, ...]]:
+    with (
+        patch(  # test-quality-ok: the callback imports this from proxy_server inside its body, so there is no injection seam
+            "litellm.proxy.proxy_server.proxy_logging_obj"
+        ) as mock_proxy_logging,
+        patch(  # test-quality-ok: same function-body import, no injection seam
+            "litellm.proxy.proxy_server.increment_spend_counters", new_callable=AsyncMock
+        ) as counters,
+        patch(  # test-quality-ok: same function-body import, no injection seam
+            "litellm.proxy.proxy_server.update_cache", new_callable=AsyncMock
+        ),
+    ):
+        mock_proxy_logging.db_spend_update_writer.update_database = AsyncMock(return_value=True)
+        mock_proxy_logging.slack_alerting_instance.customer_spend_alert = AsyncMock()
+        await run()
+        mock_proxy_logging.db_spend_update_writer.update_database.assert_awaited_once()
+        written = mock_proxy_logging.db_spend_update_writer.update_database.await_args.kwargs
+        counters.assert_awaited_once()
+        counted = dict(counters.await_args.kwargs)
+
+    row = get_logging_payload(
+        kwargs=written["kwargs"],
+        response_obj=written["completion_response"],
+        start_time=written["start_time"],
+        end_time=written["end_time"],
+    )
+    return row, counted, response_tool_call_names(written["completion_response"])
+
+
+@pytest.mark.asyncio
+async def test_sidecar_writes_the_same_spend_row_and_counters_as_the_in_process_path():
+    start_time = datetime(2026, 1, 1, 0, 0, 0)
+    end_time = datetime(2026, 1, 1, 0, 0, 2)
+
+    async def in_process() -> None:
+        await _ProxyDBLogger().async_log_success_event(_offload_kwargs(), _offload_response(), start_time, end_time)
+
+    async def via_sidecar() -> None:
+        line = build_spend_event(_offload_kwargs(), _offload_response(), start_time, end_time, store_bodies=False)
+        assert isinstance(line, bytes)
+        await run_spend_event(line)
+
+    in_process_row, in_process_counters, in_process_tools = await _spend_row_written_by(in_process)
+    sidecar_row, sidecar_counters, sidecar_tools = await _spend_row_written_by(via_sidecar)
+
+    assert sidecar_row == in_process_row
+    assert in_process_row["spend"] == 0.0125
+    assert in_process_row["team_id"] == "team-1"
+    assert in_process_row["end_user"] == "end-user-1"
+    assert in_process_row["total_tokens"] == 9000
+    assert in_process_row["model_id"] == "deployment-1"
+    assert in_process_row["request_tags"] == '["tag-a"]'
+    assert in_process_row["messages"] == "{}"
+    assert in_process_row["response"] == "{}"
+    assert sidecar_counters == in_process_counters
+    assert in_process_counters["token"] == "hash-1"
+    assert in_process_counters["response_cost"] == 0.0125
+    assert in_process_counters["budget_reservation"]["reserved_cost"] == 0.5
+    assert in_process_counters["model_access_groups"] == ("premium",)
+    assert sidecar_tools == in_process_tools == ("get_weather",)
+
+
+@pytest.mark.asyncio
+async def test_sidecar_ignores_an_undecodable_event():  # test-quality-ok: a discarded event has no observable output other than the DB writer never being reached
+    with (
+        patch(  # test-quality-ok: the callback imports this from proxy_server inside its body, so there is no injection seam
+            "litellm.proxy.proxy_server.proxy_logging_obj"
+        ) as mock_proxy_logging
+    ):
+        mock_proxy_logging.db_spend_update_writer.update_database = AsyncMock()
+        await run_spend_event(b"garbage\n")
+    mock_proxy_logging.db_spend_update_writer.update_database.assert_not_awaited()

tests/test_litellm/proxy/proxy_server/test_lifecycle.py

@@ -253,6 +253,38 @@ async def test_flush_spend_logs_queue_on_shutdown_swallows_drain_errors(monkeypa
     await ps._flush_spend_logs_queue_on_shutdown()
 
 
+@pytest.mark.asyncio
+async def test_flush_spend_counters_on_shutdown_commits_buffered_spend(monkeypatch):
+    fake_prisma = MagicMock()
+    monkeypatch.setattr(ps, "prisma_client", fake_prisma, raising=False)
+    commit = AsyncMock()
+    monkeypatch.setattr(ps.proxy_logging_obj.db_spend_update_writer, "db_update_spend_transaction_handler", commit)
+
+    await ps.flush_spend_counters_on_shutdown()
+
+    observed = {
+        "commit_calls": commit.await_count,
+        "commit_prisma": commit.await_args.kwargs["prisma_client"] is fake_prisma,
+        "commit_proxy_logging": commit.await_args.kwargs["proxy_logging_obj"] is ps.proxy_logging_obj,
+    }
+    assert observed == {"commit_calls": 1, "commit_prisma": True, "commit_proxy_logging": True}
+
+
+@pytest.mark.asyncio
+async def test_flush_spend_counters_on_shutdown_logs_and_swallows_commit_errors(monkeypatch, caplog):
+    monkeypatch.setattr(ps, "prisma_client", MagicMock(), raising=False)
+    monkeypatch.setattr(
+        ps.proxy_logging_obj.db_spend_update_writer,
+        "db_update_spend_transaction_handler",
+        AsyncMock(side_effect=RuntimeError("db gone")),
+    )
+
+    with caplog.at_level(logging.ERROR, logger="LiteLLM Proxy"):
+        await ps.flush_spend_counters_on_shutdown()
+
+    assert "Error flushing spend counters on shutdown: db gone" in caplog.text
+
+
 # ---------------------------------------------------------------------------
 # _initialize_shared_aiohttp_session
 # ---------------------------------------------------------------------------

tests/test_litellm/proxy/spend_tracking/test_spend_event.py

@@ -0,0 +1,213 @@
+import json
+from datetime import datetime
+from typing import Final
+
+import pytest
+
+import litellm
+from litellm.caching.caching import Cache
+from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.spend_tracking.spend_event import (
+    CACHE_OFF_KEY,
+    SpendEventBuildError,
+    SpendEventDecodeError,
+    build_spend_event,
+    decode_spend_event,
+    is_offloadable_success,
+    spend_event_callback_args,
+)
+from litellm.types.utils import LiteLLMBatch, ModelResponse, Usage
+
+_BIG_PROMPT: Final = "x" * 20_000
+_RESERVATION: Final = {
+    "reserved_cost": 0.5,
+    "entries": [{"counter_key": "key:hash", "reserved_cost": 0.5}],
+    "finalized": False,
+    "input_cost": 0.1,
+    "input_tokens": 5000,
+}
+
+
+def _response(tool_name: str | None = None) -> ModelResponse:
+    tool_calls: Final = (
+        [{"id": "call-1", "type": "function", "function": {"name": tool_name, "arguments": "{}"}}]
+        if tool_name is not None
+        else None
+    )
+    return ModelResponse(
+        id="chatcmpl-1",
+        model="gpt-4o-2024-08-06",
+        choices=[
+            {
+                "index": 0,
+                "message": {"role": "assistant", "content": "y" * 20_000, "tool_calls": tool_calls},
+                "finish_reason": "tool_calls" if tool_name else "stop",
+            }
+        ],
+        usage=Usage(prompt_tokens=5000, completion_tokens=4000, total_tokens=9000),
+    )
+
+
+def _success_kwargs(preset_cache_key: str | None = "preset-key") -> dict:
+    return {
+        "litellm_call_id": "call-1",
+        "call_type": "acompletion",
+        "model": "gpt-4o",
+        "custom_llm_provider": "openai",
+        "stream": False,
+        "cache_hit": None,
+        "response_cost": 0.0125,
+        "completion_start_time": datetime(2026, 1, 1, 0, 0, 1),
+        "messages": [{"role": "user", "content": _BIG_PROMPT}],
+        "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object"}}}],
+        "litellm_params": {
+            "api_base": "https://api.openai.com",
+            "preset_cache_key": preset_cache_key,
+            "proxy_server_request": {"body": {"messages": [{"role": "user", "content": _BIG_PROMPT}]}},
+            "metadata": {
+                "user_api_key": "hash-1",
+                "user_api_key_user_id": "user-1",
+                "user_api_key_team_id": "team-1",
+                "user_api_key_org_id": "org-1",
+                "user_api_key_end_user_id": "end-user-1",
+                "user_api_key_auth": UserAPIKeyAuth(api_key="hash-1", budget_reservation=dict(_RESERVATION)),
+                "model_group": "gpt-4o",
+                "model_info": {"id": "deployment-1"},
+                "tags": ["tag-a"],
+                "litellm_parent_otel_span": object(),
+            },
+        },
+        "standard_logging_object": {
+            "response_cost": 0.0125,
+            "model": "gpt-4o-2024-08-06",
+            "model_id": "deployment-1",
+            "request_tags": ["tag-a"],
+            "request_model_access_groups": ["premium"],
+            "messages": [{"role": "user", "content": _BIG_PROMPT}],
+            "response": {"choices": [{"message": {"content": "y" * 20_000}}]},
+            "model_parameters": {"temperature": 0.1},
+            "metadata": {"user_api_key_hash": "hash-1", "usage_object": {"prompt_tokens": 5000}},
+            "hidden_params": {"litellm_overhead_time_ms": 3},
+            "model_map_information": {},
+        },
+    }
+
+
+def _build(kwargs: dict, response: object, store_bodies: bool = False) -> bytes:
+    line: Final = build_spend_event(
+        kwargs, response, datetime(2026, 1, 1), datetime(2026, 1, 1, 0, 0, 2), store_bodies=store_bodies
+    )
+    assert isinstance(line, bytes)
+    return line
+
+
+def test_event_is_compact_and_omits_bodies_by_default():
+    line: Final = _build(_success_kwargs(), _response(tool_name="get_weather"))
+    assert line.endswith(b"\n")
+    assert len(line) < 4_000
+    assert _BIG_PROMPT.encode() not in line
+    assert b"yyyy" not in line
+    decoded: Final = json.loads(line)
+    assert "messages" not in decoded["standard_logging_object"]
+    assert "response" not in decoded["standard_logging_object"]
+    assert decoded["litellm_params"]["proxy_server_request"] is None
+
+
+def test_event_carries_bodies_when_spend_logs_store_them():
+    line: Final = _build(_success_kwargs(), _response(), store_bodies=True)
+    decoded: Final = json.loads(line)
+    assert decoded["standard_logging_object"]["messages"][0]["content"] == _BIG_PROMPT
+    assert decoded["standard_logging_object"]["response"]["choices"][0]["message"]["content"] == "y" * 20_000
+    assert decoded["litellm_params"]["proxy_server_request"]["body"]["messages"][0]["content"] == _BIG_PROMPT
+
+
+def test_round_trip_preserves_identity_usage_reservation_and_tools():
+    line: Final = _build(_success_kwargs(), _response(tool_name="get_weather"))
+    event: Final = decode_spend_event(line)
+    assert not isinstance(event, SpendEventDecodeError)
+    args: Final = spend_event_callback_args(event)
+
+    metadata: Final = args.kwargs["litellm_params"]["metadata"]
+    assert metadata is not None
+    assert (metadata["user_api_key"], metadata["user_api_key_team_id"], metadata["user_api_key_org_id"]) == (
+        "hash-1",
+        "team-1",
+        "org-1",
+    )
+    assert metadata["user_api_key_budget_reservation"] == _RESERVATION
+    assert "user_api_key_auth" not in metadata
+    assert "litellm_parent_otel_span" not in metadata
+    assert args.kwargs["standard_logging_object"]["request_model_access_groups"] == ["premium"]
+    assert args.kwargs["standard_logging_object"]["response_cost"] == 0.0125
+    assert args.kwargs["tools"] == ({"type": "function", "function": {"name": "get_weather"}},)
+    assert args.kwargs["completion_start_time"] == datetime(2026, 1, 1, 0, 0, 1)
+    assert (args.start_time, args.end_time) == (datetime(2026, 1, 1), datetime(2026, 1, 1, 0, 0, 2))
+    assert args.response_obj is not None
+    assert args.response_obj["id"] == "chatcmpl-1"
+    assert args.response_obj["usage"]["prompt_tokens"] == 5000
+    assert args.response_obj["usage"]["completion_tokens"] == 4000
+    tool_calls: Final = args.response_obj["choices"][0]["message"]["tool_calls"]
+    assert [call["function"]["name"] for call in tool_calls] == ["get_weather"]
+    assert "complete_streaming_response" not in args.kwargs
+
+
+def test_streaming_event_reconstructs_complete_streaming_response():
+    kwargs: Final = {**_success_kwargs(), "stream": True, "complete_streaming_response": _response()}
+    event: Final = decode_spend_event(_build(kwargs, _response()))
+    assert not isinstance(event, SpendEventDecodeError)
+    args: Final = spend_event_callback_args(event)
+    assert args.kwargs["stream"] is True
+    assert args.kwargs["complete_streaming_response"] == args.response_obj
+
+
+class _HashingCache(Cache):
+    def __init__(self) -> None:
+        pass
+
+    def get_cache_key(self, **kwargs) -> str:
+        raise AssertionError("the fast path must not hash the request body")
+
+
+@pytest.mark.parametrize(
+    ("cache", "preset", "expected"),
+    [
+        (None, "preset-key", CACHE_OFF_KEY),
+        (_HashingCache(), "preset-key", "preset-key"),
+        (_HashingCache(), None, None),
+    ],
+)
+def test_event_reuses_preset_cache_key_and_never_hashes(monkeypatch, cache, preset, expected):
+    monkeypatch.setattr(litellm, "cache", cache)
+    decoded: Final = json.loads(_build(_success_kwargs(preset_cache_key=preset), _response()))
+    assert decoded["litellm_params"]["preset_cache_key"] == expected
+
+
+def test_unbuildable_kwargs_fall_back_to_in_process_tracking():
+    kwargs: Final = {**_success_kwargs(), "response_cost": "not-a-number"}
+    assert isinstance(
+        build_spend_event(kwargs, _response(), datetime.now(), datetime.now(), False), SpendEventBuildError
+    )
+
+
+def test_undecodable_line_is_an_error_value():
+    assert isinstance(decode_spend_event(b'{"version": 2}\n'), SpendEventDecodeError)
+    assert isinstance(decode_spend_event(b"not json\n"), SpendEventDecodeError)
+
+
+def test_batch_retrieves_stay_in_process():
+    assert is_offloadable_success(_response()) is True
+    assert is_offloadable_success(None) is True
+    assert (
+        is_offloadable_success(
+            LiteLLMBatch(
+                id="batch-1",
+                completion_window="24h",
+                created_at=1,
+                endpoint="/v1/chat/completions",
+                input_file_id="f",
+                object="batch",
+                status="completed",
+            )
+        )
+        is False
+    )

tests/test_litellm/proxy/spend_tracking/test_spend_event_producer.py

@@ -0,0 +1,359 @@
+import asyncio
+from collections.abc import Awaitable, Callable
+from pathlib import Path
+from typing import Final
+
+import pytest
+import uvloop
+
+from litellm.proxy.spend_tracking.spend_event_producer import (
+    AddressError,
+    CollectorAddress,
+    CollectorSettings,
+    SpendEventProducer,
+    TcpAddress,
+    UnixAddress,
+    build_spend_event_producer,
+    open_collector_connection,
+    parse_collector_address,
+)
+
+
+class _Sidecar:
+    """A unix-socket server that records every line it receives, standing in for the collector."""
+
+    def __init__(self, path: Path, reads: bool = True, limit: int = 2**16) -> None:
+        self.path = path
+        self.reads = reads
+        self.limit = limit
+        self.lines: list[bytes] = []  # mutable-ok: test double records what the producer sent
+        self._server: asyncio.Server | None = None
+        self._stopped = asyncio.Event()
+        self._connections: list[asyncio.StreamWriter] = []  # mutable-ok: test double tracks peers to hang up on
+
+    async def __aenter__(self) -> "_Sidecar":
+        self._server = await asyncio.start_unix_server(self._on_connection, path=str(self.path), limit=self.limit)
+        return self
+
+    async def __aexit__(self, *exc: object) -> None:
+        self._stopped.set()
+        await self.hang_up()
+
+    async def hang_up(self) -> None:
+        """Exit the way a stopped sidecar does: stop listening and close every producer connection."""
+        assert self._server is not None
+        self._server.close()
+        for connection in self._connections:
+            connection.close()
+            await connection.wait_closed()
+        await self._server.wait_closed()
+
+    async def _on_connection(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
+        self._connections.append(writer)
+        if not self.reads:
+            await self._stopped.wait()
+            return
+        while line := await reader.readline():
+            self.lines.append(line)
+        writer.close()
+
+
+class _CrashingSidecar(_Sidecar):
+    """Bills a few lines, then dies mid-stream with the producer's backlog still queued behind them."""
+
+    def __init__(self, path: Path, lines_before_crash: int) -> None:
+        super().__init__(path, limit=2**20)
+        self._lines_before_crash = lines_before_crash
+
+    async def _on_connection(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
+        self._connections.append(writer)
+        for _ in range(self._lines_before_crash):
+            self.lines.append(await reader.readline())
+        writer.transport.abort()
+
+
+class _Fallback:
+    def __init__(self) -> None:
+        self.lines: list[bytes] = []  # mutable-ok: test double records what fell back to in-process
+
+    async def __call__(self, line: bytes) -> None:
+        self.lines.append(line)
+
+
+class _GatedFallback(_Fallback):
+    """A fallback that blocks, like a slow database write, until the test releases it."""
+
+    def __init__(self) -> None:
+        super().__init__()
+        self.started = asyncio.Event()
+        self.release = asyncio.Event()
+
+    async def __call__(self, line: bytes) -> None:
+        self.started.set()
+        await self.release.wait()
+        await super().__call__(line)
+
+
+class _StalledDrainWriter(asyncio.StreamWriter):
+    """Hands bytes to the real transport but never wakes ``drain()``: the loop iteration between a flush
+    completing and the writer task resuming, frozen in place."""
+
+    def __init__(self, real: asyncio.StreamWriter, reader: asyncio.StreamReader) -> None:
+        super().__init__(real.transport, real.transport.get_protocol(), reader, asyncio.get_running_loop())
+        self._real_writer_whose_finalizer_would_close_the_transport = real
+
+    async def drain(self) -> None:
+        await asyncio.Event().wait()
+
+
+async def _open_with_stalled_drain(
+    address: CollectorAddress, timeout: float
+) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]:
+    reader, writer = await open_collector_connection(address, timeout)
+    return reader, _StalledDrainWriter(writer, reader)
+
+
+def _producer(
+    path: Path,
+    fallback: _Fallback,
+    on_unavailable="fallback",
+    buffer_size: int = 100,
+    open_connection: Callable[
+        [CollectorAddress, float], Awaitable[tuple[asyncio.StreamReader, asyncio.StreamWriter]]
+    ] = open_collector_connection,
+) -> SpendEventProducer:
+    return SpendEventProducer(
+        address=UnixAddress(path=str(path)),
+        on_unavailable=on_unavailable,
+        buffer_size=buffer_size,
+        connect_timeout=1.0,
+        fallback=fallback,
+        open_connection=open_connection,
+    )
+
+
+def test_parse_collector_address():
+    assert parse_collector_address("unix:///var/run/litellm/collector.sock") == UnixAddress(
+        path="/var/run/litellm/collector.sock"
+    )
+    assert parse_collector_address("tcp://127.0.0.1:4100") == TcpAddress(host="127.0.0.1", port=4100)
+    assert parse_collector_address("tcp://localhost:4100") == TcpAddress(host="localhost", port=4100)
+    assert parse_collector_address("tcp://[::1]:4100") == TcpAddress(host="::1", port=4100)
+    assert isinstance(parse_collector_address("redis://localhost:6379"), AddressError)
+    assert isinstance(parse_collector_address("tcp://127.0.0.1"), AddressError)
+
+
+@pytest.mark.parametrize("address", ["tcp://0.0.0.0:4100", "tcp://10.0.0.5:4100", "tcp://collector.svc:4100"])
+def test_tcp_address_outside_loopback_is_refused(address: str):
+    """The socket has no authentication, so anything reachable from outside the pod would accept forged spend."""
+    error: Final = parse_collector_address(address)
+    assert isinstance(error, AddressError)
+    assert "loopback" in error.reason
+    assert build_spend_event_producer(CollectorSettings(enabled=True, address=address), _Fallback()) is None
+
+
+def test_gateway_produces_only_when_enabled_and_not_the_sidecar_itself():
+    fallback: Final = _Fallback()
+    assert build_spend_event_producer(CollectorSettings(enabled=False), fallback) is None
+    assert build_spend_event_producer(CollectorSettings(enabled=True, job_role="collector"), fallback) is None
+    assert build_spend_event_producer(CollectorSettings(enabled=True, address="redis://x"), fallback) is None
+    assert isinstance(build_spend_event_producer(CollectorSettings(enabled=True), fallback), SpendEventProducer)
+
+
+def test_settings_read_the_documented_env(monkeypatch):
+    monkeypatch.setenv("LITELLM_COLLECTOR_ENABLED", "true")
+    monkeypatch.setenv("LITELLM_COLLECTOR_ADDRESS", "tcp://127.0.0.1:4100")
+    monkeypatch.setenv("LITELLM_COLLECTOR_BUFFER_SIZE", "50")
+    monkeypatch.setenv("LITELLM_COLLECTOR_ON_UNAVAILABLE", "drop")
+    monkeypatch.setenv("LITELLM_JOB_ROLE", "collector")
+    settings: Final = CollectorSettings()
+    assert (settings.enabled, settings.address, settings.buffer_size, settings.on_unavailable) == (
+        True,
+        "tcp://127.0.0.1:4100",
+        50,
+        "drop",
+    )
+    assert settings.produces is False
+
+
+@pytest.mark.asyncio
+async def test_events_reach_the_sidecar_once_and_in_order(tmp_path: Path):
+    fallback: Final = _Fallback()
+    async with _Sidecar(tmp_path / "spend.sock") as sidecar:
+        producer: Final = _producer(sidecar.path, fallback)
+        outcomes: Final = [await producer.publish(f"event-{i}\n".encode()) for i in range(20)]
+        await producer.close(drain_timeout=5.0)
+        await asyncio.sleep(0.05)
+
+    assert outcomes == ["queued"] * 20
+    assert sidecar.lines == [f"event-{i}\n".encode() for i in range(20)]
+    assert fallback.lines == []
+    stats: Final = producer.stats()
+    assert (stats.queued, stats.sent, stats.fallback, stats.dropped) == (20, 20, 0, 0)
+
+
+@pytest.mark.asyncio
+async def test_unreachable_sidecar_falls_back_in_process_and_backs_off(tmp_path: Path):
+    fallback: Final = _Fallback()
+    producer: Final = _producer(tmp_path / "missing.sock", fallback)
+    first: Final = await producer.publish(b"event-1\n")
+    await asyncio.sleep(0.05)
+    second: Final = await producer.publish(b"event-2\n")
+    await producer.close(drain_timeout=5.0)
+
+    assert first == "queued"
+    assert second == "fallback"
+    assert fallback.lines == [b"event-1\n", b"event-2\n"]
+    stats: Final = producer.stats()
+    assert (stats.sent, stats.fallback, stats.dropped, stats.connected) == (0, 2, 0, False)
+
+
+@pytest.mark.parametrize("loop_factory", [asyncio.new_event_loop, uvloop.new_event_loop], ids=["asyncio", "uvloop"])
+def test_sidecar_hang_up_falls_back_instead_of_losing_events(
+    tmp_path: Path, loop_factory: Callable[[], asyncio.AbstractEventLoop]
+):
+    async def scenario() -> tuple[list[bytes], list[bytes], tuple[int, int, int]]:
+        fallback: Final = _Fallback()
+        sidecar: Final = _Sidecar(tmp_path / "spend.sock")
+        async with sidecar:
+            producer: Final = _producer(sidecar.path, fallback)
+            await producer.publish(b"event-1\n")
+            await asyncio.sleep(0.05)
+            await sidecar.hang_up()
+            await asyncio.sleep(0.05)
+            await producer.publish(b"event-2\n")
+            await producer.close(drain_timeout=5.0)
+        stats: Final = producer.stats()
+        return sidecar.lines, fallback.lines, (stats.sent, stats.fallback, stats.dropped)
+
+    with asyncio.Runner(loop_factory=loop_factory) as runner:
+        sidecar_lines, fallback_lines, counts = runner.run(scenario())
+
+    assert sidecar_lines == [b"event-1\n"]
+    assert fallback_lines == [b"event-2\n"]
+    assert counts == (1, 1, 0)
+
+
+@pytest.mark.parametrize("loop_factory", [asyncio.new_event_loop, uvloop.new_event_loop], ids=["asyncio", "uvloop"])
+def test_mid_stream_crash_never_bills_an_event_on_both_sides(
+    tmp_path: Path, loop_factory: Callable[[], asyncio.AbstractEventLoop]
+):
+    """Events large enough to straddle the kernel buffer, a sidecar that reads some and then drops the socket: a
+    failed write may only fall back when the sidecar cannot have read the whole line."""
+    events: Final = tuple(f"event-{i:03d}-".encode() + b"x" * 65536 + b"\n" for i in range(64))
+
+    async def scenario() -> tuple[list[bytes], list[bytes], tuple[int, int, int]]:
+        fallback: Final = _Fallback()
+        sidecar: Final = _CrashingSidecar(tmp_path / "spend.sock", lines_before_crash=3)
+        async with sidecar:
+            producer: Final = _producer(sidecar.path, fallback)
+            for event in events:
+                assert await producer.publish(event) == "queued"
+            await asyncio.sleep(0.2)
+            await producer.close(drain_timeout=5.0)
+        stats: Final = producer.stats()
+        return sidecar.lines, fallback.lines, (stats.sent, stats.fallback, stats.dropped)
+
+    with asyncio.Runner(loop_factory=loop_factory) as runner:
+        sidecar_lines, fallback_lines, counts = runner.run(scenario())
+
+    assert sidecar_lines == list(events[:3])
+    assert set(sidecar_lines).isdisjoint(fallback_lines)
+    assert len(fallback_lines) == len(set(fallback_lines))
+    assert fallback_lines[-1] == events[-1]
+    assert counts[0] + counts[1] == len(events) and counts[2] == 0
+    assert counts[0] >= len(sidecar_lines)
+
+
+@pytest.mark.asyncio
+async def test_drain_timeout_hands_the_in_flight_event_to_fallback(tmp_path: Path):
+    """A sidecar that stops reading leaves one event half-written; cancelling the writer must not lose it."""
+    fallback: Final = _Fallback()
+    stuck: Final = b"x" * (4 * 1024 * 1024) + b"\n"
+    async with _Sidecar(tmp_path / "spend.sock", reads=False) as sidecar:
+        producer: Final = _producer(sidecar.path, fallback)
+        assert await producer.publish(stuck) == "queued"
+        await asyncio.sleep(0.1)
+        await producer.close(drain_timeout=0.2)
+
+    assert fallback.lines == [stuck]
+    stats: Final = producer.stats()
+    assert (stats.sent, stats.fallback, stats.connected) == (0, 1, False)
+
+
+@pytest.mark.asyncio
+async def test_shutdown_lets_the_writer_finish_a_fallback_already_in_progress(tmp_path: Path):
+    """Cancelling the writer while it runs the pipeline in-process must neither lose nor repeat that event."""
+    fallback: Final = _GatedFallback()
+    producer: Final = _producer(tmp_path / "missing.sock", fallback)
+    assert await producer.publish(b"event-1\n") == "queued"
+    await asyncio.wait_for(fallback.started.wait(), 5.0)
+    closing: Final = asyncio.ensure_future(producer.close(drain_timeout=0.05))
+    await asyncio.sleep(0.2)
+    assert fallback.lines == []
+    fallback.release.set()
+    await asyncio.wait_for(closing, 5.0)
+
+    assert fallback.lines == [b"event-1\n"]
+    assert producer.stats().fallback == 1
+
+
+@pytest.mark.asyncio
+async def test_shutdown_does_not_replay_an_event_the_kernel_already_took(tmp_path: Path):
+    """Cancelling a drain whose bytes already left the process must not run the event a second time in-process."""
+    fallback: Final = _Fallback()
+    async with _Sidecar(tmp_path / "spend.sock") as sidecar:
+        producer: Final = _producer(sidecar.path, fallback, open_connection=_open_with_stalled_drain)
+        assert await producer.publish(b"event-1\n") == "queued"
+        await asyncio.sleep(0.1)
+        await producer.close(drain_timeout=0.2)
+        await asyncio.sleep(0.05)
+
+    assert sidecar.lines == [b"event-1\n"]
+    assert fallback.lines == []
+    stats: Final = producer.stats()
+    assert (stats.fallback, stats.dropped, stats.connected) == (0, 0, False)
+
+
+@pytest.mark.asyncio
+async def test_drop_policy_counts_instead_of_running_in_process(tmp_path: Path):
+    fallback: Final = _Fallback()
+    producer: Final = _producer(tmp_path / "missing.sock", fallback, on_unavailable="drop")
+    await producer.publish(b"event-1\n")
+    await producer.close(drain_timeout=5.0)
+    assert await producer.publish(b"event-2\n") == "dropped"
+
+    assert fallback.lines == []
+    assert producer.stats().dropped == 2
+
+
+@pytest.mark.asyncio
+async def test_full_buffer_applies_the_unavailable_policy_immediately(tmp_path: Path):
+    fallback: Final = _Fallback()
+    async with _Sidecar(tmp_path / "spend.sock") as sidecar:
+        producer: Final = _producer(sidecar.path, fallback, buffer_size=2)
+        outcomes: Final = [await producer.publish(f"event-{i}\n".encode()) for i in range(3)]
+        await producer.close(drain_timeout=5.0)
+        await asyncio.sleep(0.05)
+
+    assert outcomes == ["queued", "queued", "fallback"]
+    assert fallback.lines == [b"event-2\n"]
+    assert sidecar.lines == [b"event-0\n", b"event-1\n"]
+
+
+@pytest.mark.asyncio
+async def test_close_flushes_buffered_events_then_refuses_new_ones(tmp_path: Path):
+    fallback: Final = _Fallback()
+    async with _Sidecar(tmp_path / "spend.sock") as sidecar:
+        producer: Final = _producer(sidecar.path, fallback)
+        for i in range(50):
+            await producer.publish(f"event-{i}\n".encode())
+        assert sidecar.lines == []
+        await producer.close(drain_timeout=5.0)
+        await asyncio.sleep(0.05)
+        after_close: Final = await producer.publish(b"late\n")
+
+    assert len(sidecar.lines) == 50
+    assert after_close == "fallback"
+    assert fallback.lines == [b"late\n"]
+    assert producer.stats().connected is False

tests/test_litellm/proxy/spend_tracking/test_spend_tracking_utils.py

@@ -38,9 +38,9 @@ from litellm.proxy.spend_tracking.spend_tracking_utils import (
     _sanitize_error_information_for_spend_logs,
     _sanitize_guardrail_information_for_spend_logs,
     _sanitize_request_body_for_spend_logs_payload,
-    _should_store_prompts_and_responses_in_spend_logs,
     get_logging_payload,
     get_spend_logs_id,
+    should_store_prompts_and_responses_in_spend_logs,
 )
 from litellm.proxy.utils import hash_token
 from litellm.types.utils import (
@@ -107,6 +107,48 @@ def test_get_logging_payload_maps_openai_cached_tokens_to_cache_read_input_token
     assert additional_usage_values["prompt_tokens_details"]["cached_tokens"] == 123
 
 
+class _HashingCache(litellm.Cache):
+    def __init__(self) -> None:
+        pass
+
+    def get_cache_key(self, **kwargs) -> str:
+        raise AssertionError("a preset cache key must be reused instead of hashing the request")
+
+
+def _cache_key_in_spend_log(monkeypatch: pytest.MonkeyPatch, cache: litellm.Cache | None, preset: str | None) -> str:
+    monkeypatch.setattr(litellm, "cache", cache)
+    payload: Final = get_logging_payload(
+        kwargs={
+            "model": "gpt-4o-mini",
+            "messages": [{"role": "user", "content": "x" * 10_000}],
+            "litellm_params": {"metadata": {"user_api_key": "test-key"}, "preset_cache_key": preset},
+        },
+        response_obj=litellm.ModelResponse(id="chatcmpl-test", choices=[], usage=litellm.Usage()),
+        start_time=datetime.datetime.now(timezone.utc),
+        end_time=datetime.datetime.now(timezone.utc),
+    )
+    return payload["cache_key"]
+
+
+def test_get_logging_payload_reuses_the_preset_cache_key_instead_of_hashing_the_body(monkeypatch):
+    assert _cache_key_in_spend_log(monkeypatch, _HashingCache(), "preset-key") == "preset-key"
+
+
+def test_get_logging_payload_records_cache_off_without_hashing(monkeypatch):
+    assert _cache_key_in_spend_log(monkeypatch, None, None) == "Cache OFF"
+
+
+def test_get_logging_payload_still_hashes_when_caching_is_on_and_no_preset_key_exists(monkeypatch):
+    class _RecordingCache(litellm.Cache):
+        def __init__(self) -> None:
+            pass
+
+        def get_cache_key(self, **kwargs) -> str:
+            return "hashed-from-" + kwargs["model"]
+
+    assert _cache_key_in_spend_log(monkeypatch, _RecordingCache(), None) == "hashed-from-gpt-4o-mini"
+
+
 _TRACE_ONLY_STANDARD_LOGGING: Final = cast(
     StandardLoggingPayload,
     {
@@ -613,11 +655,11 @@ def test_sanitize_request_body_for_spend_logs_payload_circular_reference():
     assert sanitized == {"b": {"a": {}}}  # Should return empty dict for circular reference
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_vector_store_request_for_spend_logs_payload_store_prompts_true(
     mock_should_store,
 ):
-    # When _should_store_prompts_and_responses_in_spend_logs returns True
+    # When should_store_prompts_and_responses_in_spend_logs returns True
     mock_should_store.return_value = True
 
     # Sample vector store request metadata
@@ -631,11 +673,11 @@ def test_get_vector_store_request_for_spend_logs_payload_store_prompts_true(
     assert result[0]["vector_store_search_response"]["data"][0]["content"][0]["text"] == "sensitive information"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_vector_store_request_for_spend_logs_payload_store_prompts_false(
     mock_should_store,
 ):
-    # When _should_store_prompts_and_responses_in_spend_logs returns False
+    # When should_store_prompts_and_responses_in_spend_logs returns False
     mock_should_store.return_value = False
 
     # Sample vector store request metadata
@@ -651,7 +693,7 @@ def test_get_vector_store_request_for_spend_logs_payload_store_prompts_false(
     assert result[0]["vector_store_search_response"]["data"][0]["content"][0]["type"] == "text"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_vector_store_request_for_spend_logs_payload_null_input(mock_should_store):
     # When input is None
     mock_should_store.return_value = False
@@ -659,7 +701,7 @@ def test_get_vector_store_request_for_spend_logs_payload_null_input(mock_should_
     assert result is None
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_messages_for_spend_logs_realtime_returns_messages(mock_should_store):
     """
     Test that _get_messages_for_spend_logs_payload returns messages
@@ -686,7 +728,7 @@ def test_get_messages_for_spend_logs_realtime_returns_messages(mock_should_store
     assert parsed[1]["content"] == "What is the weather today?"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_messages_for_spend_logs_strips_null_bytes(mock_should_store):
     """Regression for PostgreSQL 22P05: NUL bytes must be stripped from messages."""
     mock_should_store.return_value = True
@@ -703,7 +745,7 @@ def test_get_messages_for_spend_logs_strips_null_bytes(mock_should_store):
     assert parsed[0]["content"] == "helloworld"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_messages_for_spend_logs_realtime_empty_when_disabled(mock_should_store):
     """
     Test that _get_messages_for_spend_logs_payload returns '{}' for realtime calls
@@ -721,7 +763,7 @@ def test_get_messages_for_spend_logs_realtime_empty_when_disabled(mock_should_st
     assert result == "{}"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_messages_for_spend_logs_non_realtime_returns_empty(mock_should_store):
     """
     Test that _get_messages_for_spend_logs_payload returns '{}' for non-realtime
@@ -739,7 +781,7 @@ def test_get_messages_for_spend_logs_non_realtime_returns_empty(mock_should_stor
     assert result == "{}"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_response_for_spend_logs_payload_truncates_large_base64(mock_should_store):
     from litellm.constants import MAX_STRING_LENGTH_PROMPT_IN_DB
 
@@ -767,7 +809,7 @@ def test_get_response_for_spend_logs_payload_truncates_large_base64(mock_should_
     assert parsed["data"][0]["other_field"] == "value"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_response_for_spend_logs_payload_strips_null_bytes(mock_should_store):
     """Regression for PostgreSQL 22P05: NUL bytes must be stripped from response."""
     mock_should_store.return_value = True
@@ -780,7 +822,7 @@ def test_get_response_for_spend_logs_payload_strips_null_bytes(mock_should_store
     assert json.loads(response_json)["content"] == "answerhere"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_get_response_for_spend_logs_payload_truncates_large_embedding(
     mock_should_store,
 ):
@@ -835,7 +877,7 @@ def test_truncation_includes_db_safeguard_note():
     )
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_response_truncation_logs_info_message(mock_should_store):
     """
     Test that when response is truncated before DB storage, an info log is emitted
@@ -857,7 +899,7 @@ def test_response_truncation_logs_info_message(mock_should_store):
         assert "response was truncated" in log_msg
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_request_body_truncation_logs_info_message(mock_should_store):
     """
     Test that when request body is truncated before DB storage, an info log is emitted.
@@ -1520,7 +1562,7 @@ def test_get_logging_payload_handles_missing_overhead_gracefully():
     )
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_spend_logs_redacts_request_and_response_when_turn_off_message_logging_enabled(
     mock_should_store,
 ):
@@ -1584,7 +1626,7 @@ def test_should_store_prompts_and_responses_in_spend_logs_case_insensitive_strin
     mock_get_secret_bool,
 ):
     """
-    Test that _should_store_prompts_and_responses_in_spend_logs handles
+    Test that should_store_prompts_and_responses_in_spend_logs handles
     case-insensitive string values for store_prompts_in_spend_logs in general_settings.
     """
     # Test case-insensitive string "true" variations
@@ -1594,7 +1636,7 @@ def test_should_store_prompts_and_responses_in_spend_logs_case_insensitive_strin
             {"store_prompts_in_spend_logs": true_value},
         ):
             mock_get_secret_bool.return_value = False  # Ensure env var is False
-            result = _should_store_prompts_and_responses_in_spend_logs()
+            result = should_store_prompts_and_responses_in_spend_logs()
             assert result is True, f"Expected True for '{true_value}', got {result}"
 
     # Test boolean True
@@ -1603,7 +1645,7 @@ def test_should_store_prompts_and_responses_in_spend_logs_case_insensitive_strin
         {"store_prompts_in_spend_logs": True},
     ):
         mock_get_secret_bool.return_value = False
-        result = _should_store_prompts_and_responses_in_spend_logs()
+        result = should_store_prompts_and_responses_in_spend_logs()
         assert result is True, f"Expected True for boolean True, got {result}"
 
     # Test that non-true values fall back to environment variable
@@ -1614,22 +1656,22 @@ def test_should_store_prompts_and_responses_in_spend_logs_case_insensitive_strin
         ):
             # When env var is True, should return True
             mock_get_secret_bool.return_value = True
-            result = _should_store_prompts_and_responses_in_spend_logs()
+            result = should_store_prompts_and_responses_in_spend_logs()
             assert result is True, f"Expected True (from env var) for '{false_value}', got {result}"
 
             # When env var is False, should return False
             mock_get_secret_bool.return_value = False
-            result = _should_store_prompts_and_responses_in_spend_logs()
+            result = should_store_prompts_and_responses_in_spend_logs()
             assert result is False, f"Expected False (from env var) for '{false_value}', got {result}"
 
     # Test when general_settings doesn't have the key at all
     with patch("litellm.proxy.proxy_server.general_settings", {}):
         mock_get_secret_bool.return_value = True
-        result = _should_store_prompts_and_responses_in_spend_logs()
+        result = should_store_prompts_and_responses_in_spend_logs()
         assert result is True, "Expected True (from env var) when key missing, got False"
 
         mock_get_secret_bool.return_value = False
-        result = _should_store_prompts_and_responses_in_spend_logs()
+        result = should_store_prompts_and_responses_in_spend_logs()
         assert result is False, "Expected False (from env var) when key missing, got True"
 
 
@@ -1662,7 +1704,7 @@ def test_get_spend_logs_metadata_guardrail_info_fallback_from_metadata():
     assert result["guardrail_information"] is None
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_redacts_all_prompt_carrying_fields_when_flag_false(
     mock_should_store,
 ):
@@ -1698,7 +1740,7 @@ def test_sanitize_guardrail_information_redacts_all_prompt_carrying_fields_when_
     assert entry["guardrail_action"] == "NONE"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_redacts_prompt_fields_when_flag_false(
     mock_should_store,
 ):
@@ -1762,7 +1804,7 @@ def test_sanitize_guardrail_information_redacts_prompt_fields_when_flag_false(
     }
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_preserves_guardrail_usage_when_flag_false(
     mock_should_store,
 ):
@@ -1794,7 +1836,7 @@ def test_sanitize_guardrail_information_preserves_guardrail_usage_when_flag_fals
     assert entry["guardrail_usage"] == {"topicPolicyUnits": 1, "contentPolicyUnits": 1, "wordPolicyUnits": 0}
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_passthrough_when_flag_true(
     mock_should_store,
 ):
@@ -1817,13 +1859,13 @@ def test_sanitize_guardrail_information_passthrough_when_flag_true(
     assert result == guardrail_info
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_none_passthrough(mock_should_store):
     mock_should_store.return_value = False
     assert _sanitize_guardrail_information_for_spend_logs(None) is None
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_normalizes_bare_dict_input(mock_should_store):
     """
     Regression: xecguard (xecguard.py:246) assigns a bare dict to
@@ -1855,7 +1897,7 @@ def test_sanitize_guardrail_information_normalizes_bare_dict_input(mock_should_s
     assert entry["start_time"] == 1.0
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_drops_non_dict_items_in_list(mock_should_store):
     """
     A stray non-dict item in the list (e.g. from a buggy caller that
@@ -1874,7 +1916,7 @@ def test_sanitize_guardrail_information_drops_non_dict_items_in_list(mock_should
     assert result == [{"guardrail_name": "x", "guardrail_response": REDACTED_BY_LITELM_STRING}]
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_preserves_absent_prompt_fields(mock_should_store):
     """
     Entries that never carried guardrail_request or guardrail_response must
@@ -2290,7 +2332,7 @@ def test_sanitize_request_body_strips_secret_fields():
     assert sanitized["messages"] == [{"role": "user", "content": "hi"}]
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_proxy_server_request_payload_excludes_secret_fields(mock_should_store):
     """
     End-to-end test: when the proxy_server_request body contains
@@ -2370,7 +2412,7 @@ def test_redact_prompt_leaks_empty_string():
     assert _redact_prompt_leaks_in_error_string("") == ""
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_redacts_when_not_storing_prompts(
     mock_should_store,
 ):
@@ -2398,7 +2440,7 @@ def test_sanitize_error_information_redacts_when_not_storing_prompts(
     assert sanitized["llm_provider"] == "openai"
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_skips_redaction_when_storing_prompts(
     mock_should_store,
 ):
@@ -2420,7 +2462,7 @@ def test_sanitize_error_information_skips_redaction_when_storing_prompts(
     assert REDACTED_BY_LITELM_STRING not in sanitized["error_message"]
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_caps_size_regardless_of_prompt_flag(
     mock_should_store,
 ):
@@ -2451,7 +2493,7 @@ def test_sanitize_error_information_none_passthrough():
     assert _sanitize_error_information_for_spend_logs(None) is None
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_reproduces_lit_2992(mock_should_store):
     # Mirrors the reproduced row body from LIT-2992 — a RateLimitError whose
     # message embeds 178 pydantic validation errors, each carrying a full
@@ -2536,7 +2578,7 @@ def test_redact_prompt_leaks_handles_unterminated_value():
     assert REDACTED_BY_LITELM_STRING in redacted
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_redacts_traceback_when_not_storing_prompts(
     mock_should_store,
 ):
@@ -2568,7 +2610,7 @@ def test_sanitize_error_information_redacts_traceback_when_not_storing_prompts(
     assert "ValueError: invalid request" in sanitized["traceback"]
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_skips_traceback_redaction_when_storing_prompts(
     mock_should_store,
 ):
@@ -2676,7 +2718,7 @@ def test_redact_prompt_leaks_combined_quoted_key_and_pydantic_assignment():
     assert redacted.count(REDACTED_BY_LITELM_STRING) >= 2
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_error_information_redacts_pydantic_assignment_form(
     mock_should_store,
 ):
@@ -3272,7 +3314,7 @@ def test_get_logging_payload_hashes_bearer_prefixed_api_key():
     )
 
 
-@patch("litellm.proxy.spend_tracking.spend_tracking_utils._should_store_prompts_and_responses_in_spend_logs")
+@patch("litellm.proxy.spend_tracking.spend_tracking_utils.should_store_prompts_and_responses_in_spend_logs")
 def test_sanitize_guardrail_information_preserves_headroom_compression_token_stats(
     mock_should_store,
 ):

tests/test_litellm/proxy/test_collector.py

@@ -0,0 +1,229 @@
+import asyncio
+import logging
+from collections.abc import Callable, Iterator
+from pathlib import Path
+from typing import Final
+
+import pytest
+import uvloop
+
+from litellm._logging import verbose_logger, verbose_proxy_logger, verbose_router_logger
+from litellm.proxy.collector import (
+    SpendEventConsumer,
+    address_argument,
+    apply_log_level,
+    pod_pgbouncer_database_url,
+)
+from litellm.proxy.db.pgbouncer import PgBouncerError, PgBouncerSettings
+from litellm.proxy.spend_tracking.spend_event_producer import (
+    AddressError,
+    SpendEventProducer,
+    TcpAddress,
+    UnixAddress,
+    open_collector_connection,
+)
+
+
+class _Handler:
+    def __init__(self, fail_on: bytes | None = None) -> None:
+        self.lines: list[bytes] = []  # mutable-ok: test double records the events the consumer handed over
+        self._fail_on = fail_on
+
+    async def __call__(self, line: bytes) -> None:
+        if line == self._fail_on:
+            raise RuntimeError("pipeline failed")
+        self.lines.append(line)
+
+
+async def _no_fallback(line: bytes) -> None:
+    raise AssertionError(f"unexpected fallback for {line!r}")
+
+
+class _Fallback:
+    def __init__(self) -> None:
+        self.lines: list[bytes] = []  # mutable-ok: test double records the events run in-process
+
+    async def __call__(self, line: bytes) -> None:
+        self.lines.append(line)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("transport", ["unix", "tcp"])
+async def test_consumer_handles_each_producer_line_once_in_order(tmp_path: Path, transport: str):
+    handler: Final = _Handler(fail_on=b"event-3\n")
+    consumer: Final = SpendEventConsumer(handler)
+    server: Final = await consumer.serve(
+        UnixAddress(path=str(tmp_path / "spend.sock")) if transport == "unix" else TcpAddress("127.0.0.1", 0)
+    )
+    address: Final = (
+        UnixAddress(path=str(tmp_path / "spend.sock"))
+        if transport == "unix"
+        else TcpAddress("127.0.0.1", server.sockets[0].getsockname()[1])
+    )
+    producer: Final = SpendEventProducer(
+        address=address, on_unavailable="fallback", buffer_size=100, connect_timeout=1.0, fallback=_no_fallback
+    )
+    for i in range(6):
+        await producer.publish(f"event-{i}\n".encode())
+    await producer.close(drain_timeout=5.0)
+
+    server.close()
+    assert await consumer.drain(timeout=5.0) == 0
+    assert handler.lines == [f"event-{i}\n".encode() for i in range(6) if i != 3]
+    assert (consumer.received, consumer.handled, consumer.failed) == (6, 5, 1)
+
+
+@pytest.mark.asyncio
+async def test_consumer_discards_a_truncated_trailing_event(tmp_path: Path):
+    handler: Final = _Handler()
+    consumer: Final = SpendEventConsumer(handler)
+    address: Final = UnixAddress(path=str(tmp_path / "spend.sock"))
+    server: Final = await consumer.serve(address)
+    _, writer = await open_collector_connection(address, timeout=1.0)
+    writer.write(b"whole\npartial-without-newline")
+    await writer.drain()
+    writer.close()
+    await writer.wait_closed()
+    await asyncio.sleep(0.05)
+
+    server.close()
+    assert await consumer.drain(timeout=5.0) == 0
+    assert handler.lines == [b"whole\n"]
+    assert consumer.received == 1
+
+
+@pytest.mark.asyncio
+async def test_drain_reports_producers_still_connected_after_the_timeout(tmp_path: Path):
+    consumer: Final = SpendEventConsumer(_Handler())
+    address: Final = UnixAddress(path=str(tmp_path / "spend.sock"))
+    server: Final = await consumer.serve(address)
+    _, writer = await open_collector_connection(address, timeout=1.0)
+    await asyncio.sleep(0.05)
+
+    server.close()
+    assert await consumer.drain(timeout=0.1) == 1
+    writer.close()
+    await writer.wait_closed()
+    assert await consumer.drain(timeout=5.0) == 0
+
+
+@pytest.mark.asyncio
+async def test_graceful_stop_hands_the_producer_over_to_its_fallback_without_losing_events(tmp_path: Path):
+    handler: Final = _Handler()
+    fallback: Final = _Fallback()
+    consumer: Final = SpendEventConsumer(handler)
+    address: Final = UnixAddress(path=str(tmp_path / "spend.sock"))
+    server: Final = await consumer.serve(address)
+    producer: Final = SpendEventProducer(
+        address=address, on_unavailable="fallback", buffer_size=100, connect_timeout=1.0, fallback=fallback
+    )
+    await producer.publish(b"event-1\n")
+    await asyncio.sleep(0.05)
+
+    server.close()
+    draining: Final = asyncio.ensure_future(consumer.drain(timeout=5.0))
+    await asyncio.sleep(0.05)
+    await producer.publish(b"event-2\n")
+    await producer.close(drain_timeout=5.0)
+
+    assert await draining == 0
+    assert handler.lines == [b"event-1\n"]
+    assert fallback.lines == [b"event-2\n"]
+    assert (producer.stats().sent, producer.stats().fallback) == (1, 1)
+
+
+@pytest.mark.parametrize("loop_factory", [asyncio.new_event_loop, uvloop.new_event_loop], ids=["asyncio", "uvloop"])
+def test_drain_still_hands_over_live_producers_when_another_connection_already_died(
+    tmp_path: Path, loop_factory: Callable[[], asyncio.AbstractEventLoop]
+):
+    """A transport the loop force-closed under a busy handler must not abort the half-close of the others."""
+
+    async def scenario() -> tuple[int, list[bytes]]:
+        release: Final = asyncio.Event()
+
+        async def slow_handler(line: bytes) -> None:
+            await release.wait()
+
+        consumer: Final = SpendEventConsumer(slow_handler)
+        address: Final = UnixAddress(path=str(tmp_path / "spend.sock"))
+        server: Final = await consumer.serve(address)
+        _, dead = await open_collector_connection(address, timeout=1.0)
+        dead.write(b"stuck\n")
+        await dead.drain()
+        await asyncio.sleep(0.05)
+        for connection in consumer._open_connections:  # pyright: ignore[reportPrivateUsage]  # force-close like uvloop does on a socket error
+            connection.transport.close()
+        dead.close()
+        fallback: Final = _Fallback()
+        producer: Final = SpendEventProducer(
+            address=address, on_unavailable="fallback", buffer_size=100, connect_timeout=1.0, fallback=fallback
+        )
+        await producer.publish(b"event-1\n")
+        await asyncio.sleep(0.05)
+
+        server.close()
+        draining: Final = asyncio.ensure_future(consumer.drain(timeout=0.5))
+        await asyncio.sleep(0.05)
+        await producer.publish(b"event-2\n")
+        await producer.close(drain_timeout=5.0)
+        still_open: Final = await draining
+        release.set()
+        await asyncio.sleep(0.05)
+        return still_open, fallback.lines
+
+    with asyncio.Runner(loop_factory=loop_factory) as runner:
+        still_open, fallback_lines = runner.run(scenario())
+
+    assert still_open == 2
+    assert fallback_lines == [b"event-2\n"]
+
+
+def test_address_argument():
+    assert address_argument((), default="unix:///tmp/x.sock") == "unix:///tmp/x.sock"
+    assert address_argument(("--address", "tcp://127.0.0.1:4100"), default="unix:///tmp/x.sock") == (
+        "tcp://127.0.0.1:4100"
+    )
+    assert isinstance(address_argument(("--listen", "x"), default="unix:///tmp/x.sock"), AddressError)
+
+
+def test_pod_pgbouncer_database_url_points_at_the_proxy_containers_pooler():
+    """With pgbouncer on, the sidecar must not open its own upstream connections but share the pod's pooler."""
+    upstream: Final = "postgresql://u:p@db.internal:5432/litellm?schema=public"
+    environ: Final = {"DATABASE_URL": upstream}
+    assert pod_pgbouncer_database_url(PgBouncerSettings(enabled=False), environ, token_auth=False) is None
+    assert (
+        pod_pgbouncer_database_url(PgBouncerSettings(enabled=True, port=6543), environ, token_auth=False)
+        == "postgresql://u:p@127.0.0.1:6543/litellm?schema=public&pgbouncer=true"
+    )
+    assert isinstance(pod_pgbouncer_database_url(PgBouncerSettings(enabled=True), {}, token_auth=False), PgBouncerError)
+
+
+def test_pod_pgbouncer_database_url_goes_direct_under_token_auth():
+    """The proxy's pgbouncer only knows the token that container minted, so the sidecar must mint its own upstream."""
+    iam_upstream: Final = "postgresql://u@db.internal:5432/litellm?schema=public"
+    assert (
+        pod_pgbouncer_database_url(PgBouncerSettings(enabled=True), {"DATABASE_URL": iam_upstream}, token_auth=True)
+        is None
+    )
+    assert pod_pgbouncer_database_url(PgBouncerSettings(enabled=True), {}, token_auth=True) is None
+
+
+@pytest.fixture
+def restore_log_levels() -> Iterator[None]:
+    loggers: Final = (verbose_logger, verbose_router_logger, verbose_proxy_logger)
+    levels: Final = tuple(logger.level for logger in loggers)
+    yield
+    for logger, level in zip(loggers, levels, strict=True):
+        logger.setLevel(level)
+
+
+@pytest.mark.usefixtures("restore_log_levels")
+@pytest.mark.parametrize(
+    ("litellm_log", "expected"),
+    [("DEBUG", logging.DEBUG), ("info", logging.INFO), (None, logging.WARNING), ("loud", logging.WARNING)],
+)
+def test_apply_log_level_mirrors_the_proxy_env_contract(litellm_log: str | None, expected: int):
+    verbose_proxy_logger.setLevel(logging.WARNING)
+    apply_log_level(litellm_log)
+    assert verbose_proxy_logger.isEnabledFor(expected)
+    assert not verbose_proxy_logger.isEnabledFor(expected - 10)