scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-12 08:42:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 207

feat(proxy): authenticate billing metrics with a cognito bearer token (#49852)

Browse source 
* feat(billing): support Cognito bearer auth for metrics

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): refetch billing token on 401 and require https

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): import Unpack from typing_extensions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): make billing metrics bearer only and drop mtls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): format billing metrics module

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Yassin Kortam 2026-11-04 14:08:40 -08:00 • committed by GitHub
parent 9703e01cd4
commit 732a50318b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
30 changed files with 773 additions and 1363 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
helm/litellm-helm/Chart.yaml
View file

@ -18,7 +18,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 1.1.3
version: 1.1.4
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to

11
helm/litellm-helm/README.md
View file

@ -56,8 +56,8 @@ If `db.useStackgresOperator` is used (not yet implemented):
| `billingMetrics.enabled` | Enable enterprise billable-request metering. Requires an enterprise license. | `false` |
| `billingMetrics.endpoint` | Collector that the billable-request counter is pushed to. | `https://telemetry.litellm.ai` |
| `billingMetrics.secretName` | Name of an existing Secret holding the mTLS client certificate, under the keys `tls.crt` and `tls.key`. | `litellm-billing-metrics-mtls` |
| `billingMetrics.caSecretName` | Name of an existing Secret holding a CA bundle under the key `ca.crt`. Only needed for a private or test collector whose server certificate is not on the public web PKI. | `""` |
| `billingMetrics.tokenUrl` | Cognito OAuth2 client-credentials token endpoint. | `https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token` |
| `billingMetrics.secretName` | Name of an existing Secret with `client-id` and `client-secret` keys. | `litellm-billing-metrics` |
| `billingMetrics.exportIntervalMs` | How often the counter is pushed, in milliseconds. The proxy defaults to `60000` when unset. | `""` |
#### Example `proxy_config` ConfigMap from values (default):
@ -109,10 +109,10 @@ type: Opaque
#### Enterprise billable-request metering
Enterprise licenses meter billable requests by pushing a counter to LiteLLM's collector over mutual TLS. The chart does not create the client certificate; it mounts one you already hold, read-only, so the private key is never exposed through the environment. Create the Secret under the name the chart expects, then turn the block on:
Enterprise licenses meter billable requests by pushing a counter to LiteLLM's collector with Cognito bearer authentication. Create a Secret with `client-id` and `client-secret` keys, then enable the chart values:
```
kubectl create secret tls litellm-billing-metrics-mtls --cert=client.crt --key=client.key
kubectl create secret generic litellm-billing-metrics --from-literal=client-id=... --from-literal=client-secret=...
```
```
@ -120,7 +120,8 @@ billingMetrics:
enabled: true
```
Set `billingMetrics.caSecretName` only when the collector is a private or test one whose server certificate is not on the public web PKI; the production collector needs no CA override. The chart fails the render rather than deploying a proxy that silently never exports, so a missing `secretName` or an emptied `endpoint` surfaces at `helm install` time.
The endpoint and token URL must use HTTPS. The chart requires both URLs and
the Secret name when billing metrics are enabled
### Database Settings

50
helm/litellm-helm/templates/_helpers.tpl
View file

@ -50,53 +50,27 @@ app.kubernetes.io/name: {{ include "litellm.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Enterprise billable-request metering. The client certificate identifies the
deployment to LiteLLM's collector, so it is mounted read-only from an existing
Secret rather than passed through the environment.
*/}}
{{- define "litellm.billingMetrics.certDir" -}}/etc/litellm/billing-mtls{{- end -}}
{{- define "litellm.billingMetrics.caDir" -}}/etc/litellm/billing-mtls-ca{{- end -}}
{{- define "litellm.billingMetricsEnv" -}}
- name: LITELLM_BILLING_METRICS_ENDPOINT
value: {{ required "billingMetrics.endpoint is required when billingMetrics.enabled is true" .Values.billingMetrics.endpoint | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: {{ printf "%s/tls.crt" (include "litellm.billingMetrics.certDir" .) | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: {{ printf "%s/tls.key" (include "litellm.billingMetrics.certDir" .) | quote }}
{{- if .Values.billingMetrics.caSecretName }}
- name: LITELLM_BILLING_METRICS_CA_CERT
value: {{ printf "%s/ca.crt" (include "litellm.billingMetrics.caDir" .) | quote }}
{{- end }}
- name: LITELLM_BILLING_METRICS_TOKEN_URL
value: {{ required "billingMetrics.tokenUrl is required when billingMetrics.enabled is true" .Values.billingMetrics.tokenUrl | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_ID
valueFrom:
secretKeyRef:
name: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true" .Values.billingMetrics.secretName }}
key: client-id
- name: LITELLM_BILLING_METRICS_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true" .Values.billingMetrics.secretName }}
key: client-secret
{{- with .Values.billingMetrics.exportIntervalMs }}
- name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: {{ . | quote }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumes" -}}
- name: billing-metrics-mtls
secret:
secretName: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)" .Values.billingMetrics.secretName }}
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
secret:
secretName: {{ .Values.billingMetrics.caSecretName }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumeMounts" -}}
- name: billing-metrics-mtls
mountPath: {{ include "litellm.billingMetrics.certDir" . }}
readOnly: true
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
mountPath: {{ include "litellm.billingMetrics.caDir" . }}
readOnly: true
{{- end }}
{{- end -}}
{{/*
Create the name of the service account to use
*/}}

6
helm/litellm-helm/templates/deployment.yaml
View file

@ -148,9 +148,6 @@ spec:
- name: npm
mountPath: /.npm
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
{{- end }}
{{- if include "litellm.collector.socketDir" . }}
- name: collector-socket
mountPath: {{ include "litellm.collector.socketDir" . }}
@ -234,9 +231,6 @@ spec:
items:
- key: {{ .Values.proxyConfigMap.key | default "config.yaml" }}
path: "config.yaml"
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
{{- end }}
{{- if include "litellm.collector.socketDir" . }}
- name: collector-socket
emptyDir:

283
helm/litellm-helm/tests/billing_metrics_tests.yaml
View file

@ -1,31 +1,21 @@
suite: test billingMetrics wiring on the proxy deployment
suite: test bearer-only billingMetrics wiring
templates:
- deployment.yaml
- configmap-litellm.yaml
- migrations-job.yaml
tests:
- it: is off by default, adding no env, volume, or mount
- it: is disabled by default
template: deployment.yaml
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics
- it: renders the endpoint and the mounted cert paths when enabled
- it: renders required URLs and Cognito secret references when enabled
template: deployment.yaml
set:
billingMetrics:
@ -39,119 +29,34 @@ tests:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: /etc/litellm/billing-mtls/tls.crt
name: LITELLM_BILLING_METRICS_TOKEN_URL
value: https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: /etc/litellm/billing-mtls/tls.key
# The conventional Secret name is the default, so enabling the block is enough.
- it: mounts the default cert secret read-only alongside the config volume
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- it: honours a secretName override
template: deployment.yaml
set:
billingMetrics:
enabled: true
secretName: my-billing-mtls
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: my-billing-mtls
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- it: honours an endpoint override
template: deployment.yaml
set:
billingMetrics:
enabled: true
endpoint: https://collector.internal:4318
asserts:
name: LITELLM_BILLING_METRICS_CLIENT_ID
valueFrom:
secretKeyRef:
name: litellm-billing-metrics
key: client-id
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://collector.internal:4318
# The production collector presents a public web-PKI certificate, so the CA
# override must stay absent unless a private collector is configured.
- it: omits the CA env, volume, and mount when no caSecretName is set
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
name: LITELLM_BILLING_METRICS_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: litellm-billing-metrics
key: client-secret
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
name: billing-metrics
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls-ca
mountPath: /etc/litellm/billing-mtls-ca
readOnly: true
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
name: billing-metrics
- it: mounts the CA secret when caSecretName is set
template: deployment.yaml
set:
billingMetrics:
enabled: true
caSecretName: billing-ca
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls-ca
mountPath: /etc/litellm/billing-mtls-ca
readOnly: true
- it: passes the export interval through only when set
- it: renders the export interval only when set
template: deployment.yaml
set:
billingMetrics:
@ -164,129 +69,7 @@ tests:
name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: "5000"
- it: omits the export interval when unset
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: "60000"
# Kubernetes resolves duplicate env names last-wins, so the chart-owned billing
# entries must render after .Values.envVars or a user could silently redirect
# the metering export. The three billing entries are the last ones emitted here
# (migrationJob, which appends DISABLE_SCHEMA_UPDATE, is off for this case).
- it: renders the billing endpoint after envVars so it cannot be shadowed
template: deployment.yaml
set:
migrationJob:
enabled: false
billingMetrics:
enabled: true
envVars:
LITELLM_BILLING_METRICS_ENDPOINT: https://shadowed.example
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://shadowed.example
- equal:
path: spec.template.spec.containers[0].env[-3]
value:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- equal:
path: spec.template.spec.containers[0].env[-2].name
value: LITELLM_BILLING_METRICS_CLIENT_CERT
- equal:
path: spec.template.spec.containers[0].env[-1].name
value: LITELLM_BILLING_METRICS_CLIENT_KEY
- it: keeps user-supplied volumes and mounts alongside the billing secret
template: deployment.yaml
set:
billingMetrics:
enabled: true
volumes:
- name: custom-callbacks
configMap:
name: my-callbacks
volumeMounts:
- name: custom-callbacks
mountPath: /app/callbacks
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: custom-callbacks
configMap:
name: my-callbacks
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: custom-callbacks
mountPath: /app/callbacks
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- it: still mounts the proxy config when enabled
template: deployment.yaml
set:
billingMetrics:
enabled: true
asserts:
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: litellm-config
mountPath: /etc/litellm/config.yaml
subPath: config.yaml
# Only the proxy serves billable traffic. The migrations Job must never mount
# the client certificate, and it renders its own env and volumes, so nothing
# stops a future edit from wiring the billing include into it by mistake.
- it: does not touch the migrations job when enabled
template: migrations-job.yaml
set:
billingMetrics:
enabled: true
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- notExists:
path: spec.template.spec.containers[0].volumeMounts
- notExists:
path: spec.template.spec.volumes
- it: fails loudly when enabled with an emptied secretName
template: deployment.yaml
set:
billingMetrics:
enabled: true
secretName: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)
- it: fails loudly when enabled without an endpoint
- it: requires the endpoint when enabled
template: deployment.yaml
set:
billingMetrics:
@ -295,3 +78,23 @@ tests:
asserts:
- failedTemplate:
errorMessage: billingMetrics.endpoint is required when billingMetrics.enabled is true
- it: requires the token URL when enabled
template: deployment.yaml
set:
billingMetrics:
enabled: true
tokenUrl: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.tokenUrl is required when billingMetrics.enabled is true
- it: requires the secret name when enabled
template: deployment.yaml
set:
billingMetrics:
enabled: true
secretName: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.secretName is required when billingMetrics.enabled is true

6
helm/litellm-helm/tests/collector_tests.yaml
View file

@ -191,7 +191,7 @@ tests:
metricsServer.port: 9090
billingMetrics.enabled: true
billingMetrics.endpoint: https://metering.example.com
billingMetrics.secretName: billing-mtls
billingMetrics.secretName: billing-metrics
asserts:
- contains:
path: spec.template.spec.containers[0].env
@ -214,9 +214,9 @@ tests:
name: LITELLM_BILLING_METRICS_ENDPOINT
any: true
- notContains:
path: spec.template.spec.containers[1].volumeMounts
path: spec.template.spec.containers[1].env
content:
name: billing-metrics-mtls
name: LITELLM_BILLING_METRICS_CLIENT_SECRET
any: true
- it: should give the sidecar the same scratch mounts as the proxy on a read-only root

16
helm/litellm-helm/values.yaml
View file

@ -154,19 +154,13 @@ masterkeySecretName: ""
# if set, use this secret key for the master key; otherwise, use the default key
masterkeySecretKey: ""
# Optional: enterprise billable-request metering. When enabled, the proxy counts
# successful requests to inference, MCP, and A2A endpoints and pushes them to
# LiteLLM's collector over mutual TLS. Requires an enterprise license.
# The client certificate identifies the deployment, so it is mounted read-only
# from an existing Secret and never passed through the environment.
# Optional: enterprise billable-request metering. Requires an enterprise license.
billingMetrics:
enabled: false
endpoint: https://telemetry.litellm.ai # collector to push the counter to
secretName: litellm-billing-metrics-mtls # existing Secret holding tls.crt and tls.key
# Only for private or test collectors whose server certificate is not on the
# public web PKI. The production collector needs no CA override.
caSecretName: "" # existing Secret holding ca.crt
exportIntervalMs: "" # push cadence; the proxy defaults to 60000
endpoint: https://telemetry.litellm.ai
tokenUrl: https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token
secretName: litellm-billing-metrics
exportIntervalMs: ""
proxyConfigMap:
# when true, creates a new configmap

2
helm/litellm/Chart.yaml
View file

@ -2,7 +2,7 @@ apiVersion: v2
name: litellm
description: LiteLLM componentized — gateway, UI backend, and UI as separate services
type: application
version: 0.1.0
version: 0.1.1
appVersion: "0.1.0"
annotations:
org.opencontainers.image.source: "https://github.com/BerriAI/litellm"

17
helm/litellm/README.md
View file

@ -22,6 +22,23 @@ backend:
gatewayUrl: https://gateway.example.com
```
## Enterprise billing metrics
Billing metrics use Cognito bearer authentication. Create the Secret with
`client-id` and `client-secret` keys, then enable billing metrics:
```bash
kubectl create secret generic litellm-billing-metrics --from-literal=client-id=... --from-literal=client-secret=...
```
```yaml
billingMetrics:
enabled: true
```
The chart requires an HTTPS endpoint, an HTTPS token URL, and the Secret name
when billing metrics are enabled
## Routes to another Service
For a shared hostname, `ingress.extraPaths` can route to a Service in the same namespace using `serviceName` and `servicePort`. The port can be a number or a named Service port. Omit `service` when using these fields

53
helm/litellm/templates/_helpers.tpl
View file

@ -58,56 +58,27 @@ app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" }}
{{- end -}}
{{/*
Enterprise billable-request metering. Wired into gateway and backend, not the
migrations job. The gateway serves nearly all billable traffic, but the backend
keeps the named-server MCP transport (/{mcp_server_name}/mcp), which writes a
SpendLogs row, so metering only the gateway would silently drop that traffic.
The client certificate identifies the deployment to LiteLLM's collector, so it is
mounted read-only from an existing Secret rather than passed through the
environment.
*/}}
{{- define "litellm.billingMetrics.certDir" -}}/etc/litellm/billing-mtls{{- end -}}
{{- define "litellm.billingMetrics.caDir" -}}/etc/litellm/billing-mtls-ca{{- end -}}
{{- define "litellm.billingMetricsEnv" -}}
- name: LITELLM_BILLING_METRICS_ENDPOINT
value: {{ required "billingMetrics.endpoint is required when billingMetrics.enabled is true" .Values.billingMetrics.endpoint | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: {{ printf "%s/tls.crt" (include "litellm.billingMetrics.certDir" .) | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: {{ printf "%s/tls.key" (include "litellm.billingMetrics.certDir" .) | quote }}
{{- if .Values.billingMetrics.caSecretName }}
- name: LITELLM_BILLING_METRICS_CA_CERT
value: {{ printf "%s/ca.crt" (include "litellm.billingMetrics.caDir" .) | quote }}
{{- end }}
- name: LITELLM_BILLING_METRICS_TOKEN_URL
value: {{ required "billingMetrics.tokenUrl is required when billingMetrics.enabled is true" .Values.billingMetrics.tokenUrl | quote }}
- name: LITELLM_BILLING_METRICS_CLIENT_ID
valueFrom:
secretKeyRef:
name: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true" .Values.billingMetrics.secretName }}
key: client-id
- name: LITELLM_BILLING_METRICS_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true" .Values.billingMetrics.secretName }}
key: client-secret
{{- with .Values.billingMetrics.exportIntervalMs }}
- name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: {{ . | quote }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumes" -}}
- name: billing-metrics-mtls
secret:
secretName: {{ required "billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)" .Values.billingMetrics.secretName }}
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
secret:
secretName: {{ .Values.billingMetrics.caSecretName }}
{{- end }}
{{- end -}}
{{- define "litellm.billingMetricsVolumeMounts" -}}
- name: billing-metrics-mtls
mountPath: {{ include "litellm.billingMetrics.certDir" . }}
readOnly: true
{{- if .Values.billingMetrics.caSecretName }}
- name: billing-metrics-mtls-ca
mountPath: {{ include "litellm.billingMetrics.caDir" . }}
readOnly: true
{{- end }}
{{- end -}}
{{/*
Per-component selector labels — used in both Service selectors and Deployment matchLabels.

10
helm/litellm/templates/backend/deployment.yaml
View file

@ -79,16 +79,13 @@ spec:
{{- include "litellm.billingMetricsEnv" . | nindent 12 }}
{{- end }}
{{- include "litellm.envFrom" .Values.backend | nindent 10 }}
{{- if or .Values.gateway.config.create .Values.backend.volumeMounts .Values.billingMetrics.enabled }}
{{- if or .Values.gateway.config.create .Values.backend.volumeMounts }}
volumeMounts:
{{- if .Values.gateway.config.create }}
- name: gateway-config
mountPath: /app/config/config.yaml
subPath: config.yaml
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
{{- end }}
{{- with .Values.backend.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
@ -114,16 +111,13 @@ spec:
{{- with .Values.backend.extraContainers }}
{{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- if or .Values.gateway.config.create .Values.backend.volumes .Values.billingMetrics.enabled }}
{{- if or .Values.gateway.config.create .Values.backend.volumes }}
volumes:
{{- if .Values.gateway.config.create }}
- name: gateway-config
configMap:
name: {{ include "litellm.gateway.fullname" . }}-config
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
{{- end }}
{{- with .Values.backend.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}

10
helm/litellm/templates/gateway/deployment.yaml
View file

@ -86,7 +86,7 @@ spec:
{{- include "litellm.gateway.collectorEnv" . | nindent 12 }}
{{- end }}
{{- include "litellm.envFrom" .Values.gateway | nindent 10 }}
{{- if or .Values.gateway.config.create .Values.gateway.volumeMounts .Values.billingMetrics.enabled .Values.gateway.metricsServer.enabled (include "litellm.gateway.collectorSocketDir" .) }}
{{- if or .Values.gateway.config.create .Values.gateway.volumeMounts .Values.gateway.metricsServer.enabled (include "litellm.gateway.collectorSocketDir" .) }}
volumeMounts:
{{- if .Values.gateway.config.create }}
- name: gateway-config
@ -101,9 +101,6 @@ spec:
- name: collector-socket
mountPath: {{ include "litellm.gateway.collectorSocketDir" . }}
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumeMounts" . | nindent 12 }}
{{- end }}
{{- with .Values.gateway.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
@ -206,7 +203,7 @@ spec:
{{- with .Values.gateway.extraContainers }}
{{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- if or .Values.gateway.config.create .Values.gateway.volumes .Values.billingMetrics.enabled .Values.gateway.metricsServer.enabled (include "litellm.gateway.collectorSocketDir" .) }}
{{- if or .Values.gateway.config.create .Values.gateway.volumes .Values.gateway.metricsServer.enabled (include "litellm.gateway.collectorSocketDir" .) }}
volumes:
{{- if .Values.gateway.config.create }}
- name: gateway-config
@ -222,9 +219,6 @@ spec:
emptyDir:
sizeLimit: 1Mi
{{- end }}
{{- if .Values.billingMetrics.enabled }}
{{- include "litellm.billingMetricsVolumes" . | nindent 8 }}
{{- end }}
{{- with .Values.gateway.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}

224
helm/litellm/tests/billing_metrics_tests.yaml
View file

@ -1,39 +1,30 @@
suite: test billingMetrics wiring on gateway and backend
suite: test bearer-only billingMetrics wiring
templates:
- gateway/deployment.yaml
- gateway/configmap.yaml
- backend/deployment.yaml
- backend/configmap.yaml
- migrations-job.yaml
values:
- ./values/required.yaml
tests:
- it: is off by default, adding no env, volume, or mount
- it: is disabled by default
template: gateway/deployment.yaml
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- equal:
path: spec.template.spec.containers[0].volumeMounts
value:
- name: gateway-config
mountPath: /app/config/config.yaml
subPath: config.yaml
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics
- it: renders the endpoint and the mounted cert paths when enabled
- it: wires gateway URLs and bearer credentials from Secret keys
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- contains:
path: spec.template.spec.containers[0].env
@ -43,87 +34,68 @@ tests:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_CERT
value: /etc/litellm/billing-mtls/tls.crt
name: LITELLM_BILLING_METRICS_TOKEN_URL
value: https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CLIENT_KEY
value: /etc/litellm/billing-mtls/tls.key
- it: mounts the cert secret read-only alongside the config volume
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
name: LITELLM_BILLING_METRICS_CLIENT_ID
valueFrom:
secretKeyRef:
name: litellm-billing-metrics
key: client-id
- contains:
path: spec.template.spec.volumes
path: spec.template.spec.containers[0].env
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
# The production collector presents a public web-PKI certificate, so the CA
# override must stay absent unless a private collector is configured.
- it: omits the CA env, volume, and mount when no caSecretName is set
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
name: LITELLM_BILLING_METRICS_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: litellm-billing-metrics
key: client-secret
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
name: billing-metrics
- notContains:
path: spec.template.spec.containers[0].env
path: spec.template.spec.containers[0].volumeMounts
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
name: billing-metrics
- it: mounts the CA secret when caSecretName is set
template: gateway/deployment.yaml
- it: wires billing metrics on the backend without volumes
template: backend/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
caSecretName: billing-ca
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_CA_CERT
value: /etc/litellm/billing-mtls-ca/ca.crt
- contains:
name: LITELLM_BILLING_METRICS_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: litellm-billing-metrics
key: client-secret
- notContains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls-ca
secret:
secretName: billing-ca
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls-ca
mountPath: /etc/litellm/billing-mtls-ca
readOnly: true
name: billing-metrics
- it: passes the export interval through only when set
- it: does not add billing metrics to migrations
template: migrations-job.yaml
set:
billingMetrics:
enabled: true
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
- it: renders an optional export interval
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
exportIntervalMs: 5000
asserts:
- contains:
@ -132,102 +104,27 @@ tests:
name: LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS
value: "5000"
- it: keeps user-supplied gateway volumes alongside the billing secret
- it: requires the endpoint when enabled
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
gateway.volumes:
- name: custom-callbacks
configMap:
name: my-callbacks
gateway.volumeMounts:
- name: custom-callbacks
mountPath: /app/callbacks
endpoint: ""
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: custom-callbacks
configMap:
name: my-callbacks
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- failedTemplate:
errorMessage: billingMetrics.endpoint is required when billingMetrics.enabled is true
# The backend keeps the named-server MCP transport (/{mcp_server_name}/mcp),
# which writes a SpendLogs row, so it must meter too or that traffic is lost.
- it: meters the backend as well, since it serves the MCP transport
template: backend/deployment.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: billing-metrics-mtls
mountPath: /etc/litellm/billing-mtls
readOnly: true
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: billing-mtls
- it: leaves the backend alone when metering is off
template: backend/deployment.yaml
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
# The migrations job runs prisma and serves no traffic; it must never receive
# the client key.
- it: never mounts the billing cert on the migrations job
template: migrations-job.yaml
set:
billingMetrics:
enabled: true
secretName: billing-mtls
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_BILLING_METRICS_ENDPOINT
value: https://telemetry.litellm.ai
- isNull:
path: spec.template.spec.volumes
# The conventional Secret name is the default, so enabling metering needs no
# secretName at all; the guard below only fires on an explicitly blanked one.
- it: uses the conventional secret name by default
- it: requires the token URL when enabled
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
tokenUrl: ""
asserts:
- contains:
path: spec.template.spec.volumes
content:
name: billing-metrics-mtls
secret:
secretName: litellm-billing-metrics-mtls
- failedTemplate:
errorMessage: billingMetrics.tokenUrl is required when billingMetrics.enabled is true
- it: fails loudly when the secretName is explicitly blanked
- it: requires the secret name when enabled
template: gateway/deployment.yaml
set:
billingMetrics:
@ -235,15 +132,4 @@ tests:
secretName: ""
asserts:
- failedTemplate:
errorMessage: billingMetrics.secretName is required when billingMetrics.enabled is true (an existing Secret with tls.crt and tls.key)
- it: fails loudly when enabled without an endpoint
template: gateway/deployment.yaml
set:
billingMetrics:
enabled: true
endpoint: ""
secretName: billing-mtls
asserts:
- failedTemplate:
errorMessage: billingMetrics.endpoint is required when billingMetrics.enabled is true
errorMessage: billingMetrics.secretName is required when billingMetrics.enabled is true

2
helm/litellm/tests/migration_job_tests.yaml
View file

@ -91,7 +91,7 @@ tests:
app.kubernetes.io/name: litellm
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
helm.sh/chart: litellm-0.1.0
helm.sh/chart: litellm-0.1.1
app.kubernetes.io/component: migrations
- it: renders pod-level and container-level securityContext in their own scopes

21
helm/litellm/values.yaml
View file

@ -180,23 +180,14 @@ masterKey:
secretKey: master-key
# Optional: enterprise billable-request metering. When enabled, the gateway and
# backend count successful requests to inference, MCP, and A2A endpoints and push
# them to LiteLLM's collector over mutual TLS. Both components serve billable
# routes: the backend keeps the named-server MCP transport. Requires an
# enterprise license. The client certificate identifies the deployment, so it is
# mounted read-only from an existing Secret and never passed through the env.
# backend count successful requests to inference, MCP, and A2A endpoints. Requires
# an enterprise license.
billingMetrics:
enabled: false
endpoint: https://telemetry.litellm.ai # collector to push the counter to
# An existing Secret holding the client certificate under tls.crt and its key
# under tls.key, usually created from the onboarding artifact. The default is
# the conventional name, so the common path is to create that Secret and set
# enabled: true. Override only if yours is named differently.
secretName: litellm-billing-metrics-mtls
# Only for private or test collectors whose server certificate is not on the
# public web PKI. The production collector needs no CA override.
caSecretName: "" # existing Secret holding ca.crt
exportIntervalMs: "" # push cadence; the proxy defaults to 60000
endpoint: https://telemetry.litellm.ai
tokenUrl: https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token
secretName: litellm-billing-metrics
exportIntervalMs: ""
# External Postgres connection.
database:

8
litellm/litellm_core_utils/url_utils.py
View file

@ -102,6 +102,14 @@ class SSRFError(ValueError):
"""Raised when a URL targets a blocked network."""
def is_https_url_with_host(url: str) -> bool:
try:
parsed_url: Final = urlsplit(url)
return parsed_url.scheme == "https" and bool(parsed_url.hostname)
except ValueError:
return False
def encode_url_path_segment(value: object, *, field_name: str = "path parameter") -> str:
"""Percent-encode one user-controlled URL path segment.

12
litellm/proxy/anthropic_endpoints/claude_code_endpoints/claude_code_marketplace.py
View file

@ -26,7 +26,6 @@ from collections.abc import Mapping, Sequence
from datetime import datetime, timezone
from types import MappingProxyType
from typing import Annotated, Final, Protocol, TypedDict, get_args
from urllib.parse import urlsplit
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
from fastapi.responses import JSONResponse, Response
@ -34,6 +33,7 @@ from pydantic import Json, TypeAdapter
from typing_extensions import ReadOnly
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.url_utils import is_https_url_with_host
from litellm.models.plugin_upload import PluginUpload
from litellm.proxy._types import CommonProxyErrors, ProxyErrorTypes, ProxyException, UserAPIKeyAuth
from litellm.proxy.anthropic_endpoints.claude_code_endpoints.claude_code_skill_access import (
@ -377,14 +377,6 @@ _VALID_GIT_SUBDIR_PATH_RE: Final = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9._-]*(/[a-
_VALID_SHA256_RE: Final = re.compile(r"^[0-9a-fA-F]{64}$")
def _is_https_url_with_host(url: str) -> bool:
try:
parts: Final = urlsplit(url)
except ValueError:
return False
return parts.scheme == "https" and bool(parts.hostname)
def _validate_plugin_source(source: Mapping[str, str]) -> None:
"""Validate plugin source format, raising HTTPException on invalid input."""
source_type: Final = source.get("source")
@ -421,7 +413,7 @@ def _validate_plugin_source(source: Mapping[str, str]) -> None:
},
)
elif source_type == "archive":
if not _is_https_url_with_host(source.get("url", "")):
if not is_https_url_with_host(source.get("url", "")):
raise HTTPException(
status_code=400,
detail={

282
litellm/proxy/enterprise_billing/billing_metrics.py
View file

@ -2,29 +2,33 @@
Push-based OTLP metering for enterprise litellm deployments.
Owns a dedicated OpenTelemetry meter provider and an OTLP/HTTP exporter
authenticated to our global collector with a TLS client certificate. The
collector front end terminates mutual TLS: the client certificate presented
here is validated against our CA at the edge, and the verified subject is
what identifies the deployment. It is intentionally isolated from the global
meter provider so the customer's own OTEL metrics are untouched and ours
never leak into their backend.
authenticated to our global collector with a Cognito bearer token. It is
intentionally isolated from the global meter provider so the customer's own
OTEL metrics are untouched and ours never leak into their backend.
The deployment's identity rides on the TLS client certificate, not on the
payload; the secret license key is never sent as an attribute or header.
The secret license key is never sent as an attribute or header.
"""
import os
import tempfile
from dataclasses import dataclass
from typing import TYPE_CHECKING, Final, Optional
import threading
import time
from collections.abc import Callable
from dataclasses import dataclass, field
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, TypedDict
import requests
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.metrics import Counter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from pydantic import BaseModel, ConfigDict
from requests.auth import AuthBase
from typing_extensions import ReadOnly, Unpack
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.url_utils import is_https_url_with_host
from litellm.proxy.middleware.billable_request_metrics_middleware import (
BillableCategory,
)
@ -33,23 +37,15 @@ if TYPE_CHECKING:
from litellm.proxy._types import EnterpriseLicenseData
ENDPOINT_ENV: Final = "LITELLM_BILLING_METRICS_ENDPOINT"
CLIENT_CERT_ENV: Final = "LITELLM_BILLING_METRICS_CLIENT_CERT"
CLIENT_KEY_ENV: Final = "LITELLM_BILLING_METRICS_CLIENT_KEY"
CA_CERT_ENV: Final = "LITELLM_BILLING_METRICS_CA_CERT"
CLIENT_ID_ENV: Final = "LITELLM_BILLING_METRICS_CLIENT_ID"
CLIENT_SECRET_ENV: Final = "LITELLM_BILLING_METRICS_CLIENT_SECRET"
TOKEN_URL_ENV: Final = "LITELLM_BILLING_METRICS_TOKEN_URL"
EXPORT_INTERVAL_ENV: Final = "LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS"
DEFAULT_EXPORT_INTERVAL_MS: Final = 60_000
SHUTDOWN_FLUSH_TIMEOUT_MS: Final = 5_000
_TOKEN_REFRESH_SKEW_SECONDS: Final = 60
_METRICS_PATH: Final = "/v1/metrics"
# The cert env vars take a path or the PEM itself. Secret stores that inject
# values as env content cannot mount them as files, so inline PEM is written out.
_PEM_PREFIX: Final = "-----BEGIN"
_PEM_DIR_PREFIX: Final = "litellm-billing-mtls-"
_PEM_FILE_MODE: Final = 0o600
_CLIENT_CERT_FILENAME: Final = "client.crt"
_CLIENT_KEY_FILENAME: Final = "client.key"
_CA_CERT_FILENAME: Final = "ca.crt"
METRIC_NAME: Final = "litellm.enterprise.billable_requests"
METER_NAME: Final = "litellm.enterprise.billing"
@ -59,21 +55,28 @@ AttributeValue = str | int
@dataclass(frozen=True, slots=True)
class BillingMetricsConfig:
endpoint: str
client_cert_path: str
client_key_path: str
ca_cert_path: str | None
token_url: str
client_id: str
client_secret: str = field(repr=False)
export_interval_ms: int
litellm_version: str
license_id: str | None
class _BillingMetricsOAuthToken(BaseModel):
model_config = ConfigDict(frozen=True, strict=True)
access_token: str
expires_in: int
def _metrics_endpoint(endpoint: str) -> str:
"""The OTLP/HTTP metric exporter wants the full URL including the signal path."""
trimmed: Final = endpoint.rstrip("/")
return trimmed if trimmed.endswith(_METRICS_PATH) else f"{trimmed}{_METRICS_PATH}"
def _resource_attributes(config: BillingMetricsConfig) -> dict[str, AttributeValue]:
def resource_attributes(config: BillingMetricsConfig) -> dict[str, AttributeValue]:
base: Final[dict[str, AttributeValue]] = {
"service.name": "litellm-proxy",
"litellm.version": config.litellm_version,
@ -82,7 +85,7 @@ def _resource_attributes(config: BillingMetricsConfig) -> dict[str, AttributeVal
return {**base, **license_attr}
def _billable_attributes(
def billable_attributes(
category: BillableCategory, route: str, status_code: int, model_id: str | None
) -> dict[str, AttributeValue]:
base: Final[dict[str, AttributeValue]] = {
@ -94,23 +97,95 @@ def _billable_attributes(
return {**base, **model_attr}
def build_mtls_meter_provider(config: BillingMetricsConfig) -> MeterProvider:
"""OTLP/HTTP exporter presenting a TLS client certificate.
class BillingMetricsBearerAuth(AuthBase):
def __init__(
self,
*,
client_id: str,
client_secret: str,
token_url: str,
clock: Callable[[], float] = time.monotonic,
) -> None:
self._client_id: Final = client_id
self._client_secret: Final = client_secret
self._token_url: Final = token_url
self._clock: Final = clock
self._lock: Final = threading.Lock()
self._cached_token: str | None = None
self._refresh_at: float = 0.0
The collector's load balancer terminates mutual TLS and validates the client
certificate against our CA. Server verification uses the system trust store
(the collector presents a public web-PKI certificate); ca_cert_path overrides
it only for private/test collectors.
"""
exporter: Final = OTLPMetricExporter(
endpoint=_metrics_endpoint(config.endpoint),
# None -> exporter falls back to the system trust store.
certificate_file=config.ca_cert_path,
client_certificate_file=config.client_cert_path,
client_key_file=config.client_key_path,
def __call__(self, request: requests.PreparedRequest) -> requests.PreparedRequest:
token: Final = self._get_token()
authenticated_request: Final = request.copy()
authenticated_request.headers["Authorization"] = f"Bearer {token}"
return authenticated_request
def invalidate(self) -> None:
with self._lock:
self._cached_token = None
self._refresh_at = 0.0
def _get_token(self) -> str:
with self._lock:
now: Final = self._clock()
if self._cached_token is not None and now < self._refresh_at:
return self._cached_token
token, cache_seconds = self._fetch_token()
self._cached_token = token
self._refresh_at = now + cache_seconds
return token
def _fetch_token(self) -> tuple[str, float]:
try:
# The OTel periodic exporter's background thread makes blocking token I/O safe here.
response: Final = requests.post(
self._token_url,
auth=(self._client_id, self._client_secret),
data={"grant_type": "client_credentials", "scope": "telemetry/write"},
timeout=10,
)
response.raise_for_status()
token_response: Final = _BillingMetricsOAuthToken.model_validate(response.json())
return token_response.access_token, max(0.0, float(token_response.expires_in) - _TOKEN_REFRESH_SKEW_SECONDS)
except (requests.RequestException, ValueError):
raise requests.RequestException("Billing metrics token request failed") from None
class _SessionHookKwargs(TypedDict, total=False):
stream: ReadOnly[object]
timeout: ReadOnly[object]
verify: ReadOnly[object]
cert: ReadOnly[object]
proxies: ReadOnly[object]
class _BillingMetricsBearerSession(requests.Session):
def __init__(self, auth: BillingMetricsBearerAuth) -> None:
super().__init__()
self._bearer_auth: Final = auth
self.auth = auth
self.hooks = {"response": [self._invalidate_token_on_unauthorized]}
def _invalidate_token_on_unauthorized(
self,
response: requests.Response,
**kwargs: Unpack[_SessionHookKwargs], # kwargs-ok: requests hooks forward send options
) -> requests.Response:
if response.status_code == 401:
self._bearer_auth.invalidate()
return response
def build_bearer_meter_provider(config: BillingMetricsConfig) -> MeterProvider:
auth: Final = BillingMetricsBearerAuth(
client_id=config.client_id,
client_secret=config.client_secret,
token_url=config.token_url,
)
session: Final = _BillingMetricsBearerSession(auth)
exporter: Final = OTLPMetricExporter(endpoint=_metrics_endpoint(config.endpoint), session=session)
reader: Final = PeriodicExportingMetricReader(exporter, export_interval_millis=config.export_interval_ms)
return MeterProvider(metric_readers=[reader], resource=Resource.create(_resource_attributes(config)))
return MeterProvider(metric_readers=[reader], resource=Resource.create(resource_attributes(config)))
class BillingMetricsRecorder:
@ -125,7 +200,7 @@ class BillingMetricsRecorder:
)
def record(self, *, category: BillableCategory, route: str, status_code: int, model_id: str | None) -> None:
self._counter.add(1, _billable_attributes(category, route, status_code, model_id))
self._counter.add(1, billable_attributes(category, route, status_code, model_id))
def shutdown(self) -> None:
"""Final flush + exporter-thread stop. Without this, up to one export
@ -146,115 +221,34 @@ def _export_interval_ms() -> int:
return DEFAULT_EXPORT_INTERVAL_MS
@dataclass(frozen=True, slots=True)
class _CredentialPaths:
client_cert_path: str
client_key_path: str
ca_cert_path: str | None
def _is_pem_content(value: str) -> bool:
return value.lstrip().startswith(_PEM_PREFIX)
def _write_pem(directory: str, filename: str, pem: str) -> str:
path: Final = os.path.join(directory, filename)
with open(path, "w", encoding="utf-8") as handle:
handle.write(pem if pem.endswith("\n") else f"{pem}\n")
os.chmod(path, _PEM_FILE_MODE)
return path
def _resolve_credential_paths(*, client_cert: str, client_key: str, ca_cert: str | None) -> _CredentialPaths:
"""
Accept either a filesystem path or inline PEM content for each credential.
Secret stores that inject values as environment content rather than mounted
files (ECS tasks reading AWS Secrets Manager, Cloud Run reading Secret
Manager) can only deliver the certificate as a string. The OTLP exporter
takes paths, so inline PEM is written to a private directory once, when the
recorder is built. Raises OSError if that write fails; the caller disables
metering rather than propagating.
"""
inline: Final = tuple(value for value in (client_cert, client_key, ca_cert) if value and _is_pem_content(value))
if not inline:
return _CredentialPaths(client_cert, client_key, ca_cert)
# mkdtemp is 0o700, so the 0o600 key file it holds is unreachable by other users.
directory: Final = tempfile.mkdtemp(prefix=_PEM_DIR_PREFIX)
return _CredentialPaths(
client_cert_path=(
_write_pem(directory, _CLIENT_CERT_FILENAME, client_cert) if _is_pem_content(client_cert) else client_cert
),
client_key_path=(
_write_pem(directory, _CLIENT_KEY_FILENAME, client_key) if _is_pem_content(client_key) else client_key
),
ca_cert_path=(
_write_pem(directory, _CA_CERT_FILENAME, ca_cert) if ca_cert and _is_pem_content(ca_cert) else ca_cert
),
)
def load_billing_metrics_config(
*, license_data: Optional["EnterpriseLicenseData"], litellm_version: str
*, license_data: "EnterpriseLicenseData | None", litellm_version: str
) -> BillingMetricsConfig | None:
endpoint: Final = os.getenv(ENDPOINT_ENV)
client_cert: Final = os.getenv(CLIENT_CERT_ENV)
client_key: Final = os.getenv(CLIENT_KEY_ENV)
# Optional: only for private/test collectors whose server cert is not on the
# public web PKI. The production collector needs no CA override.
ca_cert: Final = os.getenv(CA_CERT_ENV)
missing: Final = [
name
for name, value in (
(ENDPOINT_ENV, endpoint),
(CLIENT_CERT_ENV, client_cert),
(CLIENT_KEY_ENV, client_key),
)
if not value
]
if not endpoint or not client_cert or not client_key:
env: Final = MappingProxyType(
{name: os.getenv(name) or "" for name in (ENDPOINT_ENV, CLIENT_ID_ENV, CLIENT_SECRET_ENV, TOKEN_URL_ENV)}
)
missing: Final = tuple(name for name, value in env.items() if not value)
if missing:
verbose_proxy_logger.warning(
"Enterprise billing metrics disabled: licensed deployment missing config (%s)",
", ".join(missing),
)
return None
try:
paths: Final = _resolve_credential_paths(client_cert=client_cert, client_key=client_key, ca_cert=ca_cert)
except OSError as exc:
verbose_proxy_logger.warning(
"Enterprise billing metrics disabled: could not write inline certificate content to disk: %s", exc
)
return None
# Report the variable names, never their values. A value that is neither a
# readable path nor recognizable PEM is still secret material, and this
# warning would otherwise copy a client key straight into the proxy logs.
unreadable: Final = [
insecure_urls: Final = tuple(
env_name
for env_name, path in (
(CLIENT_CERT_ENV, paths.client_cert_path),
(CLIENT_KEY_ENV, paths.client_key_path),
(CA_CERT_ENV, paths.ca_cert_path),
)
if path and not os.path.isfile(path)
]
if unreadable:
verbose_proxy_logger.warning(
"Enterprise billing metrics disabled: %s did not resolve to a readable certificate file. "
"Set each to a file path, or to inline PEM content beginning with '%s'.",
", ".join(unreadable),
_PEM_PREFIX,
)
for env_name, url in ((ENDPOINT_ENV, env[ENDPOINT_ENV]), (TOKEN_URL_ENV, env[TOKEN_URL_ENV]))
if not is_https_url_with_host(url)
)
if insecure_urls:
verbose_proxy_logger.warning("Enterprise billing metrics disabled: %s must use HTTPS", insecure_urls[0])
return None
return BillingMetricsConfig(
endpoint=endpoint,
client_cert_path=paths.client_cert_path,
client_key_path=paths.client_key_path,
ca_cert_path=paths.ca_cert_path,
endpoint=env[ENDPOINT_ENV],
client_id=env[CLIENT_ID_ENV],
client_secret=env[CLIENT_SECRET_ENV],
token_url=env[TOKEN_URL_ENV],
export_interval_ms=_export_interval_ms(),
litellm_version=litellm_version,
license_id=(license_data or {}).get("user_id"),
@ -282,7 +276,7 @@ _ACTIVE_RECORDER: Final = _ActiveRecorderRegistry()
def build_billing_metrics_recorder(
*, premium: bool, license_data: Optional["EnterpriseLicenseData"], litellm_version: str
*, premium: bool, license_data: "EnterpriseLicenseData | None", litellm_version: str
) -> BillingMetricsRecorder | None:
"""Build the recorder, or None when the deployment is not licensed or metering is unconfigured."""
if not premium:
@ -296,14 +290,12 @@ def build_billing_metrics_recorder(
return None
try:
recorder: Final = BillingMetricsRecorder(build_mtls_meter_provider(config))
provider: Final = build_bearer_meter_provider(config)
recorder: Final = BillingMetricsRecorder(provider)
except Exception as exc: # noqa: BLE001 -- metering must never break proxy startup
verbose_proxy_logger.warning("Enterprise billing metrics disabled: failed to initialize exporter: %s", exc)
return None
_ACTIVE_RECORDER.set(recorder)
# The only positive signal that this component meters. Without it, a silent
# return above is indistinguishable from a working exporter in the logs, and
# a component that carries the cert but no license would look healthy.
verbose_proxy_logger.info(
"Enterprise billing metrics enabled: exporting to %s every %d ms",
config.endpoint,

28
terraform/litellm/aws/README.md
View file

@ -212,33 +212,23 @@ orthogonal to the OTLP variables above; their credentials still go in
### Enterprise billing metrics
License-gated request metering is opt-in and gated entirely on
`billing_metrics_endpoint`. Empty (default) and no billing env is added to
the container, so existing deployments are unchanged. Set it and both
gateway and backend export billable-request counts over OTLP/HTTP,
authenticating to the collector with the mTLS client certificate issued for
your deployment.
The proxy accepts the certificate, key, and CA bundle as either a file path
or literal PEM content. This stack takes the PEM, writes each one to its own
Secrets Manager entry, grants the task-execution role
`secretsmanager:GetSecretValue` on them, and injects them as
`LITELLM_BILLING_METRICS_CLIENT_CERT` / `_CLIENT_KEY` (and `_CA_CERT` when
set), so no volume mount is needed on Fargate.
License-gated request metering uses Cognito bearer authentication. Set the
endpoint, client ID, and token URL as plain values, and provide the client
secret through the sensitive Terraform variable. The client secret is stored
in Secrets Manager and injected into both gateway and backend tasks
```hcl
billing_metrics_endpoint = "https://telemetry.litellm.ai/v1/metrics"
billing_metrics_client_id = "cognito-client-id"
billing_metrics_token_url = "https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token"
```
```bash
export TF_VAR_billing_metrics_client_cert_pem="$(cat client.crt)"
export TF_VAR_billing_metrics_client_key_pem="$(cat client.key)"
export TF_VAR_billing_metrics_client_secret="cognito-client-secret"
```
`billing_metrics_ca_cert_pem` is only for private or test collectors whose
CA is not in the system trust store; leave it empty against
`telemetry.litellm.ai`. Metering requires an enterprise license, so pair
this with `litellm_license`. To tune the export cadence, set
The client ID and client secret are required when the endpoint is set
Metering also requires an enterprise license. To tune the export cadence, set
`LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS` through `gateway_extra_env` /
`backend_extra_env`

60
terraform/litellm/aws/ecs.tf
View file

@ -79,32 +79,19 @@ locals {
{ name = "OTEL_HEADERS", valueFrom = var.otel_headers_secret_arn },
] : []
# Enterprise request metering, gated on billing_metrics_endpoint. The
# endpoint rides in as a plain env var; the mTLS material is stored in
# Secrets Manager (secrets.tf) and injected as PEM-valued env vars, which
# the proxy accepts in place of file paths. Each PEM is wired only when the
# operator supplied it, so an empty ca_cert_pem falls back to the system
# trust store.
billing_metrics_enabled = var.billing_metrics_endpoint != ""
billing_metrics_client_cert_enabled = local.billing_metrics_enabled && var.billing_metrics_client_cert_pem != ""
billing_metrics_client_key_enabled = local.billing_metrics_enabled && var.billing_metrics_client_key_pem != ""
billing_metrics_ca_cert_enabled = local.billing_metrics_enabled && var.billing_metrics_ca_cert_pem != ""
# Enterprise request metering, gated on billing_metrics_endpoint.
billing_metrics_enabled = var.billing_metrics_endpoint != ""
billing_metrics_client_secret_enabled = local.billing_metrics_enabled && var.billing_metrics_client_secret != ""
billing_metrics_env = local.billing_metrics_enabled ? [
{ name = "LITELLM_BILLING_METRICS_ENDPOINT", value = var.billing_metrics_endpoint },
{ name = "LITELLM_BILLING_METRICS_CLIENT_ID", value = var.billing_metrics_client_id },
{ name = "LITELLM_BILLING_METRICS_TOKEN_URL", value = var.billing_metrics_token_url },
] : []
billing_metrics_secrets = concat(
local.billing_metrics_client_cert_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CLIENT_CERT", valueFrom = aws_secretsmanager_secret.billing_metrics_client_cert[0].arn },
] : [],
local.billing_metrics_client_key_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CLIENT_KEY", valueFrom = aws_secretsmanager_secret.billing_metrics_client_key[0].arn },
] : [],
local.billing_metrics_ca_cert_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CA_CERT", valueFrom = aws_secretsmanager_secret.billing_metrics_ca_cert[0].arn },
] : [],
)
billing_metrics_secrets = local.billing_metrics_client_secret_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CLIENT_SECRET", valueFrom = aws_secretsmanager_secret.billing_metrics_client_secret[0].arn },
] : []
managed_db_env = var.create_database ? [
{ name = "IAM_TOKEN_DB_AUTH", value = "true" },
@ -338,27 +325,12 @@ locals {
# ---------- Gateway ----------
resource "aws_ecs_task_definition" "gateway" {
# Metering needs a client certificate AND its key. Each secret is created only
# when its own PEM is supplied, so an endpoint set with a missing key would
# otherwise apply cleanly and leave the proxy logging "missing config" and
# never exporting. ca_cert_pem stays optional: empty means fall back to the
# system trust store.
#
# The guard lives here, on an unconditional resource, rather than on the cert
# secret: that secret is count-gated on the cert itself, so it has zero
# instances in exactly the case this must catch. Adding count or for_each to
# this resource would silently stop the guard from evaluating.
#
# endpoint cert key -> result
# "" any any -> metering off, no secrets created
# set set set -> metering on
# set any-missing -> plan fails here
lifecycle {
precondition {
condition = var.billing_metrics_endpoint == "" || (
var.billing_metrics_client_cert_pem != "" && var.billing_metrics_client_key_pem != ""
var.billing_metrics_client_id != "" && var.billing_metrics_client_secret != ""
)
error_message = "billing_metrics_client_cert_pem and billing_metrics_client_key_pem are both required when billing_metrics_endpoint is set."
error_message = "billing_metrics_client_id and billing_metrics_client_secret are required when billing_metrics_endpoint is set."
}
precondition {
@ -465,9 +437,7 @@ resource "aws_ecs_service" "gateway" {
aws_secretsmanager_secret_version.license,
aws_secretsmanager_secret_version.database_url,
aws_secretsmanager_secret_version.redis_url,
aws_secretsmanager_secret_version.billing_metrics_client_cert,
aws_secretsmanager_secret_version.billing_metrics_client_key,
aws_secretsmanager_secret_version.billing_metrics_ca_cert,
aws_secretsmanager_secret_version.billing_metrics_client_secret,
]
tags = local.tags
@ -481,9 +451,9 @@ resource "aws_ecs_task_definition" "backend" {
lifecycle {
precondition {
condition = var.billing_metrics_endpoint == "" || (
var.billing_metrics_client_cert_pem != "" && var.billing_metrics_client_key_pem != ""
var.billing_metrics_client_id != "" && var.billing_metrics_client_secret != ""
)
error_message = "billing_metrics_client_cert_pem and billing_metrics_client_key_pem are both required when billing_metrics_endpoint is set."
error_message = "billing_metrics_client_id and billing_metrics_client_secret are required when billing_metrics_endpoint is set."
}
}
@ -566,9 +536,7 @@ resource "aws_ecs_service" "backend" {
aws_secretsmanager_secret_version.ui_password,
aws_secretsmanager_secret_version.database_url,
aws_secretsmanager_secret_version.redis_url,
aws_secretsmanager_secret_version.billing_metrics_client_cert,
aws_secretsmanager_secret_version.billing_metrics_client_key,
aws_secretsmanager_secret_version.billing_metrics_ca_cert,
aws_secretsmanager_secret_version.billing_metrics_client_secret,
]
tags = local.tags

4
terraform/litellm/aws/iam.tf
View file

@ -53,9 +53,7 @@ data "aws_iam_policy_document" "secrets_access" {
[aws_secretsmanager_secret.master_key.arn],
aws_secretsmanager_secret.license[*].arn,
aws_secretsmanager_secret.ui_password[*].arn,
aws_secretsmanager_secret.billing_metrics_client_cert[*].arn,
aws_secretsmanager_secret.billing_metrics_client_key[*].arn,
aws_secretsmanager_secret.billing_metrics_ca_cert[*].arn,
aws_secretsmanager_secret.billing_metrics_client_secret[*].arn,
aws_secretsmanager_secret.database_url[*].arn,
aws_secretsmanager_secret.redis_url[*].arn,
local.extra_secret_arns,

54
terraform/litellm/aws/secrets.tf
View file

@ -75,59 +75,21 @@ resource "aws_secretsmanager_secret_version" "ui_password" {
secret_string = var.ui_password
}
# Billing-metrics mTLS material — only created when metering is enabled
# (billing_metrics_endpoint non-empty) and the operator supplied the PEM.
# The task-execution role gets GetSecretValue via iam.tf, and gateway +
# backend pick the env vars up through shared_secrets in ecs.tf.
resource "aws_secretsmanager_secret" "billing_metrics_client_cert" {
count = local.billing_metrics_client_cert_enabled ? 1 : 0
resource "aws_secretsmanager_secret" "billing_metrics_client_secret" {
count = local.billing_metrics_client_secret_enabled ? 1 : 0
name = "${local.name}-billing-metrics-client-cert"
description = "LITELLM_BILLING_METRICS_CLIENT_CERT for gateway + backend."
name = "${local.name}-billing-metrics-client-secret"
description = "LITELLM_BILLING_METRICS_CLIENT_SECRET for gateway + backend."
recovery_window_in_days = 0
tags = local.tags
}
resource "aws_secretsmanager_secret_version" "billing_metrics_client_cert" {
count = local.billing_metrics_client_cert_enabled ? 1 : 0
resource "aws_secretsmanager_secret_version" "billing_metrics_client_secret" {
count = local.billing_metrics_client_secret_enabled ? 1 : 0
secret_id = aws_secretsmanager_secret.billing_metrics_client_cert[0].id
secret_string = var.billing_metrics_client_cert_pem
}
resource "aws_secretsmanager_secret" "billing_metrics_client_key" {
count = local.billing_metrics_client_key_enabled ? 1 : 0
name = "${local.name}-billing-metrics-client-key"
description = "LITELLM_BILLING_METRICS_CLIENT_KEY for gateway + backend."
recovery_window_in_days = 0
tags = local.tags
}
resource "aws_secretsmanager_secret_version" "billing_metrics_client_key" {
count = local.billing_metrics_client_key_enabled ? 1 : 0
secret_id = aws_secretsmanager_secret.billing_metrics_client_key[0].id
secret_string = var.billing_metrics_client_key_pem
}
resource "aws_secretsmanager_secret" "billing_metrics_ca_cert" {
count = local.billing_metrics_ca_cert_enabled ? 1 : 0
name = "${local.name}-billing-metrics-ca-cert"
description = "LITELLM_BILLING_METRICS_CA_CERT for gateway + backend."
recovery_window_in_days = 0
tags = local.tags
}
resource "aws_secretsmanager_secret_version" "billing_metrics_ca_cert" {
count = local.billing_metrics_ca_cert_enabled ? 1 : 0
secret_id = aws_secretsmanager_secret.billing_metrics_ca_cert[0].id
secret_string = var.billing_metrics_ca_cert_pem
secret_id = aws_secretsmanager_secret.billing_metrics_client_secret[0].id
secret_string = var.billing_metrics_client_secret
}
resource "aws_secretsmanager_secret" "db_master_password" {

54
terraform/litellm/aws/variables.tf
View file

@ -764,13 +764,7 @@ variable "otel_headers_secret_arn" {
# ---------- Enterprise billing metrics ----------
#
# License-gated request metering. Opt-in and gated entirely on
# billing_metrics_endpoint: leave it empty (the default) and nothing
# metering-related lands in the container env. Set it and gateway + backend
# export billable-request counts over OTLP/HTTP, authenticating to the
# collector with an mTLS client cert. The proxy accepts the cert, key, and CA
# as either a file path or literal PEM content, so on Fargate they are
# injected straight from Secrets Manager as env vars and no volume is needed.
# License-gated request metering. The client secret is stored in Secrets Manager.
variable "billing_metrics_endpoint" {
description = <<-EOT
@ -784,44 +778,34 @@ variable "billing_metrics_endpoint" {
default = ""
}
variable "billing_metrics_client_cert_pem" {
variable "billing_metrics_client_id" {
description = <<-EOT
PEM content of the mTLS client certificate issued for this deployment.
When billing_metrics_endpoint is set, the stack stores this in a
`<tenant>-litellm-<env>-billing-metrics-client-cert` Secrets Manager
entry, grants the task-execution role GetSecretValue on it, and exposes
it to gateway + backend as LITELLM_BILLING_METRICS_CLIENT_CERT. Required
whenever metering is enabled.
Cognito OAuth2 client ID exposed to the gateway and backend as
LITELLM_BILLING_METRICS_CLIENT_ID. Required when billing_metrics_endpoint
is set.
EOT
type = string
default = ""
}
variable "billing_metrics_client_secret" {
description = <<-EOT
Cognito OAuth2 client secret stored in Secrets Manager and exposed to
the gateway and backend as LITELLM_BILLING_METRICS_CLIENT_SECRET.
Required when billing_metrics_endpoint is set.
EOT
type = string
default = ""
sensitive = true
}
variable "billing_metrics_client_key_pem" {
variable "billing_metrics_token_url" {
description = <<-EOT
PEM content of the private key matching
billing_metrics_client_cert_pem. Stored in a
`<tenant>-litellm-<env>-billing-metrics-client-key` Secrets Manager
entry and exposed as LITELLM_BILLING_METRICS_CLIENT_KEY. Required
whenever metering is enabled.
Cognito OAuth2 client-credentials token endpoint exposed to gateway and
backend as LITELLM_BILLING_METRICS_TOKEN_URL.
EOT
type = string
default = ""
sensitive = true
}
variable "billing_metrics_ca_cert_pem" {
description = <<-EOT
PEM content of the CA bundle used to verify the metering collector.
Only needed for private or test collectors whose CA is not in the
system trust store; telemetry.litellm.ai is publicly trusted, so leave
this empty for production. When set, it is exposed as
LITELLM_BILLING_METRICS_CA_CERT.
EOT
type = string
default = ""
sensitive = true
default = "https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token"
}
# ---------- Collector sidecar ----------

28
terraform/litellm/gcp/README.md
View file

@ -210,33 +210,23 @@ Behavior matches the AWS stack 1:1; the only naming differences are
### Enterprise billing metrics
License-gated request metering is opt-in and gated entirely on
`billing_metrics_endpoint`. Empty (default) and no billing env is added to
the container, so existing deployments are unchanged. Set it and both
gateway and backend export billable-request counts over OTLP/HTTP,
authenticating to the collector with the mTLS client certificate issued for
your deployment.
The proxy accepts the certificate, key, and CA bundle as either a file path
or literal PEM content. This stack takes the PEM, writes each one to its own
Secret Manager entry, grants the runtime service account
`roles/secretmanager.secretAccessor` on them, and injects them as Cloud Run
secret env vars `LITELLM_BILLING_METRICS_CLIENT_CERT` / `_CLIENT_KEY` (and
`_CA_CERT` when set), so no volume mount is needed.
License-gated request metering uses Cognito bearer authentication. Set the
endpoint, client ID, and token URL as plain values, and provide the client
secret through the sensitive Terraform variable. The client secret is stored
in Secret Manager and injected into both gateway and backend services
```hcl
billing_metrics_endpoint = "https://telemetry.litellm.ai/v1/metrics"
billing_metrics_client_id = "cognito-client-id"
billing_metrics_token_url = "https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token"
```
```bash
export TF_VAR_billing_metrics_client_cert_pem="$(cat client.crt)"
export TF_VAR_billing_metrics_client_key_pem="$(cat client.key)"
export TF_VAR_billing_metrics_client_secret="cognito-client-secret"
```
`billing_metrics_ca_cert_pem` is only for private or test collectors whose
CA is not in the system trust store; leave it empty against
`telemetry.litellm.ai`. Metering requires an enterprise license, so pair
this with `litellm_license`. To tune the export cadence, set
The client ID and client secret are required when the endpoint is set
Metering also requires an enterprise license. To tune the export cadence, set
`LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS` through `gateway_extra_env` /
`backend_extra_env`

37
terraform/litellm/gcp/cloudrun.tf
View file

@ -62,32 +62,19 @@ locals {
{ name = "OTEL_HEADERS", secret = var.otel_headers_secret, version = "latest" },
] : []
# Enterprise request metering, gated on billing_metrics_endpoint. The
# endpoint rides in as a plain env var; the mTLS material lives in Secret
# Manager (secrets.tf) and is injected as PEM-valued env vars, which the
# proxy accepts in place of file paths. Each PEM is wired only when the
# operator supplied it, so an empty ca_cert_pem falls back to the system
# trust store.
billing_metrics_enabled = var.billing_metrics_endpoint != ""
billing_metrics_client_cert_enabled = local.billing_metrics_enabled && var.billing_metrics_client_cert_pem != ""
billing_metrics_client_key_enabled = local.billing_metrics_enabled && var.billing_metrics_client_key_pem != ""
billing_metrics_ca_cert_enabled = local.billing_metrics_enabled && var.billing_metrics_ca_cert_pem != ""
# Enterprise request metering, gated on billing_metrics_endpoint.
billing_metrics_enabled = var.billing_metrics_endpoint != ""
billing_metrics_client_secret_enabled = local.billing_metrics_enabled && var.billing_metrics_client_secret != ""
billing_metrics_env_kv = local.billing_metrics_enabled ? [
{ name = "LITELLM_BILLING_METRICS_ENDPOINT", value = var.billing_metrics_endpoint },
{ name = "LITELLM_BILLING_METRICS_CLIENT_ID", value = var.billing_metrics_client_id },
{ name = "LITELLM_BILLING_METRICS_TOKEN_URL", value = var.billing_metrics_token_url },
] : []
billing_metrics_env_secrets = concat(
local.billing_metrics_client_cert_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CLIENT_CERT", secret = google_secret_manager_secret.billing_metrics_client_cert[0].id, version = "latest" },
] : [],
local.billing_metrics_client_key_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CLIENT_KEY", secret = google_secret_manager_secret.billing_metrics_client_key[0].id, version = "latest" },
] : [],
local.billing_metrics_ca_cert_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CA_CERT", secret = google_secret_manager_secret.billing_metrics_ca_cert[0].id, version = "latest" },
] : [],
)
billing_metrics_env_secrets = local.billing_metrics_client_secret_enabled ? [
{ name = "LITELLM_BILLING_METRICS_CLIENT_SECRET", secret = google_secret_manager_secret.billing_metrics_client_secret[0].id, version = "latest" },
] : []
# Cloud Run v2 secret env vars use value_source.secret_key_ref pointing at a
# secret resource ID. Shared between gateway and backend (the migrations
@ -481,9 +468,7 @@ resource "google_cloud_run_v2_service" "gateway" {
google_secret_manager_secret_iam_member.license,
google_secret_manager_secret_iam_member.extras,
google_secret_manager_secret_iam_member.otel_headers,
google_secret_manager_secret_iam_member.billing_metrics_client_cert,
google_secret_manager_secret_iam_member.billing_metrics_client_key,
google_secret_manager_secret_iam_member.billing_metrics_ca_cert,
google_secret_manager_secret_iam_member.billing_metrics_client_secret,
google_secret_manager_secret_iam_member.metrics_run_monitoring,
google_project_iam_member.runtime_metric_writer,
google_storage_bucket_iam_member.proxy_config_runtime,
@ -603,9 +588,7 @@ resource "google_cloud_run_v2_service" "backend" {
google_secret_manager_secret_iam_member.ui_password,
google_secret_manager_secret_iam_member.extras,
google_secret_manager_secret_iam_member.otel_headers,
google_secret_manager_secret_iam_member.billing_metrics_client_cert,
google_secret_manager_secret_iam_member.billing_metrics_client_key,
google_secret_manager_secret_iam_member.billing_metrics_ca_cert,
google_secret_manager_secret_iam_member.billing_metrics_client_secret,
google_storage_bucket_iam_member.proxy_config_runtime,
google_sql_user.app,
terraform_data.migration,

28
terraform/litellm/gcp/iam.tf
View file

@ -10,9 +10,9 @@ resource "google_service_account" "runtime" {
lifecycle {
precondition {
condition = !var.create_runtime || var.billing_metrics_endpoint == "" || (
var.billing_metrics_client_cert_pem != "" && var.billing_metrics_client_key_pem != ""
var.billing_metrics_client_id != "" && var.billing_metrics_client_secret != ""
)
error_message = "billing_metrics_client_cert_pem and billing_metrics_client_key_pem are both required when billing_metrics_endpoint is set and create_runtime is true."
error_message = "billing_metrics_client_id and billing_metrics_client_secret are required when billing_metrics_endpoint is set and create_runtime is true."
}
}
}
@ -91,28 +91,10 @@ resource "google_secret_manager_secret_iam_member" "otel_headers" {
member = "serviceAccount:${google_service_account.runtime.email}"
}
# Billing-metrics mTLS accessors — only created when request metering is
# enabled and the matching PEM was supplied.
resource "google_secret_manager_secret_iam_member" "billing_metrics_client_cert" {
count = local.billing_metrics_client_cert_enabled ? 1 : 0
resource "google_secret_manager_secret_iam_member" "billing_metrics_client_secret" {
count = local.billing_metrics_client_secret_enabled ? 1 : 0
secret_id = google_secret_manager_secret.billing_metrics_client_cert[0].id
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:${google_service_account.runtime.email}"
}
resource "google_secret_manager_secret_iam_member" "billing_metrics_client_key" {
count = local.billing_metrics_client_key_enabled ? 1 : 0
secret_id = google_secret_manager_secret.billing_metrics_client_key[0].id
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:${google_service_account.runtime.email}"
}
resource "google_secret_manager_secret_iam_member" "billing_metrics_ca_cert" {
count = local.billing_metrics_ca_cert_enabled ? 1 : 0
secret_id = google_secret_manager_secret.billing_metrics_ca_cert[0].id
secret_id = google_secret_manager_secret.billing_metrics_client_secret[0].id
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:${google_service_account.runtime.email}"
}

52
terraform/litellm/gcp/secrets.tf
View file

@ -64,59 +64,21 @@ resource "google_secret_manager_secret_version" "ui_password" {
secret_data = var.ui_password
}
# Billing-metrics mTLS material — only created when metering is enabled
# (billing_metrics_endpoint non-empty) and the operator supplied the PEM.
# The runtime SA gets accessor permission via iam.tf, and gateway + backend
# pick the env vars up through billing_metrics_env_secrets in cloudrun.tf.
resource "google_secret_manager_secret" "billing_metrics_client_cert" {
count = local.billing_metrics_client_cert_enabled ? 1 : 0
resource "google_secret_manager_secret" "billing_metrics_client_secret" {
count = local.billing_metrics_client_secret_enabled ? 1 : 0
secret_id = "${local.name}-billing-metrics-client-cert"
secret_id = "${local.name}-billing-metrics-client-secret"
labels = local.labels
replication {
auto {}
}
}
resource "google_secret_manager_secret_version" "billing_metrics_client_cert" {
count = local.billing_metrics_client_cert_enabled ? 1 : 0
resource "google_secret_manager_secret_version" "billing_metrics_client_secret" {
count = local.billing_metrics_client_secret_enabled ? 1 : 0
secret = google_secret_manager_secret.billing_metrics_client_cert[0].id
secret_data = var.billing_metrics_client_cert_pem
}
resource "google_secret_manager_secret" "billing_metrics_client_key" {
count = local.billing_metrics_client_key_enabled ? 1 : 0
secret_id = "${local.name}-billing-metrics-client-key"
labels = local.labels
replication {
auto {}
}
}
resource "google_secret_manager_secret_version" "billing_metrics_client_key" {
count = local.billing_metrics_client_key_enabled ? 1 : 0
secret = google_secret_manager_secret.billing_metrics_client_key[0].id
secret_data = var.billing_metrics_client_key_pem
}
resource "google_secret_manager_secret" "billing_metrics_ca_cert" {
count = local.billing_metrics_ca_cert_enabled ? 1 : 0
secret_id = "${local.name}-billing-metrics-ca-cert"
labels = local.labels
replication {
auto {}
}
}
resource "google_secret_manager_secret_version" "billing_metrics_ca_cert" {
count = local.billing_metrics_ca_cert_enabled ? 1 : 0
secret = google_secret_manager_secret.billing_metrics_ca_cert[0].id
secret_data = var.billing_metrics_ca_cert_pem
secret = google_secret_manager_secret.billing_metrics_client_secret[0].id
secret_data = var.billing_metrics_client_secret
}
resource "google_secret_manager_secret" "metrics_run_monitoring" {

55
terraform/litellm/gcp/variables.tf
View file

@ -602,14 +602,7 @@ variable "gateway_metrics_collector_image" {
# ---------- Enterprise billing metrics ----------
#
# License-gated request metering. Opt-in and gated entirely on
# billing_metrics_endpoint: leave it empty (the default) and nothing
# metering-related is added to the container env. Set it and gateway +
# backend export billable-request counts over OTLP/HTTP, authenticating to
# the collector with an mTLS client cert. The proxy accepts the cert, key,
# and CA as either a file path or literal PEM content, so on Cloud Run they
# are injected straight from Secret Manager as env vars and no volume is
# needed.
# License-gated request metering. The client secret is stored in Secret Manager.
variable "billing_metrics_endpoint" {
description = <<-EOT
@ -623,44 +616,34 @@ variable "billing_metrics_endpoint" {
default = ""
}
variable "billing_metrics_client_cert_pem" {
variable "billing_metrics_client_id" {
description = <<-EOT
PEM content of the mTLS client certificate issued for this deployment.
When billing_metrics_endpoint is set, the stack stores this in a
`<tenant>-litellm-<env>-billing-metrics-client-cert` Secret Manager
entry, grants the runtime SA accessor on it, and exposes it to gateway +
backend as LITELLM_BILLING_METRICS_CLIENT_CERT. Required whenever
metering is enabled.
Cognito OAuth2 client ID exposed to the gateway and backend as
LITELLM_BILLING_METRICS_CLIENT_ID. Required when billing_metrics_endpoint
is set.
EOT
type = string
default = ""
}
variable "billing_metrics_client_secret" {
description = <<-EOT
Cognito OAuth2 client secret stored in Secret Manager and exposed to
the gateway and backend as LITELLM_BILLING_METRICS_CLIENT_SECRET.
Required when billing_metrics_endpoint is set.
EOT
type = string
default = ""
sensitive = true
}
variable "billing_metrics_client_key_pem" {
variable "billing_metrics_token_url" {
description = <<-EOT
PEM content of the private key matching
billing_metrics_client_cert_pem. Stored in a
`<tenant>-litellm-<env>-billing-metrics-client-key` Secret Manager entry
and exposed as LITELLM_BILLING_METRICS_CLIENT_KEY. Required whenever
metering is enabled.
Cognito OAuth2 client-credentials token endpoint exposed to gateway and
backend as LITELLM_BILLING_METRICS_TOKEN_URL.
EOT
type = string
default = ""
sensitive = true
}
variable "billing_metrics_ca_cert_pem" {
description = <<-EOT
PEM content of the CA bundle used to verify the metering collector.
Only needed for private or test collectors whose CA is not in the
system trust store; telemetry.litellm.ai is publicly trusted, so leave
this empty for production. When set, it is exposed as
LITELLM_BILLING_METRICS_CA_CERT.
EOT
type = string
default = ""
sensitive = true
default = "https://litellm-prod.auth.us-west-1.amazoncognito.com/oauth2/token"
}
# ---------- Collector sidecar ----------

14
tests/unit/litellm_core_utils/test_url_utils.py
View file

@ -16,11 +16,25 @@ from litellm.litellm_core_utils.url_utils import (
encode_url_path_segment,
encode_url_path_segments,
ensure_api_route,
is_https_url_with_host,
join_api_base,
validate_url,
)
@pytest.mark.parametrize(
("url", "expected"),
(
("https://example.com/path", True),
("http://example.com/path", False),
("https:///path", False),
("https://[", False),
),
)
def test_is_https_url_with_host(url: str, expected: bool) -> None:
assert is_https_url_with_host(url) is expected
@pytest.fixture
def mock_dns_public(monkeypatch):
"""Resolve any hostname to 93.184.216.34 (public)."""

707
tests/unit/proxy/enterprise_billing/test_billing_metrics.py
View file

@ -1,398 +1,379 @@
"""
Tests for the enterprise billing-metrics recorder and its factory.
These verify the license gate, the missing-config and missing-cert disable
paths, the OTLP/HTTP exporter wiring (client cert+key authenticate us to the
collector's mTLS-terminating front end; CA override optional for private
collectors), the metric attribute mapping, and that recording produces the
expected OTLP counter via an in-memory reader.
"""
import os
import socket
import stat
from pathlib import Path
from typing import Dict, List, Optional
from collections.abc import Generator, Mapping
from typing import Final
import pytest
import requests
import requests_mock
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from opentelemetry.sdk.metrics.export import InMemoryMetricReader, Sum
from pydantic import BaseModel, ConfigDict, TypeAdapter
from litellm.proxy._types import EnterpriseLicenseData
from litellm.proxy.enterprise_billing import billing_metrics as bm
from litellm.proxy.middleware.billable_request_metrics_middleware import BillableCategory
_ENV_VARS = (
class _RecordedRequest(BaseModel):
model_config = ConfigDict(from_attributes=True, frozen=True)
url: str | None
body: str | bytes | None
headers: dict[str, str]
def _request_history(http_mock: requests_mock.Mocker) -> tuple[_RecordedRequest, ...]:
return TypeAdapter(tuple[_RecordedRequest, ...]).validate_python(http_mock.request_history)
def _metric_category(attributes: Mapping[str, object] | None) -> str:
if attributes is None:
return ""
return str(attributes.get("litellm.endpoint.category"))
_ENV_VARS: Final = (
bm.ENDPOINT_ENV,
bm.CLIENT_CERT_ENV,
bm.CLIENT_KEY_ENV,
bm.CA_CERT_ENV,
bm.CLIENT_ID_ENV,
bm.CLIENT_SECRET_ENV,
bm.TOKEN_URL_ENV,
bm.EXPORT_INTERVAL_ENV,
)
@pytest.fixture(autouse=True)
def clear_env(monkeypatch):
def clear_env(monkeypatch: pytest.MonkeyPatch) -> Generator[None, None, None]:
for name in _ENV_VARS:
monkeypatch.delenv(name, raising=False)
yield
bm.shutdown_billing_metrics_recorder()
def _write_certs(tmp_path: Path) -> Dict[str, str]:
files = {
bm.CA_CERT_ENV: ("ca.pem", b"ca-bytes"),
bm.CLIENT_CERT_ENV: ("client.pem", b"client-cert-bytes"),
bm.CLIENT_KEY_ENV: ("client.key", b"client-key-bytes"),
}
paths = {}
for env_name, (filename, content) in files.items():
path = tmp_path / filename
path.write_bytes(content)
paths[env_name] = str(path)
return paths
def _set_full_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4318")
monkeypatch.setenv(bm.CLIENT_ID_ENV, "client-id")
monkeypatch.setenv(bm.CLIENT_SECRET_ENV, "client-secret")
monkeypatch.setenv(bm.TOKEN_URL_ENV, "https://auth.example/oauth2/token")
def _set_full_env(monkeypatch, tmp_path: Path) -> Dict[str, str]:
paths = _write_certs(tmp_path)
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CA_CERT_ENV, paths[bm.CA_CERT_ENV])
monkeypatch.setenv(bm.CLIENT_CERT_ENV, paths[bm.CLIENT_CERT_ENV])
monkeypatch.setenv(bm.CLIENT_KEY_ENV, paths[bm.CLIENT_KEY_ENV])
return paths
def _config(tmp_path: Path, license_id: Optional[str] = "org-1") -> bm.BillingMetricsConfig:
paths = _write_certs(tmp_path)
def _config(license_id: str | None = "org-1") -> bm.BillingMetricsConfig:
return bm.BillingMetricsConfig(
endpoint="https://collector.example:4317",
client_cert_path=paths[bm.CLIENT_CERT_ENV],
client_key_path=paths[bm.CLIENT_KEY_ENV],
ca_cert_path=paths[bm.CA_CERT_ENV],
endpoint="https://collector.example:4318",
token_url="https://auth.example/oauth2/token",
client_id="client-id",
client_secret="client-secret",
export_interval_ms=60_000,
litellm_version="1.2.3",
license_id=license_id,
)
# ── Factory gating ────────────────────────────────────────────────────────────
@pytest.mark.parametrize("missing_env", (bm.ENDPOINT_ENV, bm.CLIENT_ID_ENV, bm.CLIENT_SECRET_ENV, bm.TOKEN_URL_ENV))
def test_missing_required_env_disables_metrics_without_logging_values(
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
missing_env: str,
) -> None:
_set_full_env(monkeypatch)
monkeypatch.setenv(bm.CLIENT_SECRET_ENV, "secret-for-test")
monkeypatch.delenv(missing_env)
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
assert f"licensed deployment missing config ({missing_env})" in caplog.text
assert "secret-for-test" not in caplog.text
def test_not_premium_returns_none(tmp_path, monkeypatch):
_set_full_env(monkeypatch, tmp_path)
def test_empty_required_env_disables_metrics(monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture) -> None:
_set_full_env(monkeypatch)
monkeypatch.setenv(bm.ENDPOINT_ENV, "")
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
assert bm.ENDPOINT_ENV in caplog.text
@pytest.mark.parametrize(
("env_name", "url"),
(
(bm.ENDPOINT_ENV, "http://collector.example:4318"),
(bm.TOKEN_URL_ENV, "http://auth.example/oauth2/token"),
),
)
def test_http_urls_disable_metrics(
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
env_name: str,
url: str,
) -> None:
_set_full_env(monkeypatch)
monkeypatch.setenv(env_name, url)
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
assert f"{env_name} must use HTTPS" in caplog.text
def test_valid_config_contains_bearer_credentials(monkeypatch: pytest.MonkeyPatch) -> None:
_set_full_env(monkeypatch)
license_data: Final[EnterpriseLicenseData] = {"user_id": "org-1"}
config: Final = bm.load_billing_metrics_config(license_data=license_data, litellm_version="1.0")
assert config is not None
assert config == bm.BillingMetricsConfig(
endpoint="https://collector.example:4318",
token_url="https://auth.example/oauth2/token",
client_id="client-id",
client_secret="client-secret",
export_interval_ms=bm.DEFAULT_EXPORT_INTERVAL_MS,
litellm_version="1.0",
license_id="org-1",
)
assert "client-secret" not in repr(config)
assert "license-secret" not in repr(config)
def test_not_premium_returns_none_without_warning(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
_set_full_env(monkeypatch)
assert bm.build_billing_metrics_recorder(premium=False, license_data=None, litellm_version="1.0") is None
assert "Enterprise billing metrics disabled" not in caplog.text
def test_premium_without_config_returns_none(monkeypatch):
def test_premium_without_config_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
assert bm.build_billing_metrics_recorder(premium=True, license_data={"user_id": "x"}, litellm_version="1.0") is None
def test_premium_with_missing_cert_files_returns_none(monkeypatch, tmp_path):
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CA_CERT_ENV, str(tmp_path / "missing-ca.pem"))
monkeypatch.setenv(bm.CLIENT_CERT_ENV, str(tmp_path / "missing-cert.pem"))
monkeypatch.setenv(bm.CLIENT_KEY_ENV, str(tmp_path / "missing-key.pem"))
assert bm.build_billing_metrics_recorder(premium=True, license_data=None, litellm_version="1.0") is None
def test_premium_builds_recorder_and_logs_enabled(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
_set_full_env(monkeypatch)
caplog.set_level("INFO", logger=bm.verbose_proxy_logger.name)
token_url: Final = "https://auth.example/oauth2/token"
export_url: Final = "https://collector.example:4318/v1/metrics"
http_mock: Final = requests_mock.Mocker()
with http_mock:
http_mock.post(token_url, json={"access_token": "access-token", "expires_in": 3600})
http_mock.post(export_url, status_code=200)
recorder: Final = bm.build_billing_metrics_recorder(
premium=True, license_data={"user_id": "org-1"}, litellm_version="1.0"
)
assert recorder is not None
assert "Enterprise billing metrics enabled: exporting to https://collector.example:4318 every 60000 ms" in caplog.text
bm.shutdown_billing_metrics_recorder()
def test_premium_with_full_config_builds_recorder(monkeypatch, tmp_path):
"""Builds a real MeterProvider, so the exporter is stubbed: the live one
resolves the collector and opens a TLS connection during the shutdown flush.
The getaddrinfo spy keeps that stub from being quietly dropped later."""
_set_full_env(monkeypatch, tmp_path)
monkeypatch.setattr(bm, "OTLPMetricExporter", _fake_exporter_class({}))
class _ShutdownCountingMeterProvider(MeterProvider):
def __init__(self) -> None:
super().__init__()
self.shutdown_calls = 0
resolved: List[str] = []
real_getaddrinfo = socket.getaddrinfo
def shutdown(self, timeout_millis: float = 30_000) -> None:
self.shutdown_calls += 1
super().shutdown(timeout_millis=timeout_millis)
def _spy_getaddrinfo(host, port, *args, **kwargs):
resolved.append(str(host))
return real_getaddrinfo(host, port, *args, **kwargs)
monkeypatch.setattr(socket, "getaddrinfo", _spy_getaddrinfo)
def test_shutdown_flushes_active_recorder_once(monkeypatch: pytest.MonkeyPatch) -> None:
provider: Final = _ShutdownCountingMeterProvider()
_set_full_env(monkeypatch)
recorder = bm.build_billing_metrics_recorder(
premium=True, license_data={"user_id": "org-1"}, litellm_version="1.0"
def create_provider(config: bm.BillingMetricsConfig) -> MeterProvider:
return provider
monkeypatch.setattr(bm, "build_bearer_meter_provider", create_provider)
recorder: Final = bm.build_billing_metrics_recorder(premium=True, license_data=None, litellm_version="1.0")
assert recorder is not None
bm.shutdown_billing_metrics_recorder()
bm.shutdown_billing_metrics_recorder()
assert provider.shutdown_calls == 1
class _FakeClock:
def __init__(self, now: float) -> None:
self._now = now
def __call__(self) -> float:
return self._now
def set(self, now: float) -> None:
self._now = now
def test_bearer_auth_fetches_and_caches_client_credentials_token() -> None:
token_url: Final = "https://auth.example/oauth2/token"
export_url: Final = "https://collector.example/v1/metrics"
auth: Final = bm.BillingMetricsBearerAuth(
client_id="client-id",
client_secret="client-secret",
token_url=token_url,
clock=lambda: 100.0,
)
assert isinstance(recorder, bm.BillingMetricsRecorder)
recorder.record(category=BillableCategory.LLM, route="/chat/completions", status_code=200, model_id=None)
bm.shutdown_billing_metrics_recorder()
session: Final = requests.Session()
session.auth = auth
http_mock: Final = requests_mock.Mocker()
with http_mock:
http_mock.post(token_url, json={"access_token": "access-token", "expires_in": 3600})
http_mock.post(export_url, status_code=200)
session.post(export_url, data=b"metrics")
session.post(export_url, data=b"metrics")
assert [host for host in resolved if "collector.example" in host] == []
request_records: Final = _request_history(http_mock)
token_requests: Final = tuple(request for request in request_records if request.url == token_url)
export_requests: Final = tuple(request for request in request_records if request.url == export_url)
assert len(token_requests) == 1
assert token_requests[0].headers["Authorization"].startswith("Basic ")
assert token_requests[0].body == "grant_type=client_credentials&scope=telemetry%2Fwrite"
assert tuple(request.headers["Authorization"] for request in export_requests) == (
"Bearer access-token",
"Bearer access-token",
)
def test_building_the_recorder_logs_an_affirmative_line(monkeypatch, tmp_path):
"""
Every disable path logs; a successful build must log too. Otherwise an
operator cannot tell a metering component from one that silently returned
None, which is how an unlicensed component looks healthy while exporting
nothing.
"""
_set_full_env(monkeypatch, tmp_path)
def test_bearer_auth_refreshes_before_token_expiry() -> None:
token_url: Final = "https://auth.example/oauth2/token"
export_url: Final = "https://collector.example/v1/metrics"
clock: Final = _FakeClock(1000.0)
auth: Final = bm.BillingMetricsBearerAuth(
client_id="client-id",
client_secret="client-secret",
token_url=token_url,
clock=clock,
)
session: Final = requests.Session()
session.auth = auth
http_mock: Final = requests_mock.Mocker()
with http_mock:
http_mock.post(
token_url,
[
{"json": {"access_token": "first-token", "expires_in": 120}},
{"json": {"access_token": "second-token", "expires_in": 120}},
],
)
http_mock.post(export_url, status_code=200)
session.post(export_url, data=b"metrics")
clock.set(1059.0)
session.post(export_url, data=b"metrics")
clock.set(1060.0)
session.post(export_url, data=b"metrics")
request_records: Final = _request_history(http_mock)
token_requests: Final = tuple(request for request in request_records if request.url == token_url)
export_requests: Final = tuple(request for request in request_records if request.url == export_url)
assert len(token_requests) == 2
assert tuple(request.headers["Authorization"] for request in export_requests) == (
"Bearer first-token",
"Bearer first-token",
"Bearer second-token",
)
def test_bearer_export_sets_authorization_header() -> None:
config: Final = _config()
token_url: Final = config.token_url
export_url: Final = f"{config.endpoint}/v1/metrics"
http_mock: Final = requests_mock.Mocker()
with http_mock:
http_mock.post(token_url, json={"access_token": "export-token", "expires_in": 3600})
http_mock.post(export_url, status_code=200)
provider: Final = bm.build_bearer_meter_provider(config)
try:
provider.get_meter("test").create_counter("requests").add(1)
assert provider.force_flush(timeout_millis=1000)
finally:
provider.shutdown(timeout_millis=1000)
export_requests: Final = tuple(request for request in _request_history(http_mock) if request.url == export_url)
assert export_requests
assert all(request.headers["Authorization"] == "Bearer export-token" for request in export_requests)
def test_unauthorized_export_refetches_token_before_next_export() -> None:
config: Final = _config()
token_url: Final = config.token_url
export_url: Final = f"{config.endpoint}/v1/metrics"
http_mock: Final = requests_mock.Mocker()
with http_mock:
http_mock.post(
token_url,
[
{"json": {"access_token": "first-token", "expires_in": 3600}},
{"json": {"access_token": "second-token", "expires_in": 3600}},
],
)
http_mock.post(export_url, [{"status_code": 401}, {"status_code": 200}])
provider: Final = bm.build_bearer_meter_provider(config)
try:
counter: Final = provider.get_meter("test").create_counter("requests")
counter.add(1)
provider.force_flush(timeout_millis=1000)
counter.add(1)
provider.force_flush(timeout_millis=1000)
finally:
provider.shutdown(timeout_millis=1000)
requests: Final = _request_history(http_mock)
token_requests: Final = tuple(request for request in requests if request.url == token_url)
export_requests: Final = tuple(request for request in requests if request.url == export_url)
assert len(token_requests) == 2
assert tuple(request.headers["Authorization"] for request in export_requests[:2]) == (
"Bearer first-token",
"Bearer second-token",
)
def test_failed_bearer_token_fetch_does_not_raise_from_export() -> None:
config: Final = _config()
token_url: Final = config.token_url
export_url: Final = f"{config.endpoint}/v1/metrics"
http_mock: Final = requests_mock.Mocker()
with http_mock:
http_mock.post(token_url, status_code=400)
provider: Final = bm.build_bearer_meter_provider(config)
try:
provider.get_meter("test").create_counter("requests").add(1)
assert provider.force_flush(timeout_millis=1000)
finally:
provider.shutdown(timeout_millis=1000)
request_urls: Final = tuple(request.url for request in _request_history(http_mock))
assert token_url in request_urls
assert export_url not in request_urls
def test_export_interval_default_and_override(monkeypatch: pytest.MonkeyPatch) -> None:
_set_full_env(monkeypatch)
default_config: Final = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert default_config is not None
assert default_config.export_interval_ms == bm.DEFAULT_EXPORT_INTERVAL_MS
monkeypatch.setenv(bm.EXPORT_INTERVAL_ENV, "5000")
monkeypatch.setattr(bm, "OTLPMetricExporter", _fake_exporter_class({}))
infos: List[str] = []
monkeypatch.setattr(bm.verbose_proxy_logger, "info", lambda msg, *args: infos.append(msg % args if args else msg))
recorder = bm.build_billing_metrics_recorder(premium=True, license_data={"user_id": "org-1"}, litellm_version="1.0")
assert recorder is not None
joined = "\n".join(infos)
assert "https://collector.example:4317" in joined
assert "5000" in joined
def test_unlicensed_build_does_not_warn(monkeypatch, tmp_path):
"""Unlicensed is the common OSS case; warning there would be pure noise."""
_set_full_env(monkeypatch, tmp_path)
warnings: List[str] = []
monkeypatch.setattr(bm.verbose_proxy_logger, "warning", lambda msg, *args: warnings.append(str(msg)))
assert bm.build_billing_metrics_recorder(premium=False, license_data=None, litellm_version="1.0") is None
assert warnings == []
def test_shutdown_flushes_active_recorder_once(monkeypatch, tmp_path):
"""The shutdown hook must flush the recorder the factory built (buffered
counts are lost on restart otherwise) and be idempotent for repeat calls."""
_set_full_env(monkeypatch, tmp_path)
shutdowns = []
class _SpyProvider:
def get_meter(self, name):
return MeterProvider().get_meter(name)
def shutdown(self, timeout_millis=None):
shutdowns.append(timeout_millis)
monkeypatch.setattr(bm, "build_mtls_meter_provider", lambda config: _SpyProvider())
recorder = bm.build_billing_metrics_recorder(premium=True, license_data=None, litellm_version="1.0")
assert recorder is not None
bm.shutdown_billing_metrics_recorder()
bm.shutdown_billing_metrics_recorder()
assert shutdowns == [bm.SHUTDOWN_FLUSH_TIMEOUT_MS]
def test_shutdown_without_active_recorder_is_noop():
bm.shutdown_billing_metrics_recorder()
# ── Config loading ────────────────────────────────────────────────────────────
def test_load_config_carries_license_id(monkeypatch, tmp_path):
_set_full_env(monkeypatch, tmp_path)
config = bm.load_billing_metrics_config(license_data={"user_id": "org-42"}, litellm_version="9.9")
assert config is not None and config.license_id == "org-42" and config.litellm_version == "9.9"
def test_load_config_with_empty_string_env_is_disabled(monkeypatch, tmp_path):
"""An env var set to the empty string is as unusable as an unset one and
must disable metering rather than produce a config with a blank endpoint."""
paths = _write_certs(tmp_path)
monkeypatch.setenv(bm.ENDPOINT_ENV, "")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, paths[bm.CLIENT_CERT_ENV])
monkeypatch.setenv(bm.CLIENT_KEY_ENV, paths[bm.CLIENT_KEY_ENV])
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
_CLIENT_CERT_PEM = "-----BEGIN CERTIFICATE-----\nclient-cert-body\n-----END CERTIFICATE-----"
_CLIENT_KEY_PEM = "-----BEGIN PRIVATE KEY-----\nclient-key-body\n-----END PRIVATE KEY-----"
_CA_CERT_PEM = "-----BEGIN CERTIFICATE-----\nca-body\n-----END CERTIFICATE-----"
def test_load_config_materializes_inline_pem_content(monkeypatch):
"""
ECS and Cloud Run inject secrets as env content, not as mounted files, so the
cert env vars must accept PEM directly. The exporter takes paths, so the PEM
is written to disk and the config points at those files.
"""
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, _CLIENT_CERT_PEM)
monkeypatch.setenv(bm.CLIENT_KEY_ENV, _CLIENT_KEY_PEM)
monkeypatch.setenv(bm.CA_CERT_ENV, _CA_CERT_PEM)
config = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert config is not None
assert config.ca_cert_path is not None
written = {
config.client_cert_path: _CLIENT_CERT_PEM,
config.client_key_path: _CLIENT_KEY_PEM,
config.ca_cert_path: _CA_CERT_PEM,
}
for path, pem in written.items():
assert path != pem, "config must carry a file path, not the PEM itself"
assert os.path.isfile(path)
assert Path(path).read_text(encoding="utf-8") == f"{pem}\n"
# The private key must not be world- or group-readable.
assert stat.S_IMODE(os.stat(config.client_key_path).st_mode) == 0o600
def test_load_config_accepts_a_mix_of_pem_content_and_file_paths(monkeypatch, tmp_path):
"""A deployment may mount the CA but inject the client credentials inline."""
paths = _write_certs(tmp_path)
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, _CLIENT_CERT_PEM)
monkeypatch.setenv(bm.CLIENT_KEY_ENV, _CLIENT_KEY_PEM)
monkeypatch.setenv(bm.CA_CERT_ENV, paths[bm.CA_CERT_ENV])
config = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert config is not None
assert config.ca_cert_path == paths[bm.CA_CERT_ENV]
assert Path(config.client_cert_path).read_text(encoding="utf-8") == f"{_CLIENT_CERT_PEM}\n"
def test_load_config_leaves_file_paths_untouched(monkeypatch, tmp_path):
"""Path-valued env vars keep working; nothing is copied or rewritten."""
paths = _set_full_env(monkeypatch, tmp_path)
config = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert config is not None
assert config.client_cert_path == paths[bm.CLIENT_CERT_ENV]
assert config.client_key_path == paths[bm.CLIENT_KEY_ENV]
assert config.ca_cert_path == paths[bm.CA_CERT_ENV]
def test_load_config_with_inline_pem_disabled_when_unwritable(monkeypatch):
"""A failure to materialize the PEM disables metering instead of raising."""
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, _CLIENT_CERT_PEM)
monkeypatch.setenv(bm.CLIENT_KEY_ENV, _CLIENT_KEY_PEM)
def _explode(prefix=None):
raise OSError("read-only filesystem")
monkeypatch.setattr(bm.tempfile, "mkdtemp", _explode)
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
def test_load_config_never_logs_credential_values(monkeypatch):
"""
A value that is neither a readable path nor `-----BEGIN`-prefixed PEM is
still secret material. The disable warning must name the env vars, never
echo their contents, or a malformed key lands in the proxy logs.
"""
secret_material = "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQ-not-pem-prefixed"
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, secret_material)
monkeypatch.setenv(bm.CLIENT_KEY_ENV, secret_material)
logged: List[str] = []
def _capture(msg, *args):
logged.append(msg % args if args else msg)
monkeypatch.setattr(bm.verbose_proxy_logger, "warning", _capture)
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
joined = "\n".join(logged)
assert secret_material not in joined
assert bm.CLIENT_CERT_ENV in joined and bm.CLIENT_KEY_ENV in joined
def test_load_config_with_empty_pem_env_is_disabled(monkeypatch):
"""Empty stays empty: an unset secret must not be mistaken for inline PEM."""
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://collector.example:4317")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, "")
monkeypatch.setenv(bm.CLIENT_KEY_ENV, "")
assert bm.load_billing_metrics_config(license_data=None, litellm_version="1.0") is None
def test_export_interval_default_and_override(monkeypatch):
assert bm._export_interval_ms() == bm.DEFAULT_EXPORT_INTERVAL_MS
monkeypatch.setenv(bm.EXPORT_INTERVAL_ENV, "5000")
assert bm._export_interval_ms() == 5000
override_config: Final = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert override_config is not None
assert override_config.export_interval_ms == 5000
monkeypatch.setenv(bm.EXPORT_INTERVAL_ENV, "not-a-number")
assert bm._export_interval_ms() == bm.DEFAULT_EXPORT_INTERVAL_MS
invalid_config: Final = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert invalid_config is not None
assert invalid_config.export_interval_ms == bm.DEFAULT_EXPORT_INTERVAL_MS
# ── OTLP/HTTP exporter wiring ─────────────────────────────────────────────────
def test_resource_attributes_include_license_id() -> None:
attributes: Final = bm.resource_attributes(_config(license_id="org-7"))
assert attributes == {
"service.name": "litellm-proxy",
"litellm.version": "1.2.3",
"litellm.license.id": "org-7",
}
def test_metrics_endpoint_appends_signal_path():
assert bm._metrics_endpoint("https://telemetry.example.com") == "https://telemetry.example.com/v1/metrics"
assert bm._metrics_endpoint("https://telemetry.example.com/") == "https://telemetry.example.com/v1/metrics"
assert bm._metrics_endpoint("https://telemetry.example.com/v1/metrics") == "https://telemetry.example.com/v1/metrics"
def test_resource_attributes_omit_license_id_when_absent() -> None:
assert "litellm.license.id" not in bm.resource_attributes(_config(license_id=None))
def _fake_exporter_class(captured: Dict[str, object]) -> type:
"""A no-network stand-in for OTLPMetricExporter. Tests that build a real
MeterProvider must install this: the real exporter resolves the collector
host and opens a TLS connection on the reader's first export and on the
shutdown flush."""
class _FakeExporter:
# PeriodicExportingMetricReader probes these on the exporter it wraps.
_preferred_temporality: dict = {}
_preferred_aggregation: dict = {}
def __init__(self, **kwargs):
captured.update(kwargs)
def export(self, *args, **kwargs):
return None
def shutdown(self, *args, **kwargs):
return None
def force_flush(self, *args, **kwargs):
return True
return _FakeExporter
def test_meter_provider_wires_client_cert_into_http_exporter(tmp_path, monkeypatch):
"""Client cert+key authenticate us at the collector's mTLS front end; CA override rides certificate_file."""
captured: Dict[str, object] = {}
monkeypatch.setattr(bm, "OTLPMetricExporter", _fake_exporter_class(captured))
config = _config(tmp_path)
provider = bm.build_mtls_meter_provider(config)
provider.shutdown()
assert captured["endpoint"] == "https://collector.example:4317/v1/metrics"
assert captured["client_certificate_file"] == config.client_cert_path
assert captured["client_key_file"] == config.client_key_path
assert captured["certificate_file"] == config.ca_cert_path
def test_load_config_without_ca_is_valid(monkeypatch, tmp_path):
"""The production collector presents a public web-PKI cert: no CA override required."""
paths = _write_certs(tmp_path)
monkeypatch.setenv(bm.ENDPOINT_ENV, "https://telemetry.example.com")
monkeypatch.setenv(bm.CLIENT_CERT_ENV, paths[bm.CLIENT_CERT_ENV])
monkeypatch.setenv(bm.CLIENT_KEY_ENV, paths[bm.CLIENT_KEY_ENV])
config = bm.load_billing_metrics_config(license_data=None, litellm_version="1.0")
assert config is not None and config.ca_cert_path is None
# ── Resource and metric attributes ────────────────────────────────────────────
def test_resource_attributes_include_license_id(tmp_path):
attrs = bm._resource_attributes(_config(tmp_path, license_id="org-7"))
assert attrs["service.name"] == "litellm-proxy"
assert attrs["litellm.version"] == "1.2.3"
assert attrs["litellm.license.id"] == "org-7"
def test_resource_attributes_omit_license_id_when_absent(tmp_path):
attrs = bm._resource_attributes(_config(tmp_path, license_id=None))
assert "litellm.license.id" not in attrs
def test_billable_attributes_with_model_id():
attrs = bm._billable_attributes(BillableCategory.LLM, "/chat/completions", 200, "deploy-3")
assert attrs == {
def test_billable_attributes_include_model_id() -> None:
assert bm.billable_attributes(
BillableCategory.LLM, "/chat/completions", 200, "deploy-3"
) == {
"litellm.endpoint.category": "llm",
"http.route": "/chat/completions",
"http.response.status_code": 200,
@ -400,33 +381,35 @@ def test_billable_attributes_with_model_id():
}
def test_billable_attributes_omit_model_id_when_none():
attrs = bm._billable_attributes(BillableCategory.MCP, "/mcp", 200, None)
assert "litellm.model_id" not in attrs
def test_billable_attributes_omit_model_id_when_none() -> None:
attributes: Final = bm.billable_attributes(
BillableCategory.MCP, "/mcp", 200, None
)
assert "litellm.model_id" not in attributes
# ── End-to-end recording via in-memory reader ─────────────────────────────────
def _counter_points(reader: InMemoryMetricReader) -> tuple[tuple[str, int], ...]:
data: Final = reader.get_metrics_data()
if data is None:
return ()
return tuple(
(_metric_category(point.attributes), int(point.value))
for resource_metric in data.resource_metrics
for scope_metric in resource_metric.scope_metrics
for metric in scope_metric.metrics
if metric.name == bm.METRIC_NAME and isinstance(metric.data, Sum)
for point in metric.data.data_points
)
def _counter_points(reader: InMemoryMetricReader):
data = reader.get_metrics_data()
for resource_metric in data.resource_metrics:
for scope_metric in resource_metric.scope_metrics:
for metric in scope_metric.metrics:
if metric.name == bm.METRIC_NAME:
return list(metric.data.data_points)
return []
def test_record_increments_counter_with_attributes():
reader = InMemoryMetricReader()
recorder = bm.BillingMetricsRecorder(MeterProvider(metric_readers=[reader]))
def test_record_increments_counter_with_attributes() -> None:
reader: Final = InMemoryMetricReader()
recorder: Final = bm.BillingMetricsRecorder(MeterProvider(metric_readers=[reader]))
recorder.record(category=BillableCategory.LLM, route="/chat/completions", status_code=200, model_id="m1")
recorder.record(category=BillableCategory.LLM, route="/chat/completions", status_code=200, model_id="m1")
recorder.record(category=BillableCategory.MCP, route="/mcp", status_code=200, model_id=None)
points = _counter_points(reader)
by_category = {point.attributes["litellm.endpoint.category"]: point.value for point in points}
assert by_category["llm"] == 2
assert by_category["mcp"] == 1
points: Final = _counter_points(reader)
by_category: Final = dict(points)
assert by_category == {"llm": 2, "mcp": 1}