scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-12 23:01:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 87

feat: Add Istio Ingress Support

Browse source
This commit is contained in:
StyleTang 2026-03-13 15:13:33 +08:00
parent 9cd7ad2634
commit bd69eec692
6 changed files with 960 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

79
deploy/charts/litellm-helm/README.md
View file

@ -155,6 +155,85 @@ data:
Source: [GitHub Gist from troyharvey](https://gist.github.com/troyharvey/4506472732157221e04c6b15e3b3f094)
### Istio Service Mesh Settings
These settings configure Istio-native networking resources (Gateway, VirtualService, DestinationRule).
Use these **instead of** `ingress` when your cluster runs Istio. Uses `networking.istio.io/v1` (GA since Istio 1.22).
| Name | Description | Value |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | -------------------- |
| `istio.enabled` | Enable Istio networking resources | `false` |
| `istio.hosts` | List of hostnames for the Gateway and VirtualService | `["api.example.local"]` |
| `istio.gateway.enabled` | Create an Istio Gateway resource. Set `false` to use an existing shared gateway (recommended in production) | `false` |
| `istio.gateway.selector` | Label selector to bind the Gateway to an ingress gateway pod | `{istio: ingressgateway}` |
| `istio.gateway.httpPort` | HTTP port number on the Gateway server | `80` |
| `istio.gateway.tls.enabled` | Add a HTTPS server block with TLS to the Gateway (also sets HTTP→HTTPS redirect) | `false` |
| `istio.gateway.tls.httpsPort` | HTTPS port number on the Gateway server | `443` |
| `istio.gateway.tls.mode` | Istio TLS mode: `SIMPLE`, `MUTUAL`, or `PASSTHROUGH` | `SIMPLE` |
| `istio.gateway.tls.credentialName` | Name of the Kubernetes Secret (in `istio-system`) holding the TLS certificate | `""` |
| `istio.gateway.labels` | Additional labels for the Gateway resource | `{}` |
| `istio.gateway.annotations` | Additional annotations for the Gateway resource | `{}` |
| `istio.virtualService.enabled` | Create a VirtualService resource | `true` |
| `istio.virtualService.gateways` | Additional gateway references (e.g. `["istio-system/shared-gateway"]`). The chart's own Gateway is auto-included | `[]` |
| `istio.virtualService.http` | Custom HTTP route rules. When set, overrides the default catch-all route | `[]` |
| `istio.virtualService.timeout` | Request timeout for the default route (e.g. `"30s"`) | `""` |
| `istio.virtualService.retries` | Retry policy for the default route. See [Istio HTTPRetry](https://istio.io/latest/docs/reference/config/networking/virtual-service/#HTTPRetry) | `{}` |
| `istio.virtualService.labels` | Additional labels for the VirtualService resource | `{}` |
| `istio.virtualService.annotations` | Additional annotations for the VirtualService resource | `{}` |
| `istio.destinationRule.enabled` | Create a DestinationRule resource for circuit breaking and connection pool tuning | `false` |
| `istio.destinationRule.trafficPolicy` | Istio TrafficPolicy block (connectionPool, outlierDetection, tls, etc.) | `{}` |
| `istio.destinationRule.labels` | Additional labels for the DestinationRule resource | `{}` |
| `istio.destinationRule.annotations` | Additional annotations for the DestinationRule resource | `{}` |
#### Example: VirtualService with existing shared Gateway
```yaml
istio:
enabled: true
hosts:
- litellm.example.com
gateway:
enabled: false # platform team owns the Gateway
virtualService:
enabled: true
gateways:
- istio-system/shared-gateway
timeout: "60s"
retries:
attempts: 3
perTryTimeout: "20s"
retryOn: "gateway-error,connect-failure,refused-stream"
```
#### Example: Standalone Gateway with TLS + DestinationRule
```yaml
istio:
enabled: true
hosts:
- litellm.example.com
gateway:
enabled: true
tls:
enabled: true
credentialName: litellm-tls-cert
virtualService:
enabled: true
destinationRule:
enabled: true
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
http2MaxRequests: 1000
outlierDetection:
consecutive5xxErrors: 5
interval: "30s"
baseEjectionTime: "30s"
```
### Migration Job Settings
The migration job supports both ArgoCD and Helm hooks to ensure database migrations run at the appropriate time during deployments.

22
deploy/charts/litellm-helm/templates/istio-destinationrule.yaml Normal file
View file

@ -0,0 +1,22 @@
{{- if and .Values.istio.enabled .Values.istio.destinationRule.enabled -}}
{{- $fullName := include "litellm.fullname" . -}}
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: {{ $fullName }}
labels:
{{- include "litellm.labels" . | nindent 4 }}
{{- with .Values.istio.destinationRule.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.istio.destinationRule.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
host: {{ $fullName }}
{{- with .Values.istio.destinationRule.trafficPolicy }}
trafficPolicy:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}

50
deploy/charts/litellm-helm/templates/istio-gateway.yaml Normal file
View file

@ -0,0 +1,50 @@
{{- if and .Values.istio.enabled .Values.istio.gateway.enabled -}}
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
name: {{ include "litellm.fullname" . }}
labels:
{{- include "litellm.labels" . | nindent 4 }}
{{- with .Values.istio.gateway.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.istio.gateway.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
selector:
{{- toYaml .Values.istio.gateway.selector | nindent 4 }}
servers:
- port:
number: {{ .Values.istio.gateway.httpPort | default 80 }}
name: http
protocol: HTTP
{{- if .Values.istio.gateway.tls.enabled }}
tls:
httpsRedirect: true
{{- end }}
hosts:
{{- range .Values.istio.hosts }}
- {{ . | quote }}
{{- end }}
{{- if .Values.istio.gateway.tls.enabled }}
- port:
number: {{ .Values.istio.gateway.tls.httpsPort | default 443 }}
name: https
protocol: HTTPS
tls:
mode: {{ .Values.istio.gateway.tls.mode | default "SIMPLE" }}
credentialName: {{ .Values.istio.gateway.tls.credentialName | quote }}
{{- if .Values.istio.gateway.tls.minProtocolVersion }}
minProtocolVersion: {{ .Values.istio.gateway.tls.minProtocolVersion }}
{{- end }}
{{- if .Values.istio.gateway.tls.maxProtocolVersion }}
maxProtocolVersion: {{ .Values.istio.gateway.tls.maxProtocolVersion }}
{{- end }}
hosts:
{{- range .Values.istio.hosts }}
- {{ . | quote }}
{{- end }}
{{- end }}
{{- end }}

49
deploy/charts/litellm-helm/templates/istio-virtualservice.yaml Normal file
View file

@ -0,0 +1,49 @@
{{- if and .Values.istio.enabled .Values.istio.virtualService.enabled -}}
{{- $fullName := include "litellm.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: {{ $fullName }}
labels:
{{- include "litellm.labels" . | nindent 4 }}
{{- with .Values.istio.virtualService.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.istio.virtualService.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
hosts:
{{- range .Values.istio.hosts }}
- {{ . | quote }}
{{- end }}
gateways:
{{- if .Values.istio.gateway.enabled }}
- {{ $fullName }}
{{- end }}
{{- range .Values.istio.virtualService.gateways }}
- {{ . | quote }}
{{- end }}
http:
{{- if .Values.istio.virtualService.http }}
{{- toYaml .Values.istio.virtualService.http | nindent 4 }}
{{- else }}
- match:
- uri:
prefix: /
{{- if .Values.istio.virtualService.timeout }}
timeout: {{ .Values.istio.virtualService.timeout }}
{{- end }}
{{- if .Values.istio.virtualService.retries }}
retries:
{{- toYaml .Values.istio.virtualService.retries | nindent 8 }}
{{- end }}
route:
- destination:
host: {{ $fullName }}
port:
number: {{ $svcPort }}
{{- end }}
{{- end }}

696
deploy/charts/litellm-helm/tests/istio_tests.yaml Normal file
View file

@ -0,0 +1,696 @@
suite: Istio Gateway and VirtualService Tests
templates:
- istio-gateway.yaml
- istio-virtualservice.yaml
- istio-destinationrule.yaml
tests:
#
# === Disabled by Default ===
#
- it: should not create Gateway or VirtualService by default
asserts:
- hasDocuments:
count: 0
template: istio-gateway.yaml
- hasDocuments:
count: 0
template: istio-virtualservice.yaml
- it: should not create Gateway or VirtualService when istio.enabled is false
set:
istio.enabled: false
asserts:
- hasDocuments:
count: 0
template: istio-gateway.yaml
- hasDocuments:
count: 0
template: istio-virtualservice.yaml
#
# === VirtualService Only (existing Gateway) ===
#
- it: should create VirtualService without Gateway when gateway.enabled is false
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: false
virtualService:
enabled: true
gateways:
- istio-system/shared-gateway
asserts:
- hasDocuments:
count: 0
template: istio-gateway.yaml
- hasDocuments:
count: 1
template: istio-virtualservice.yaml
- isKind:
of: VirtualService
template: istio-virtualservice.yaml
- isAPIVersion:
of: networking.istio.io/v1
template: istio-virtualservice.yaml
- it: should reference external gateway in VirtualService spec
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: false
virtualService:
enabled: true
gateways:
- istio-system/shared-gateway
asserts:
- equal:
path: spec.gateways[0]
value: istio-system/shared-gateway
template: istio-virtualservice.yaml
#
# === Gateway + VirtualService Together ===
#
- it: should create both Gateway and VirtualService when both enabled
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- hasDocuments:
count: 1
template: istio-gateway.yaml
- hasDocuments:
count: 1
template: istio-virtualservice.yaml
- isKind:
of: Gateway
template: istio-gateway.yaml
- isKind:
of: VirtualService
template: istio-virtualservice.yaml
- it: should use correct API version for both resources
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- isAPIVersion:
of: networking.istio.io/v1
template: istio-gateway.yaml
- isAPIVersion:
of: networking.istio.io/v1
template: istio-virtualservice.yaml
#
# === Gateway Configuration ===
#
- it: should set Gateway name using fullname
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
asserts:
- equal:
path: metadata.name
value: RELEASE-NAME-litellm
template: istio-gateway.yaml
- it: should set Gateway selector from values
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
selector:
istio: ingressgateway
asserts:
- equal:
path: spec.selector.istio
value: ingressgateway
template: istio-gateway.yaml
- it: should set custom Gateway selector
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
selector:
istio: my-custom-gateway
asserts:
- equal:
path: spec.selector.istio
value: my-custom-gateway
template: istio-gateway.yaml
- it: should configure Gateway hosts
set:
istio:
enabled: true
hosts:
- api.example.com
- api2.example.com
gateway:
enabled: true
asserts:
- equal:
path: spec.servers[0].hosts[0]
value: api.example.com
template: istio-gateway.yaml
- equal:
path: spec.servers[0].hosts[1]
value: api2.example.com
template: istio-gateway.yaml
- it: should configure HTTP-only Gateway by default
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
asserts:
- equal:
path: spec.servers[0].port.number
value: 80
template: istio-gateway.yaml
- equal:
path: spec.servers[0].port.protocol
value: HTTP
template: istio-gateway.yaml
- isNull:
path: spec.servers[1]
template: istio-gateway.yaml
- it: should configure Gateway with TLS
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
tls:
enabled: true
credentialName: litellm-tls-cert
asserts:
# HTTP server with redirect
- equal:
path: spec.servers[0].port.protocol
value: HTTP
template: istio-gateway.yaml
- equal:
path: spec.servers[0].tls.httpsRedirect
value: true
template: istio-gateway.yaml
# HTTPS server
- equal:
path: spec.servers[1].port.number
value: 443
template: istio-gateway.yaml
- equal:
path: spec.servers[1].port.protocol
value: HTTPS
template: istio-gateway.yaml
- equal:
path: spec.servers[1].tls.mode
value: SIMPLE
template: istio-gateway.yaml
- equal:
path: spec.servers[1].tls.credentialName
value: litellm-tls-cert
template: istio-gateway.yaml
- it: should support custom TLS mode and ports
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
httpPort: 8080
tls:
enabled: true
httpsPort: 8443
mode: MUTUAL
credentialName: litellm-mtls-cert
minProtocolVersion: TLSV1_2
asserts:
- equal:
path: spec.servers[0].port.number
value: 8080
template: istio-gateway.yaml
- equal:
path: spec.servers[1].port.number
value: 8443
template: istio-gateway.yaml
- equal:
path: spec.servers[1].tls.mode
value: MUTUAL
template: istio-gateway.yaml
- equal:
path: spec.servers[1].tls.minProtocolVersion
value: TLSV1_2
template: istio-gateway.yaml
- it: should add Gateway labels
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
labels:
team: platform
asserts:
- equal:
path: metadata.labels.team
value: platform
template: istio-gateway.yaml
- it: should add Gateway annotations
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
annotations:
external-dns.alpha.kubernetes.io/hostname: api.example.com
asserts:
- equal:
path: metadata.annotations["external-dns.alpha.kubernetes.io/hostname"]
value: api.example.com
template: istio-gateway.yaml
#
# === VirtualService Configuration ===
#
- it: should set VirtualService name using fullname
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- equal:
path: metadata.name
value: RELEASE-NAME-litellm
template: istio-virtualservice.yaml
- it: should configure VirtualService hosts
set:
istio:
enabled: true
hosts:
- api.example.com
- api2.example.com
virtualService:
enabled: true
gateways:
- some-gateway
asserts:
- equal:
path: spec.hosts[0]
value: api.example.com
template: istio-virtualservice.yaml
- equal:
path: spec.hosts[1]
value: api2.example.com
template: istio-virtualservice.yaml
- it: should auto-reference chart Gateway when gateway.enabled
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- equal:
path: spec.gateways[0]
value: RELEASE-NAME-litellm
template: istio-virtualservice.yaml
- it: should include both chart Gateway and additional gateways
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
gateways:
- mesh
asserts:
- equal:
path: spec.gateways[0]
value: RELEASE-NAME-litellm
template: istio-virtualservice.yaml
- equal:
path: spec.gateways[1]
value: mesh
template: istio-virtualservice.yaml
- it: should generate default route to service
set:
service:
port: 4000
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- equal:
path: spec.http[0].match[0].uri.prefix
value: /
template: istio-virtualservice.yaml
- equal:
path: spec.http[0].route[0].destination.host
value: RELEASE-NAME-litellm
template: istio-virtualservice.yaml
- equal:
path: spec.http[0].route[0].destination.port.number
value: 4000
template: istio-virtualservice.yaml
- it: should use custom service port in VirtualService route
set:
service:
port: 8080
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- equal:
path: spec.http[0].route[0].destination.port.number
value: 8080
template: istio-virtualservice.yaml
- it: should allow custom HTTP routes
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
http:
- match:
- uri:
prefix: /v1
route:
- destination:
host: litellm-v1
port:
number: 4000
- match:
- uri:
prefix: /v2
route:
- destination:
host: litellm-v2
port:
number: 4000
asserts:
- equal:
path: spec.http[0].match[0].uri.prefix
value: /v1
template: istio-virtualservice.yaml
- equal:
path: spec.http[0].route[0].destination.host
value: litellm-v1
template: istio-virtualservice.yaml
- equal:
path: spec.http[1].match[0].uri.prefix
value: /v2
template: istio-virtualservice.yaml
- it: should add VirtualService labels
set:
istio:
enabled: true
hosts:
- api.example.com
virtualService:
enabled: true
labels:
version: v1
gateways:
- some-gateway
asserts:
- equal:
path: metadata.labels.version
value: v1
template: istio-virtualservice.yaml
- it: should add VirtualService annotations
set:
istio:
enabled: true
hosts:
- api.example.com
virtualService:
enabled: true
annotations:
external-dns.alpha.kubernetes.io/hostname: api.example.com
gateways:
- some-gateway
asserts:
- equal:
path: metadata.annotations["external-dns.alpha.kubernetes.io/hostname"]
value: api.example.com
template: istio-virtualservice.yaml
- it: should include standard labels on Gateway
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
asserts:
- isNotNull:
path: metadata.labels["app.kubernetes.io/name"]
template: istio-gateway.yaml
- isNotNull:
path: metadata.labels["helm.sh/chart"]
template: istio-gateway.yaml
- it: should include standard labels on VirtualService
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: true
asserts:
- isNotNull:
path: metadata.labels["app.kubernetes.io/name"]
template: istio-virtualservice.yaml
- isNotNull:
path: metadata.labels["helm.sh/chart"]
template: istio-virtualservice.yaml
- it: should not create VirtualService when virtualService.enabled is false
set:
istio:
enabled: true
hosts:
- api.example.com
gateway:
enabled: true
virtualService:
enabled: false
asserts:
- hasDocuments:
count: 1
template: istio-gateway.yaml
- hasDocuments:
count: 0
template: istio-virtualservice.yaml
#
# === DestinationRule Tests ===
#
- it: should not create DestinationRule by default
asserts:
- hasDocuments:
count: 0
template: istio-destinationrule.yaml
- it: should not create DestinationRule when istio.enabled is false
set:
istio.enabled: false
istio.destinationRule.enabled: true
asserts:
- hasDocuments:
count: 0
template: istio-destinationrule.yaml
- it: should create DestinationRule when enabled
set:
istio:
enabled: true
destinationRule:
enabled: true
asserts:
- hasDocuments:
count: 1
template: istio-destinationrule.yaml
- isKind:
of: DestinationRule
template: istio-destinationrule.yaml
- isAPIVersion:
of: networking.istio.io/v1
template: istio-destinationrule.yaml
- it: should set DestinationRule name using fullname
set:
istio:
enabled: true
destinationRule:
enabled: true
asserts:
- equal:
path: metadata.name
value: RELEASE-NAME-litellm
template: istio-destinationrule.yaml
- it: should set DestinationRule host to service fullname
set:
istio:
enabled: true
destinationRule:
enabled: true
asserts:
- equal:
path: spec.host
value: RELEASE-NAME-litellm
template: istio-destinationrule.yaml
- it: should render trafficPolicy when provided
set:
istio:
enabled: true
destinationRule:
enabled: true
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
http1MaxPendingRequests: 100
asserts:
- equal:
path: spec.trafficPolicy.connectionPool.tcp.maxConnections
value: 100
template: istio-destinationrule.yaml
- equal:
path: spec.trafficPolicy.connectionPool.http.http1MaxPendingRequests
value: 100
template: istio-destinationrule.yaml
- it: should render outlierDetection in trafficPolicy
set:
istio:
enabled: true
destinationRule:
enabled: true
trafficPolicy:
outlierDetection:
consecutive5xxErrors: 5
interval: "30s"
baseEjectionTime: "30s"
asserts:
- equal:
path: spec.trafficPolicy.outlierDetection.consecutive5xxErrors
value: 5
template: istio-destinationrule.yaml
- it: should add DestinationRule labels and annotations
set:
istio:
enabled: true
destinationRule:
enabled: true
labels:
team: platform
annotations:
note: managed-by-helm
asserts:
- equal:
path: metadata.labels.team
value: platform
template: istio-destinationrule.yaml
- equal:
path: metadata.annotations.note
value: managed-by-helm
template: istio-destinationrule.yaml
- it: should include standard labels on DestinationRule
set:
istio:
enabled: true
destinationRule:
enabled: true
asserts:
- isNotNull:
path: metadata.labels["app.kubernetes.io/name"]
template: istio-destinationrule.yaml
- isNotNull:
path: metadata.labels["helm.sh/chart"]
template: istio-destinationrule.yaml

64
deploy/charts/litellm-helm/values.yaml
View file

@ -137,6 +137,70 @@ ingress:
# hosts:
# - chart-example.local
# Istio Gateway and VirtualService configuration
# Use this instead of 'ingress' when running with Istio service mesh
istio:
enabled: false
# Hosts that the Gateway and VirtualService will match on
hosts:
- api.example.local
gateway:
enabled: false
labels: {}
annotations: {}
# Selector to bind the Gateway to the Istio ingress gateway deployment
selector:
istio: ingressgateway
# HTTP port for the Gateway server
httpPort: 80
tls:
enabled: false
# HTTPS port for the Gateway server
httpsPort: 443
# TLS mode: SIMPLE, MUTUAL, PASSTHROUGH, etc.
mode: SIMPLE
# Name of the Kubernetes secret holding the TLS cert/key (in istio-system namespace)
credentialName: ""
# Minimum TLS protocol version (e.g. TLSV1_2)
# minProtocolVersion: TLSV1_2
# Maximum TLS protocol version
# maxProtocolVersion: TLSV1_3
virtualService:
enabled: true
labels: {}
annotations: {}
# Additional gateways to attach to (e.g. use an existing shared gateway)
# If istio.gateway.enabled is true, the chart's own Gateway is automatically included
gateways: []
# # Example: use an existing shared gateway instead of creating one
# # gateways:
# # - istio-system/shared-gateway
# Custom HTTP route rules (overrides the default catch-all route)
# http: []
# Timeout for the default route (e.g. "30s")
# timeout: "30s"
# Retry policy for the default route
# retries:
# attempts: 3
# perTryTimeout: "10s"
# retryOn: "gateway-error,connect-failure,refused-stream"
destinationRule:
enabled: false
labels: {}
annotations: {}
# Traffic policy for circuit breaking and connection pool tuning
# trafficPolicy:
# connectionPool:
# tcp:
# maxConnections: 100
# http:
# http1MaxPendingRequests: 100
# http2MaxRequests: 1000
# outlierDetection:
# consecutive5xxErrors: 5
# interval: "30s"
# baseEjectionTime: "30s"
# masterkey: changeit
# if set, use this secret for the master key; otherwise, autogenerate a new one