litellm/deploy/performance

Performance stack (EKS)

Primary path: Helm + Argo CD — deploy/charts/litellm-helm with values in helm/litellm-values.yaml (bundled Bitnami Postgres + Redis, shared Redis cache, LoadBalancer, EKS nodeSelector). Observability: optional kube-prometheus-stack Application (Helm chart in cluster, not Bitnami node pools).

Legacy path: in-cluster Postgres + Redis + static Prometheus under legacy/kustomize/ (manual kubectl kustomize only; legacy Postgres uses emptyDir).

---

What runs where

Piece	Role
EKS	Cluster from eksctl-cluster.yaml (or console).
Helm	Packages LiteLLM, Postgres + Redis subcharts, migration Job, mock via extraResources.
Argo CD	Watches Git and syncs Application manifests in argocd/.

---

Runbook: Helm + Argo (recommended)

Step	Action
1–3	AWS CLI, kubectl, eksctl — see AWS CLI + kubectl.
4	eksctl create cluster -f deploy/performance/eksctl-cluster.yaml
5	aws eks update-kubeconfig --region us-east-1 --name litellm-performance (or your metadata.name / region from eksctl-cluster.yaml)
6	EBS CSI — Bitnami Postgres (and often Redis) use PVCs. Install the EKS EBS CSI add-on and ensure a default StorageClass exists.
7	Install Argo CD — kubectl create namespace argocd then kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
8	Register Git repo if private (Argo Settings → Repositories).
9	Apply Applications (pin targetRevision in each file to your branch if needed):

kubectl apply -n argocd -f deploy/performance/argocd/application-litellm.yaml
kubectl apply -n argocd -f deploy/performance/argocd/application-kube-prometheus-stack.yaml

| 10 | Argo UI → sync litellm-helm, then kube-prometheus-stack (large first sync). | | 11 | Postgres / Redis — override defaults in helm/litellm-values.yaml (postgresql.auth, redis.auth) for non-dev clusters; re-sync after edits. | | 12 | Master key — chart creates litellm-masterkey by default; override with masterkeySecretName + your Secret if required. | | 13 | Verify: kubectl -n performance get pods,svc — litellm Service LoadBalancer on port 4000. |

---

Layout

deploy/performance/
  argocd/
    application-litellm.yaml
    application-kube-prometheus-stack.yaml
  helm/
    litellm-values.yaml
  eksctl-cluster.yaml
  legacy/
    dev-config.yaml                 # Proxy YAML for legacy Kustomize only
    kustomize/                      # Raw Postgres + Redis + Prometheus (optional)

---

Helm values

• Source of truth: helm/litellm-values.yaml → proxy_config. Legacy overlay uses legacy/dev-config.yaml only if you still run Kustomize.
• Argo must sync a targetRevision that contains this file (paths in application-litellm.yaml are relative to deploy/charts/litellm-helm).

---

Troubleshooting reference

Stuck Postgres Pending / PVC FailedBinding

Install EBS CSI and a StorageClass; see EKS EBS CSI.

Stuck LiteLLM Init:* / old wait-db

That was an older raw Deployment spec. This Helm chart does not use those init containers.

EBS CSI / IRSA / OIDC

The EBS CSI controller must use IRSA (eks.amazonaws.com/role-arn on the ServiceAccount). Typical fixes:

• Associate OIDC: eksctl utils associate-iam-oidc-provider --cluster <name> --region <region> --approve
• Create role + annotate SA: eksctl create iamserviceaccount ... --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy
• Flag: --override-existing-serviceaccounts (plural)
• CloudFormation stack ROLLBACK_COMPLETE: disable termination protection, delete stack, retry
• Set add-on role: aws eks update-addon --cluster-name <name> --addon-name aws-ebs-csi-driver --service-account-role-arn <role-arn> --resolve-conflicts OVERWRITE

Legacy: Kustomize (Postgres + Redis + Prometheus)

Create the namespace and Secret first (password must match DATABASE_URL):

kubectl apply -f deploy/performance/legacy/kustomize/namespace.yaml
kubectl create secret generic litellm-dotenv \
  --namespace performance \
  --from-literal=DATABASE_PASSWORD='choose-a-password' \
  --from-literal=DATABASE_URL='postgresql://litellm:choose-a-password@postgres:5432/litellm' \
  --from-literal=PROXY_MASTER_KEY='sk-your-admin-key' \
  --dry-run=client -o yaml | kubectl apply -f -

Then apply the overlay (from repo root):

kubectl kustomize deploy/performance/legacy/kustomize \
  --load-restrictor=LoadRestrictionsNone \
  | kubectl apply -f -

Postgres in this overlay is emptyDir (ephemeral). For PVC-backed DBs, use the Helm chart.

---

TODOs / next steps

• Pin Argo targetRevision to a tag or SHA for reproducibility.
• kube-prometheus-stack: Add scrape config for LiteLLM metrics via chart values.
• Secrets: External Secrets / SSM for postgresql.auth, redis.auth, and PROXY_MASTER_KEY in production.

---

AWS CLI + kubectl (first-time setup)

1. Install AWS CLI v2

Follow the official install guide. Verify:

aws --version   # aws-cli/2.x

2. Credentials

Use aws configure, SSO, or roles your org supports. Verify:

aws sts get-caller-identity

3. kubectl

Install a kubectl version compatible with your EKS control plane.

4. eksctl

Install eksctl. Create/delete cluster:

eksctl create cluster -f deploy/performance/eksctl-cluster.yaml
# eksctl delete cluster -f deploy/performance/eksctl-cluster.yaml

---

Argo CD UI (quick login)

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo
kubectl -n argocd port-forward svc/argocd-server 8080:443

Open https://localhost:8080, user admin. Docs: Argo CD getting started.

---

Teardown

Argo-managed: delete the Applications (or let prune), then remove workloads/namespaces as needed.

Legacy Kustomize:

kubectl kustomize deploy/performance/legacy/kustomize \
  --load-restrictor=LoadRestrictionsNone \
  | kubectl delete -f -

After removing mock-files-api from Git

kubectl -n performance delete deploy,svc mock-files-api --ignore-not-found
kubectl -n performance delete configmap mock-files-api-code --ignore-not-found