3d5a9ede05
- Add AWS ECS Fargate stack with Aurora Postgres (IAM auth), ElastiCache Redis, S3, ALB with path-based routing to gateway/backend/ui components, Application Auto Scaling, and automated DB bootstrap + prisma migration via local-exec provisioners - Add GCP Cloud Run stack with Cloud SQL Postgres (password auth), Memorystore Redis, GCS, external HTTPS load balancer with serverless NEGs and URL map routing, and automated prisma migration via Cloud Run Job - Both stacks support typed proxy_config input mirroring the helm chart's gateway.config.proxy_config, per-component extra env vars, and Secret Manager references for provider API keys - Gateway/backend services depend on terraform_data.migration so they never start before the schema is in place, eliminating crash-loop windows on first apply - AWS stack uses IAM database authentication with a one-shot Fargate bootstrap task that creates and grants the rds_iam role to the application user; GCP stack uses password auth assembled at container startup to avoid Cloud SQL Auth Proxy sidecar complexity - Add .gitignore rules for Terraform state files, plan files, tfvars inputs, provider binaries, and crash logs while explicitly keeping .terraform.lock.hcl for provider version pinning - Include terraform.tfvars.example files, provider lock files, and comprehensive README documentation covering architecture, TLS setup, image pull strategies, and quick-start instructions for both stacks Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|---|---|---|
| .. | ||
| aws | ||
| gcp | ||
| README.md | ||
LiteLLM Terraform stacks
Two self-contained Terraform root modules that deploy the componentized
LiteLLM proxy — the gateway, backend, and UI as three independent containers
(see helm/litellm/ for the canonical chart with the same split).
| Stack | Compute | Database (writer + reader) | Cache | Object store | Public entrypoint |
|---|---|---|---|---|---|
aws/ |
ECS Fargate | Aurora Postgres (IAM auth) | ElastiCache | S3 | Application LB |
gcp/ |
Cloud Run | Cloud SQL Postgres (password auth) | Memorystore | GCS | External HTTPS LB |
Each stack creates its own VPC and managed data stores — drop in a tfvars
file and run terraform apply. Both stacks support a typed proxy_config
input (mirrors helm/litellm's gateway.config.proxy_config) and per-component
extra env vars / secret-manager refs.
Components
The proxy is split into three deployables:
| Component | Default image | Port | Role |
|---|---|---|---|
gateway |
ghcr.io/berriai/litellm-gateway:main-stable |
4000 | LLM data plane (/v1/chat/completions, /v1/embeddings, …) |
backend |
ghcr.io/berriai/litellm-backend:main-stable |
4001 | Management API (/key/*, /user/*, /team/*, /model/*, …) |
ui |
ghcr.io/berriai/litellm-ui:main-stable |
3000 | Static Next.js dashboard served by nginx |
The load balancer routes gateway path prefixes (mirrored verbatim from
gateway/routes/allowlist.py) to the gateway, UI asset paths (/,
/litellm-asset-prefix/*, /_next/*, /favicon.ico) to the UI, and
everything else to the backend.
Architecture
AWS (terraform/litellm/aws/)
┌───────────────────────────────────────┐
│ Public Internet │
└─────────────────┬─────────────────────┘
│ HTTP/80
┌───────────────▼───────────────┐
│ Application Load Balancer │
│ (path-routing listener) │
└─┬─────────────┬─────────────┬─┘
│ │ │
UI assets, / │ /v1/chat, │ /key/* │
/_next/*, … │ /v1/embed, │ /user/* │
│ … │ … │
┌─────────────▼───┐ ┌──────▼──────┐ ┌───▼──────────────┐
│ ECS Service │ │ ECS Service │ │ ECS Service │
│ (ui) │ │ (gateway) │ │ (backend) │
│ Fargate :3000 │ │ Fargate:4000│ │ Fargate :4001 │
└─────────────────┘ └──────┬──────┘ └────────┬─────────┘
│ │
┌─── private subnets (one per AZ) ──────────────────────┐
│ │
│ ┌────────────────────────┐ ┌────────────────┐ │
│ │ Aurora Postgres │ │ ElastiCache │ │
│ │ cluster (IAM auth) │ │ Redis (1 node)│ │
│ │ ┌───────┐ ┌───────┐ │ └────────────────┘ │
│ │ │writer │ │reader │ │ │
│ │ └───────┘ └───────┘ │ ┌────────────────┐ │
│ └────────────────────────┘ │ S3 bucket │ │
│ │ (versioned) │ │
│ ┌────────────────────────┐ └────────────────┘ │
│ │ Secrets Manager │ │
│ │ • LITELLM_MASTER_KEY │ ┌────────────────┐ │
│ │ • DB master password │ │ One-off ECS │ │
│ │ • user-supplied API │ │ task: prisma │ │
│ │ keys (referenced) │ │ migrate deploy │ │
│ └────────────────────────┘ └────────────────┘ │
│ │
└─── VPC ───────────────────────────────────────────────┘
│ NAT gateway in one public subnet
▼
egress to LLM providers
GCP (terraform/litellm/gcp/)
┌───────────────────────────────────────┐
│ Public Internet │
└─────────────────┬─────────────────────┘
│ HTTP/80
┌───────────────▼───────────────┐
│ External HTTPS Load Balancer │
│ (global, URL map routing) │
└─┬─────────────┬─────────────┬─┘
│ │ │
│ Serverless NEGs (one per service)
│ │ │
┌─────────────▼───┐ ┌──────▼──────┐ ┌───▼──────────────┐
│ Cloud Run │ │ Cloud Run │ │ Cloud Run │
│ (ui) │ │ (gateway) │ │ (backend) │
│ :3000 │ │ :4000 │ │ :4001 │
└─────────────────┘ └──────┬──────┘ └────────┬─────────┘
│ │
│ Serverless VPC Access connector
┌─── VPC (private services access range) ──────────────────┐
│ │
│ ┌────────────────────────┐ ┌──────────────────┐ │
│ │ Cloud SQL Postgres │ │ Memorystore │ │
│ │ ┌───────┐ ┌───────┐ │ │ Redis │ │
│ │ │writer │ │reader │ │ └──────────────────┘ │
│ │ └───────┘ └───────┘ │ │
│ └────────────────────────┘ ┌──────────────────┐ │
│ │ GCS bucket │ │
│ ┌────────────────────────┐ │ (versioned) │ │
│ │ Secret Manager │ └──────────────────┘ │
│ │ • LITELLM_MASTER_KEY │ │
│ │ • DB password │ ┌──────────────────┐ │
│ │ • user-supplied API │ │ Cloud Run Job: │ │
│ │ keys (referenced) │ │ prisma migrate │ │
│ └────────────────────────┘ │ deploy │ │
│ └──────────────────┘ │
└──────────────────────────────────────────────────────────┘
Images
Both stacks take per-component image references as variables. The defaults
point at the public ghcr.io/berriai/litellm-<component>:main-stable
images, so the stack is runnable end-to-end without pre-flight setup —
pin to a specific tag for production:
-
AWS can pull from any registry the task execution role can reach. The role gets
AmazonECSTaskExecutionRolePolicyattached, which grants ECR pull permissions for repositories in the same account. -
GCP Cloud Run can only pull from Artifact Registry or
gcr.io-style registries. To use images hosted elsewhere, mirror them into Artifact Registry first.
Migrations
LiteLLM's proxy runs prisma migrate deploy at startup, but on first apply
the gateway/backend can race the empty database. Both stacks expose a
one-off migration task that runs python litellm/proxy/prisma_migration.py
against the backend image:
- AWS: an
aws_ecs_task_definition(litellm-migrations). Run withaws ecs run-task— the command is printed interraform output. - GCP: a
google_cloud_run_v2_job(litellm-migrations). Run withgcloud run jobs execute— the command is printed interraform output.
Run the migration job once after the first terraform apply and before the
gateway/backend services start serving traffic.
What's not included
- TLS certificates / custom domains. Both stacks expose plain-HTTP load balancers; bring your own ACM cert (AWS) or managed cert (GCP) and wire it into the LB resource.
- Remote state backends. Default local state — add an
s3orgcsbackend block toversions.tfwhen graduating to a team environment. - Observability beyond the cloud provider's defaults (CloudWatch logs on
AWS, Cloud Logging on GCP). Wire your own Prometheus / Datadog / Langfuse
via the
*_extra_envvariables.