# LiteLLM Terraform stacks

Two self-contained, reusable Terraform **modules** that deploy the
**componentized** LiteLLM proxy — the gateway, backend, and UI as three
independent containers (see `helm/litellm/` for the canonical chart with the
same split).

Each module declares **no `provider` block of its own**, so it can be called
with `count` / `for_each` / `depends_on` and the caller controls region,
assume-role / impersonation, aliases, and `default_tags`. A ready-to-run root
that wires the provider lives at `<stack>/examples/default/` — that's the
one-command deploy path. To embed a stack in your own config, call the module
by source:

```hcl
module "litellm" {
  source = "github.com/BerriAI/litellm//terraform/litellm/aws?ref=<tag>"
  # ... inputs ...
}
```

| Stack  | Compute     | Database (writer + reader)         | Cache       | Object store | Public entrypoint  |
| ------ | ----------- | ---------------------------------- | ----------- | ------------ | ------------------ |
| `aws/` | ECS Fargate | Aurora Postgres (IAM auth)         | ElastiCache | S3           | Application LB     |
| `gcp/` | Cloud Run   | Cloud SQL Postgres (password auth) | Memorystore | GCS          | External HTTPS LB  |

Each stack creates its own VPC and managed data stores — from
`<stack>/examples/default/`, drop in a tfvars file and run `terraform apply`.
Both stacks support a typed `proxy_config` input (mirrors `helm/litellm`'s
`gateway.config.proxy_config`) and per-component extra env vars /
secret-manager refs.

## Components

The proxy is split into three deployables:

| Component | Default image                            | Port | Role                                                                 |
| --------- | ---------------------------------------- | ---- | -------------------------------------------------------------------- |
| `gateway` | `ghcr.io/berriai/litellm-gateway:main-stable` | 4000 | LLM data plane (`/v1/chat/completions`, `/v1/embeddings`, …)         |
| `backend` | `ghcr.io/berriai/litellm-backend:main-stable` | 4001 | Management API (`/key/*`, `/user/*`, `/team/*`, `/model/*`, …)       |
| `ui`      | `ghcr.io/berriai/litellm-ui:main-stable`      | 3000 | Static Next.js dashboard served by nginx                             |

The load balancer routes gateway path prefixes (mirrored verbatim from
`gateway/routes/allowlist.py`) to the gateway, UI asset paths (`/`,
`/litellm-asset-prefix/*`, `/_next/*`, `/favicon.ico`) to the UI, and
everything else to the backend.

## Architecture

### AWS (`terraform/litellm/aws/`)

```
                        ┌───────────────────────────────────────┐
                        │            Public Internet            │
                        └─────────────────┬─────────────────────┘
                                          │ HTTP/80
                          ┌───────────────▼───────────────┐
                          │   Application Load Balancer   │
                          │   (path-routing listener)     │
                          └─┬─────────────┬─────────────┬─┘
                            │             │             │
            UI assets, /    │  /v1/chat,  │   /key/*    │
            /_next/*, …     │  /v1/embed, │   /user/*   │
                            │  …          │   …         │
              ┌─────────────▼───┐  ┌──────▼──────┐  ┌───▼──────────────┐
              │    ECS Service  │  │ ECS Service │  │   ECS Service    │
              │       (ui)      │  │  (gateway)  │  │    (backend)     │
              │   Fargate :3000 │  │ Fargate:4000│  │  Fargate :4001   │
              └─────────────────┘  └──────┬──────┘  └────────┬─────────┘
                                          │                  │
              ┌─── private subnets (one per AZ) ──────────────────────┐
              │                                                       │
              │   ┌────────────────────────┐    ┌────────────────┐   │
              │   │  Aurora Postgres       │    │  ElastiCache   │   │
              │   │  cluster (IAM auth)    │    │  Redis (1 node)│   │
              │   │  ┌───────┐  ┌───────┐  │    └────────────────┘   │
              │   │  │writer │  │reader │  │                         │
              │   │  └───────┘  └───────┘  │    ┌────────────────┐   │
              │   └────────────────────────┘    │  S3 bucket     │   │
              │                                  │  (versioned)   │   │
              │   ┌────────────────────────┐    └────────────────┘   │
              │   │  Secrets Manager       │                         │
              │   │  • LITELLM_MASTER_KEY  │    ┌────────────────┐   │
              │   │  • DB master password  │    │ One-off ECS    │   │
              │   │  • user-supplied API   │    │ task: prisma   │   │
              │   │    keys (referenced)   │    │ migrate deploy │   │
              │   └────────────────────────┘    └────────────────┘   │
              │                                                       │
              └─── VPC ───────────────────────────────────────────────┘
                          │ NAT gateway in one public subnet
                          ▼
                    egress to LLM providers
```

### GCP (`terraform/litellm/gcp/`)

```
                        ┌───────────────────────────────────────┐
                        │            Public Internet            │
                        └─────────────────┬─────────────────────┘
                                          │ HTTP/80
                          ┌───────────────▼───────────────┐
                          │ External HTTPS Load Balancer  │
                          │   (global, URL map routing)   │
                          └─┬─────────────┬─────────────┬─┘
                            │             │             │
                            │ Serverless NEGs (one per service)
                            │             │             │
              ┌─────────────▼───┐  ┌──────▼──────┐  ┌───▼──────────────┐
              │   Cloud Run     │  │  Cloud Run  │  │    Cloud Run     │
              │      (ui)       │  │  (gateway)  │  │    (backend)     │
              │      :3000      │  │   :4000     │  │      :4001       │
              └─────────────────┘  └──────┬──────┘  └────────┬─────────┘
                                          │                  │
                                          │ Serverless VPC Access connector
              ┌─── VPC (private services access range) ──────────────────┐
              │                                                          │
              │   ┌────────────────────────┐    ┌──────────────────┐    │
              │   │  Cloud SQL Postgres    │    │  Memorystore     │    │
              │   │  ┌───────┐  ┌───────┐  │    │  Redis           │    │
              │   │  │writer │  │reader │  │    └──────────────────┘    │
              │   │  └───────┘  └───────┘  │                            │
              │   └────────────────────────┘    ┌──────────────────┐    │
              │                                  │  GCS bucket      │    │
              │   ┌────────────────────────┐    │  (versioned)     │    │
              │   │  Secret Manager        │    └──────────────────┘    │
              │   │  • LITELLM_MASTER_KEY  │                            │
              │   │  • DB password         │    ┌──────────────────┐    │
              │   │  • user-supplied API   │    │ Cloud Run Job:   │    │
              │   │    keys (referenced)   │    │ prisma migrate   │    │
              │   └────────────────────────┘    │ deploy           │    │
              │                                  └──────────────────┘    │
              └──────────────────────────────────────────────────────────┘
```

## Images

Both stacks take per-component image references as variables. The defaults
point at the public `ghcr.io/berriai/litellm-<component>:main-stable`
images, so the stack is runnable end-to-end without pre-flight setup —
pin to a specific tag for production:

- **AWS** can pull from any registry the task execution role can reach.
  The role gets `AmazonECSTaskExecutionRolePolicy` attached, which grants
  ECR pull permissions for repositories in the same account.

- **GCP Cloud Run** can only pull from Artifact Registry or
  `gcr.io`-style registries. To use images hosted elsewhere, mirror them
  into Artifact Registry first.

## Migrations

LiteLLM's proxy runs `prisma migrate deploy` at startup, but on first apply
the gateway/backend can race the empty database. Both stacks expose a
one-off migration task that runs `python litellm/proxy/prisma_migration.py`
against the backend image:

- AWS: an `aws_ecs_task_definition` (`litellm-migrations`). Run with
  `aws ecs run-task` — the command is printed in `terraform output`.
- GCP: a `google_cloud_run_v2_job` (`litellm-migrations`). Run with
  `gcloud run jobs execute` — the command is printed in `terraform output`.

Run the migration job once after the first `terraform apply` and before the
gateway/backend services start serving traffic.

## Feature parity between stacks

The two modules expose the same conceptual surface; concrete inputs differ
only where the underlying cloud forces it.

| Capability                       | AWS input(s)                                            | GCP input(s)                                              |
| -------------------------------- | ------------------------------------------------------- | --------------------------------------------------------- |
| Tenant + env naming              | `tenant`, `env`                                         | `tenant`, `env`                                           |
| Pre-shared master key / license  | `litellm_master_key`, `litellm_license`                 | `litellm_master_key`, `litellm_license`                   |
| UI admin password                | `ui_password`                                           | `ui_password`                                             |
| Per-deployment tags / labels     | `tags` (`map(string)`)                                  | `labels` (`map(string)`)                                  |
| TLS posture                      | `acm_certificate_arn`, `allow_plaintext_alb`            | `lb_domains`, `allow_plaintext_lb`                        |
| Force destroy of object store    | `s3_force_destroy`                                      | `gcs_force_destroy`                                       |
| Database deletion protection     | `skip_final_snapshot`                                   | `cloudsql_deletion_protection`                            |
| `proxy_config` (typed YAML map)  | `proxy_config`                                          | `proxy_config`                                            |
| Coordination Redis               | `REDIS_*` from ElastiCache (automatic)                  | `REDIS_*` from Memorystore (automatic)                    |
| Extra plain env per component    | `gateway_extra_env`, `backend_extra_env`                | `gateway_extra_env`, `backend_extra_env`                  |
| Extra secret-backed env          | `gateway_extra_secrets`, `backend_extra_secrets` (ARNs) | `gateway_extra_secrets`, `backend_extra_secrets` (resource IDs) |
| Uvicorn `--workers` on gateway   | `gateway_num_workers`                                   | `gateway_num_workers`                                     |
| OpenTelemetry v2 (opt-in)        | `otel_endpoint`, `otel_exporter`, `otel_environment_name`, `otel_capture_message_content`, `otel_headers_secret_arn` | `otel_endpoint`, `otel_exporter`, `otel_environment_name`, `otel_capture_message_content`, `otel_headers_secret` |

Each module stamps its own stack-identity tag (`litellm:stack` on AWS,
`litellm-stack` on GCP — GCP label keys forbid colons) plus
`managed-by = "terraform"` onto every taggable / labelable resource and
merges `var.tags` / `var.labels` on top. Provider `default_tags` on AWS
merge on top of all of these.

Coordination Redis needs no input on either cloud. Each module provisions the
managed Redis (ElastiCache on AWS, Memorystore on GCP) and exports `REDIS_HOST`,
`REDIS_PORT` and `REDIS_SSL` (plus `REDIS_SSL_CA_CERTS` on GCP) into the gateway
and backend env. The proxy falls back to those variables to build its
coordination Redis, which backs cross-pod tpm/rpm rate limits, spend tracking
and the pod lock manager. This is independent of LLM response caching, which
stays off unless you enable `litellm_settings.cache` in `proxy_config`.

To coordinate through a Redis the module does not manage, set
`general_settings.coordination_redis` in `var.proxy_config`. An explicit block
overrides the `REDIS_*` env fallback; see the commented example in each
stack's `examples/default/terraform.tfvars.example`

OTel is opt-in on both clouds: leave `otel_endpoint` empty and nothing
OTel-related is added to the container env; set it and both gateway and
backend get `LITELLM_OTEL_V2=true` plus the full `OTEL_*` block, with
`OTEL_SERVICE_NAME` stamped per component
(`<tenant>-litellm-<env>-gateway` and `-backend`). Any `OTEL_*` key set
in `gateway_extra_env` / `backend_extra_env` wins for that service.

## What's not included

- TLS certificates / custom domains. Both stacks expose plain-HTTP load
  balancers; bring your own ACM cert (AWS) or managed cert (GCP) and wire
  it into the LB resource.
- Remote state backends. Default local state — add an `s3` or `gcs`
  backend block to `versions.tf` when graduating to a team environment.
- Observability beyond the cloud provider's defaults (CloudWatch logs on
  AWS, Cloud Logging on GCP). Wire your own Prometheus / Datadog / Langfuse
  via the `*_extra_env` variables, or turn on OTel v2 (see the parity
  table above).

## HCP Terraform no-code (1-click) deploy

Both stacks are publishable as no-code modules in HCP Terraform's private
registry. The end-user flow is: open the no-code launch URL, fill in a
few inputs, hit *Create workspace*, and HCP runs plan/apply against your
cloud account using a variable-set of credentials (static keys or
dynamic-credentials OIDC).

Required overrides the launcher must supply per stack:

- **AWS** (`terraform/litellm/aws`): `region`, `azs`, `tenant`, `env`.
  The image vars (`gateway_image`, `backend_image`, `ui_image`,
  `migrations_image`) can be left at their defaults — the GHCR images
  are anonymous-readable and ECS Fargate pulls them without extra
  credentials.

- **GCP** (`terraform/litellm/gcp`): `project`, `tenant`, `env`, **and
  one of**:
  - `image_registry` pointed at an Artifact Registry **remote** repository
    backed by `https://ghcr.io` (e.g.
    `us-central1-docker.pkg.dev/<project>/litellm/berriai`), so Cloud Run
    pulls the four upstream `litellm-*` images through it; or
  - all four per-component `*_image` URIs pointing at images mirrored
    into a regular Artifact Registry repo.

  The defaults (`ghcr.io/berriai`) cause Cloud Run admission to reject
  the service spec — Cloud Run only authenticates against Artifact
  Registry, `[region.]gcr.io`, or `docker.io`. See
  `terraform/litellm/gcp/README.md#image-pulls` for the
  `gcloud artifacts repositories create … --mode=remote-repository`
  command that sets up the passthrough repo (one-time, per project).

What still requires a manual step regardless of HCP no-code:

- The one-off migration task. The stacks auto-run it via `local-exec`
  during `terraform apply`, but that requires the `aws` / `gcloud` CLI
  on the runner. HCP-hosted runners don't have them; use an HCP agent
  pool with a custom image that includes the relevant CLI, or run the
  command printed in the `migration_run_command` output by hand after
  the first apply.
