litellm/terraform/litellm/aws

LiteLLM on AWS (ECS Fargate)

Deploys the componentized LiteLLM proxy on AWS:

• VPC with public + private subnets across the AZs you pass in, one NAT gateway (skipped when you pass an existing vpc_id)
• Aurora Postgres cluster — one writer instance + one reader instance, IAM database authentication enabled (skipped when create_database = false)
• ElastiCache Redis (private, replication group with multi-AZ failover and at-rest + in-transit encryption) for caching + rate limiting (skipped when create_redis = false)
• S3 bucket (private, versioned, SSE-S3) — exposed to gateway + backend as S3_BUCKET_NAME / S3_REGION_NAME for cache backend, request log archival, and /v1/files storage
• Secrets Manager entries for LITELLM_MASTER_KEY (auto-generated, sk-…) and the Aurora master password (bootstrap-only)
• ECS Fargate cluster running three services — gateway, backend, ui
• Application Load Balancer (public, HTTP/80) with path-based routing:
  • LLM data-plane prefixes (/v1/chat/*, /v1/embeddings, …) → gateway
  • UI assets (/, /_next/*, /litellm-asset-prefix/*, …) → ui
  • Everything else (management API: /key/*, /user/*, …) → backend
• One-off migration task (litellm-migrations) that runs prisma migrate deploy from the dedicated ghcr.io/berriai/litellm-migrations image

Bring your own networking, database, and Redis

The three infrastructure pieces the stack would otherwise own are each optional, so it can slot into an account where networking and data stores are already provisioned (often by another team, in another Terraform state).

Networking. Set vpc_id plus public_subnet_ids and private_subnet_ids and no VPC, subnet, route table, internet gateway, or NAT gateway is created. The ALB goes in the public subnets, the ECS tasks and any subnet group the stack still needs go in the private ones, and vpc_cidr / azs go unused. The private subnets need their own egress (NAT gateway, or VPC endpoints covering ECR, S3, CloudWatch Logs, and Secrets Manager) since tasks pull images, resolve secrets, and call LLM providers.

Security groups stay module-owned in either mode: the ALB group, the tasks group, and the database/cache groups when it creates those. To let the tasks reach infrastructure the module doesn't manage, either allow inbound from the group named by the task_security_group_id output, or attach a group of your own with additional_task_security_group_ids.

vpc_id             = "vpc-0123456789abcdef0"
public_subnet_ids  = ["subnet-aaa", "subnet-bbb"]
private_subnet_ids = ["subnet-ccc", "subnet-ddd"]

Database and Redis. create_database and create_redis default to true (today's behavior). Set one to false and pass a connection string to use something you already run: the value lands in a Secrets Manager entry and reaches gateway, backend, and the migration task as DATABASE_URL / REDIS_URL, both of which outrank the discrete DATABASE_* / REDIS_* vars in the proxy, so nothing appears in plain text in a task definition.

create_database = false
database_url    = "postgresql://litellm:...@db.internal:5432/litellm"
create_redis    = false
redis_url       = "rediss://:...@cache.internal:6379"

The schema migration still runs on every apply against an existing database; only the Aurora-specific IAM-user bootstrap drops out, since those credentials are already in the URL.

Leaving the URL empty runs without the component entirely:

• No database: no virtual keys, teams, spend tracking, or UI persistence, and STORE_MODEL_IN_DB is not set, so models come from proxy_config. Requests authenticate with LITELLM_MASTER_KEY only.
• No Redis: rate limits, budgets, and router cooldowns are per-task rather than cluster-wide, which is only sane at one task per service.

Aurora + IAM auth

The cluster runs with iam_database_authentication_enabled = true. Enabling that on the cluster doesn't by itself let any Postgres user log in with an IAM token — you also need to CREATE USER ... GRANT rds_iam once. bootstrap.tf does this automatically during terraform apply via a one-shot Fargate task (postgres:16-alpine running the bootstrap SQL with the master password from Secrets Manager). The SQL is idempotent, so re-applies are safe.

The same apply also runs the prisma schema migration via the existing litellm-migrations task definition, and the gateway/backend services depends_on the migration so they don't start until the schema is in place.

At runtime, the proxy assembles DATABASE_URL from DATABASE_HOST/PORT/USER/NAME plus a short-lived IAM token — see litellm/proxy/auth/rds_iam_token.py. The task role has rds-db:connect scoped to the IAM-authed user on the cluster.

Break-glass. If you need to run the bootstrap or migration by hand (e.g., to re-apply against an externally provisioned cluster), db_bootstrap_sql and migration_run_command are still exposed as outputs.

Prerequisite. terraform apply shells out to aws ecs run-task / aws ecs wait in local-exec provisioners, so the machine running terraform needs the aws CLI installed and authenticated.

Configuring the proxy

proxy_config (preferred)

Mirrors the helm chart's gateway.config.proxy_config. The map is YAML-encoded and uploaded to S3 (config/litellm-config.yaml in the stack's bucket); the gateway and backend container entrypoints download it to /tmp/litellm-config.yaml at task start via boto3 and set CONFIG_FILE_PATH to match. The S3 object's etag is wired into the task definition, so editing proxy_config produces a new task-def revision and a rolling redeploy of both services.

proxy_config = {
  model_list = [
    {
      model_name = "gpt-4o"
      litellm_params = {
        model   = "openai/gpt-4o"
        api_key = "os.environ/OPENAI_API_KEY"
      }
    },
  ]
  general_settings = {
    master_key   = "os.environ/LITELLM_MASTER_KEY"
    database_url = "os.environ/DATABASE_URL"
  }
}

LiteLLM resolves os.environ/<NAME> references in the YAML against the container's environment. That means provider API keys belong in *_extra_secrets (next section), and your YAML just references them by name.

Extra env vars

Non-sensitive plaintext (feature flags, observability hosts, etc.):

gateway_extra_env = {
  LANGFUSE_HOST = "https://us.cloud.langfuse.com"
}
backend_extra_env = {
  STORE_MODEL_IN_DB = "True"
}

Extra secrets (API keys)

Sensitive values — provider API keys, third-party tokens — live in existing Secrets Manager secrets. Reference them by ARN:

gateway_extra_secrets = {
  OPENAI_API_KEY    = "arn:aws:secretsmanager:us-west-2:111122223333:secret:openai-api-key-AbCdEf"
  ANTHROPIC_API_KEY = "arn:aws:secretsmanager:us-west-2:111122223333:secret:anthropic-api-key-GhIjKl"
}

What happens under the hood:

• The execution role auto-gains secretsmanager:GetSecretValue on every ARN listed here.
• ECS resolves each secret at task launch and injects its value into the container as the env var named on the left.
• The proxy_config YAML references the resulting env var via os.environ/OPENAI_API_KEY.

To pluck a single field out of a JSON secret, use ECS's :fieldName:: suffix:

gateway_extra_secrets = {
  OPENAI_API_KEY = "arn:…:secret:provider-keys-AbCdEf:openai_api_key::"
}

To create the secret beforehand:

aws secretsmanager create-secret \
  --name openai-api-key \
  --secret-string "sk-proj-..."

Observability (OpenTelemetry v2)

OTel v2 (https://docs.litellm.ai/docs/observability/opentelemetry_v2) is opt-in and gated entirely on otel_endpoint. Empty (default) and nothing OTel-related is added to the container env. Set it and both gateway and backend gain LITELLM_OTEL_V2=true plus the OTEL_* block, with OTEL_SERVICE_NAME stamped per component (${tenant}-litellm-${env}-gateway and -backend) so spans land tagged with the right hop. Any OTEL_* key set in gateway_extra_env / backend_extra_env overrides the default for that service.

otel_endpoint         = "http://otel-collector.internal:4318"
otel_exporter         = "otlp_http"   # otlp_grpc, console
otel_environment_name = "prod"        # defaults to var.env

For collectors that require an auth header, store the comma-separated key=value string in Secrets Manager and reference it via otel_headers_secret_arn. The execution role auto-gains secretsmanager:GetSecretValue on that ARN.

otel_headers_secret_arn = "arn:aws:secretsmanager:us-west-2:111122223333:secret:honeycomb-otel-headers-AbCdEf"

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT defaults to no_content; flip otel_capture_message_content = "prompt_and_completion" only after auditing what lands in the backend, since prompts and completions are typically sensitive.

Vendor presets (Arize, Phoenix, Langfuse OTel, Weave, Langtrace, Levo, AgentOps) live under proxy_config.litellm_settings.callbacks and are orthogonal to the OTLP variables above; their credentials still go in *_extra_secrets.

Enterprise billing metrics

License-gated request metering is opt-in and gated entirely on billing_metrics_endpoint. Empty (default) and no billing env is added to the container, so existing deployments are unchanged. Set it and both gateway and backend export billable-request counts over OTLP/HTTP, authenticating to the collector with the mTLS client certificate issued for your deployment.

The proxy accepts the certificate, key, and CA bundle as either a file path or literal PEM content. This stack takes the PEM, writes each one to its own Secrets Manager entry, grants the task-execution role secretsmanager:GetSecretValue on them, and injects them as LITELLM_BILLING_METRICS_CLIENT_CERT / _CLIENT_KEY (and _CA_CERT when set), so no volume mount is needed on Fargate.

billing_metrics_endpoint = "https://telemetry.litellm.ai/v1/metrics"

export TF_VAR_billing_metrics_client_cert_pem="$(cat client.crt)"
export TF_VAR_billing_metrics_client_key_pem="$(cat client.key)"

billing_metrics_ca_cert_pem is only for private or test collectors whose CA is not in the system trust store; leave it empty against telemetry.litellm.ai. Metering requires an enterprise license, so pair this with litellm_license. To tune the export cadence, set LITELLM_BILLING_METRICS_EXPORT_INTERVAL_MS through gateway_extra_env / backend_extra_env

Prometheus metrics sidecar

gateway_metrics_port adds a metrics sidecar (python -m litellm.proxy.prometheus_metrics_server) to the gateway task that aggregates the workers' samples over a shared task volume, so a scrape never runs on an inference worker. The ALB never routes to that port and the tasks security group only opens it to gateway_metrics_scrape_cidrs. Needs gateway_image v1.101.0 or newer. See Prometheus metrics for the metrics themselves.

gateway_metrics_port         = 4001
gateway_metrics_scrape_cidrs = ["10.0.0.0/16"]

In-container connection pool

Each of the gateway_num_workers uvicorn workers opens its own Prisma pool straight to Postgres, so one task holds workers x connection_limit connections and the fleet's footprint against the database ceiling grows with every task. gateway_connection_pool_enabled runs a PgBouncer (transaction mode, loopback) inside the gateway container that all workers share, capping the task at gateway_pool_max_db_connections upstream connections however many workers it runs. gateway_pool_max_client_conn bounds the worker-side connections the pooler accepts. The module sets LITELLM_PGBOUNCER_ENABLED, LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS and LITELLM_PGBOUNCER_MAX_CLIENT_CONN on the gateway container only; the backend and the migration task keep the direct connection.

gateway_num_workers             = 4
gateway_connection_pool_enabled = true
gateway_pool_max_db_connections = 20
gateway_pool_max_client_conn    = 1000

The pool works with the module-created Aurora as well as an existing database via database_url. Against Aurora it authenticates with the same rotating IAM tokens the workers used to (see Aurora + IAM auth): the pooler mints a token from the task role, renews it before it expires and hands the workers a loopback URL with a static password instead

The componentized gateway_image starts through python -m gateway.launch, which reads these variables, starts the pooler once per task and hands the workers its loopback URL; the classic litellm image honours them the same way.

Scaling the gateway on requests and tokens

By default the gateway service target-tracks CPU (gateway_cpu_target) and memory (gateway_memory_target). Two more targets add workload signals next to them. Application Auto Scaling evaluates every attached policy and follows the one asking for the most tasks, so the resource policies keep working as a floor while requests or tokens drive scale-out

Both targets are per task per second, the way load is usually quoted (1k rps, 75M tok/s). CloudWatch is the limit on how fast they react: target tracking evaluates every metric, predefined or custom, aggregated over 60-second periods and has no period setting, so ECS reacts on a roughly one-minute cadence whatever unit the variable is written in. The Kubernetes charts get a faster signal because the Prometheus rate() window and scrape interval are theirs to shorten

gateway_target_requests_per_second adds an ALBRequestCountPerTarget policy on the gateway target group. The ALB publishes that metric as requests per minute per registered task, so the policy's target value is 60 times the variable: 90 rps becomes a target of 5,400 per minute. No agent or sidecar is needed

gateway_target_tokens_per_second adds a metric-math policy over a CloudWatch metric of the gateway's litellm_total_tokens_metric_total counter and the service's RunningTaskCount from Container Insights. Nothing native to ECS carries token throughput, so you publish that metric yourself with the CloudWatch agent's Prometheus scraper pointed at the metrics sidecar above. The agent emits the delta of a counter between scrapes, so Sum over the 60-second period is the tokens served in that minute; the expression divides by 60 (tokens_per_second) and then by the task count (tokens_per_second_per_task). Tokens are counted when a response completes, so long streams show up late in this signal. gateway_tokens_metric tells the policy where the agent publishes: the namespace, the metric name (defaults to the counter name) and the dimensions from your metric_declaration

gateway_metrics_port               = 4001
gateway_target_requests_per_second = 90
gateway_target_tokens_per_second   = 6000000
gateway_tokens_metric = {
  namespace  = "LiteLLM/Prometheus"
  dimensions = { ClusterName = "acme-litellm-prod", TaskDefinitionFamily = "acme-litellm-prod-gateway" }
}

Worked example for the request policy: 1,000 rps across 10 tasks is 100 rps per task (the ALB reports it as 6,000 per minute per target) against a target of 90 (5,400), so target tracking sizes the service to ceil(10 * 100 / 90) = 12 tasks. The token policy does the same arithmetic: ten tasks handle 4,200,000,000 tokens in a minute, tokens / 60 is 70,000,000 tokens per second and tokens_per_second / running_tasks is 7,000,000 against a target of 6,000,000, so the service grows to ceil(10 * 7000000 / 6000000) = 12. Container Insights must be enabled on the cluster for RunningTaskCount to exist

Collector sidecar

collector_enabled = true adds a second container to the gateway task that runs python -m litellm.proxy.collector from the gateway image, and sets LITELLM_COLLECTOR_ENABLED=true on the gateway so its uvicorn workers ship spend events (SpendLogs writes, key/team/user spend updates, budget alerts) to the sidecar instead of running that pipeline in the request path. This is the Terraform counterpart of helm's gateway.collector. The default (false) leaves the task definition exactly as before.

Fargate tasks share one network namespace, so the sidecar listens on loopback TCP (tcp://127.0.0.1:${collector_port}, default 4010) instead of the Unix socket helm uses; the proxy rejects any non-loopback address. The sidecar gets the same database, Redis, master-key, license, proxy config, and gateway_extra_env / gateway_extra_secrets values as the gateway container, runs with LITELLM_JOB_ROLE=collector, and is non-essential with an ECS restart policy, so a sidecar crash restarts it in place while the gateway falls back to in-process spend tracking.

collector_enabled = true
# collector_cpu               = 512    # carved out of gateway_cpu
# collector_memory            = 2048   # MiB, carved out of gateway_memory
# collector_buffer_size       = 1000
# collector_on_unavailable    = "fallback"  # or "drop"
# collector_drain_timeout_seconds = 10

Both sidecar reservations must leave room for the gateway container inside gateway_cpu / gateway_memory (the plan fails otherwise). Service autoscaling keeps tracking the whole task's CPU and memory, sidecar included

Tenant deployment

Every resource the stack creates is named ${tenant}-litellm-${env} (or that plus a per-resource suffix), so multiple tenants and multiple environments coexist in the same account as long as the (tenant, env) pair differs:

tenant	env	Example resource name
acme	stage	acme-litellm-stage-gateway
acme	prod	acme-litellm-prod-master-key
globex	dev	globex-litellm-dev-license

For a per-tenant instance via the example root, the only inputs that change are the tenant slug, env, and the two pre-issued secrets:

cd terraform/litellm/aws/examples/default
export TF_VAR_litellm_master_key="sk-..."   # the tenant's master key
export TF_VAR_litellm_license="lic-..."     # their LITELLM_LICENSE

terraform apply \
  -var "region=us-west-2" \
  -var 'azs=["us-west-2a","us-west-2b"]' \
  -var "tenant=acme" \
  -var "env=stage"

To run many tenants from a single config, call the module with for_each instead of one root per tenant (see "Using as a module"):

module "litellm" {
  for_each = toset(["acme", "globex"])
  source   = "github.com/BerriAI/litellm//terraform/litellm/aws?ref=<tag>"
  tenant   = each.key
  env      = "prod"
  region   = "us-west-2"
  azs      = ["us-west-2a", "us-west-2b"]
}

(This for_each form is only possible because the module declares no provider block — the original root-with-provider layout forbade it.)

Both litellm_master_key and litellm_license are optional:

• Omit litellm_master_key → the stack auto-generates a random sk-… value (trial/dev path).
• Omit litellm_license → no license secret is created and gateway/ backend run without LITELLM_LICENSE (OSS-only).

Use TF_VAR_* env vars rather than tfvars files for these — values written to a tfvars file end up in terraform.tfstate and any committed example files.

Quick start

cd terraform/litellm/aws/examples/default
cp terraform.tfvars.example terraform.tfvars
# Edit: region, tenant, env, azs, proxy_config, gateway_extra_secrets.

terraform init
terraform apply

examples/default/ is a thin root that configures the aws provider and calls the module (../../). It exposes a curated variable surface; for advanced knobs (per-component CPU/memory/workers, autoscaling, RDS/Redis sizing, per-component image pins) set them on the module "litellm" block in examples/default/main.tf, or call the module from your own config — see "Using as a module" below.

That single apply provisions everything, runs the DB user bootstrap, runs the schema migration, and only then starts the gateway/backend services. When it returns, the stack is serving traffic.

terraform output alb_url
# UI login: admin / <master key>
aws secretsmanager get-secret-value \
  --secret-id "$(terraform output -raw master_key_secret_arn)" \
  --query SecretString --output text

Using as a module

The directory itself is a module with no provider block — the caller owns provider config. That means you can call it directly with for_each (many tenants from one config), count (conditional stacks), depends_on, an assume-role / aliased provider, etc.:

provider "aws" {
  region = "us-west-2"
  assume_role { role_arn = "arn:aws:iam::111122223333:role/deployer" }
}

module "litellm" {
  source = "github.com/BerriAI/litellm//terraform/litellm/aws?ref=<tag>"

  region = "us-west-2"
  tenant = "acme"
  env    = "prod"
  azs    = ["us-west-2a", "us-west-2b"]
  # ...any of the inputs in variables.tf...
}

Tags: the module threads its own litellm:stack / managed-by / var.tags onto every taggable resource. Any default_tags on your provider merge on top — set org-wide tags there, per-deployment tags via the tags input.

Image pulls

The defaults pull from ghcr.io/berriai/litellm-<component>:v1.86.0-dev, which is anonymous-readable. There are four images: litellm-gateway, litellm-backend, litellm-ui, and litellm-migrations (slim image used only by the one-off migration task — runs prisma migrate deploy against the writer DB and exits). Bump them together when bumping LiteLLM. To pull from a private registry:

• ECR (same account): the execution role already has AmazonECSTaskExecutionRolePolicy, which grants ECR pull for repos in the same account. No extra config needed.
• ECR (cross-account): attach a policy to the execution role allowing ecr:GetAuthorizationToken + ecr:BatchGetImage on the foreign repo ARNs.
• Other private registries (GHCR with a PAT, Docker Hub, …): create a secret holding {"auths":{"<registry>":{"auth":"<base64-user:token>"}}} in Secrets Manager and set repositoryCredentials.credentialsParameter on the task def container — extend ecs.tf accordingly.

TLS

terraform plan refuses to provision an HTTP-only ALB by default — TLS is the supported posture. Two paths:

Production / staging — provide an ACM certificate:

1. Create or import an ACM cert in var.region covering the DNS name you plan to point at the ALB.
2. Set acm_certificate_arn = "arn:aws:acm:..." in tfvars and apply.

Result: a 443 listener carries the path-routing rules; the 80 listener serves a permanent 301 redirect to HTTPS, so HTTP clients are automatically upgraded.

Trial / dev — explicitly opt into HTTP-only:

Set allow_plaintext_alb = true in tfvars. Without this flag, plan fails with a clear error pointing at the precondition. Intended for short-lived trial / dev stacks only.

Storage and database retention

Two opt-in tripwires guard against accidental data loss on terraform destroy:

• skip_final_snapshot (Aurora; default false) — destroying the cluster takes a <cluster>-final-<short-sha> snapshot first.
• s3_force_destroy (S3 bucket holding request log archives, /v1/files content, and the S3 cache backend; default false) — terraform destroy against a non-empty bucket fails.

Neither applies to a database you brought yourself: its lifecycle stays with whoever provisioned it, and terraform destroy leaves it alone.

Flip either to true only for ephemeral / CI stacks where you accept losing the contents.

Files

File	What's in it
versions.tf	Terraform + required_providers constraints (module declares no provider config)
examples/default/	Thin root: aws provider (with an optional default_tags slot for org-wide tags) + a call to the module. The one-command deploy path.
variables.tf	All input variables
locals.tf	Path-prefix lists for ALB routing (mirror of helm/.../ingress.yaml)
network.tf	VPC, subnets, IGW, NAT, route tables (all optional), security groups
secrets.tf	Secrets Manager entries + random passwords
rds.tf	Aurora Postgres cluster + writer / reader instances
redis.tf	ElastiCache Redis
s3.tf	S3 bucket + task-role policy scoped to it
iam.tf	Task execution + task roles, including rds-db:connect
ecs.tf	ECS cluster, task definitions, services for the three components
alb.tf	ALB, listener, target groups, path-routing rules
migrations.tf	One-off migration task definition
outputs.tf	DNS name, secret ARN, bootstrap SQL, migration run-task command