mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00

fix(terraform/gcp): prompt for image_registry in DeployStack one-click (#29852 )

* fix(terraform/gcp): prompt for image_registry in DeployStack one-click

The four litellm-* images live on GHCR and Cloud Run rejects ghcr.io URIs
at apply time, so every deploy has to point image_registry at an Artifact
Registry remote repo. The DeployStack installer didn't surface
image_registry as a prompt, so a click-through user landed on the
ghcr.io/berriai default and the apply failed ~20 min in, after Cloud SQL
had already provisioned. Add image_registry to custom_settings with a
PROJECT_ID-placeholder default and a description that flags the ghcr.io
rejection so the failure happens at the prompt, not after billing the
slow path. TUTORIAL.md is reworded to tell the user what to enter at the
new prompt instead of "edit terraform.tfvars before applying".

* fix(terraform/gcp): generalize image_registry default to any region

Per Greptile feedback on #29852, the prior default hardcoded us-central1
and would silently produce a Cloud Run-incompatible image path for any
deployment in another region. The user would substitute PROJECT_ID, miss
the region segment, and reproduce the original late-apply failure. Use
REGION as a second placeholder and tighten the prompt copy so both
substitutions are mandatory.

* fix(terraform/gcp): make destroy work without manual intervention

Three Cloud Run v2 services and the migrations Cloud Run v2 job all
default to deletion_protection=true at the provider level, which has no
data-safety value on stateless resources and blocks terraform destroy
with an error that can only be unstuck with a tfvars edit + apply
roundtrip. Wire deletion_protection=false directly on all four; the
operator-facing tripwire that matters is cloudsql_deletion_protection,
which guards the only resource that actually holds data.

The litellm Cloud SQL database also drops cleanly only if every
connection is closed first. Cloud Run services and the migrations job
hold connections open until they're torn down, so destroy races and
fails with "database is being accessed by other users". Setting
deletion_policy=ABANDON on the database resource lets terraform skip
the explicit drop; the Cloud SQL instance deletion takes the database
with it anyway.

Together these turn destroy into a single command, matching the AWS
stack's behavior.

2026-06-06 20:21:06 +00:00

5.5 KiB

Raw Blame History

Deploy LiteLLM on GCP

This walkthrough provisions the full LiteLLM stack on GCP via Cloud Run, Cloud SQL, Memorystore Redis, and an external HTTPS load balancer. You'll answer a few prompts; DeployStack writes a terraform.tfvars and runs terraform apply against the project you select.

Prerequisites

Pick the GCP project you want to deploy into, then make sure billing is enabled on it. The stack provisions paid resources (Cloud SQL, Memorystore, an LB anycast IP).

Enable required APIs

The stack needs these APIs enabled in the target project. Click to enable, or run the gcloud command below.

gcloud services enable \
  run.googleapis.com \
  sqladmin.googleapis.com \
  redis.googleapis.com \
  secretmanager.googleapis.com \
  vpcaccess.googleapis.com \
  compute.googleapis.com \
  servicenetworking.googleapis.com \
  storage.googleapis.com \
  artifactregistry.googleapis.com

Create the Artifact Registry passthrough to GHCR

Cloud Run only pulls from Artifact Registry, gcr.io, or docker.io; it rejects ghcr.io URIs at apply time. The four LiteLLM images live on GHCR, so the stack needs a remote Artifact Registry repo pointed at GHCR. This is a one-time setup per project.

gcloud artifacts repositories create litellm \
  --repository-format=docker \
  --location=<walkthrough-watcher-constant key="region" default="us-central1"/> \
  --mode=remote-repository \
  --remote-repo-config-desc="GitHub Container Registry passthrough" \
  --remote-docker-repo=https://ghcr.io

If the repo already exists, this command exits with a clear error and you can move on. When deploystack install prompts for image_registry, enter <region>-docker.pkg.dev/<your-project>/litellm/berriai (substituting your region and project). The shipped default contains a PROJECT_ID placeholder that will fail at apply time if left unedited.

(Optional) Set tenant secrets

The stack auto-generates a LITELLM_MASTER_KEY if you don't supply one. If you have an enterprise license or want a pre-chosen master key, export them as TF_VAR_* env vars before running the installer so they end up in Secret Manager but not in terraform.tfvars.

export TF_VAR_litellm_master_key="sk-..."   # optional; auto-generated if omitted
export TF_VAR_litellm_license="lic-..."     # optional; OSS-only without it
export TF_VAR_ui_password="..."             # optional; falls back to master_key for UI login

Skip this step entirely for a trial deploy.

Run the installer

DeployStack will prompt for project, region, tenant, env, image tag, image_registry, and TLS posture, then run terraform apply. Open <walkthrough-editor-open-file filePath="terraform/litellm/gcp/examples/default/deploystack.json">deploystack.json</walkthrough-editor-open-file> if you want to see the prompt definitions first.

deploystack install

The first apply takes 20-25 minutes; most of that is Cloud SQL provisioning. The migration Cloud Run Job runs automatically once the database is ready, and only then do gateway, backend, and UI start.

Grab the LB URL

terraform output lb_url

For trial deploys (allow_plaintext_lb=true), this is http://<lb-ip>. The UI lives at /ui; sign in with username admin and the master key:

gcloud secrets versions access latest \
  --secret="$(terraform output -raw master_key_secret_id)"

Going to TLS

If you picked allow_plaintext_lb=true to bootstrap but want HTTPS for real, point a DNS A record at the LB IP, then re-run terraform with lb_domains set and allow_plaintext_lb removed:

terraform apply \
  -var 'lb_domains=["proxy.example.com"]'

Google-managed certs sit in PROVISIONING for 15-60 minutes after DNS propagates. You can watch the state with gcloud compute ssl-certificates describe <tenant>-litellm-<env>-cert.

Adding provider API keys

Provider keys (OpenAI, Anthropic, etc.) belong in Secret Manager, not in terraform.tfvars. Create the secret first, then reference its resource ID from gateway_extra_secrets and re-apply:

echo -n "sk-proj-..." | gcloud secrets create openai-api-key --data-file=-

Edit terraform.tfvars:

gateway_extra_secrets = {
  OPENAI_API_KEY = "projects/<your-project>/secrets/openai-api-key"
}
proxy_config = {
  model_list = [
    {
      model_name = "gpt-4o"
      litellm_params = {
        model   = "openai/gpt-4o"
        api_key = "os.environ/OPENAI_API_KEY"
      }
    },
  ]
}

Then terraform apply.

Tearing it all down

deploystack uninstall

cloudsql_deletion_protection is true by default; flip it to false in terraform.tfvars and apply before uninstalling if you actually want the DB gone. Same goes for gcs_force_destroy on the bucket.

You're done

Full configuration reference is in <walkthrough-editor-open-file filePath="terraform/litellm/gcp/README.md">README.md</walkthrough-editor-open-file>, and every input variable on the underlying module lives in <walkthrough-editor-open-file filePath="terraform/litellm/gcp/variables.tf">variables.tf</walkthrough-editor-open-file>.

5.5 KiB Raw Blame History