mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yuneng Jiang
|
b5823d5894
|
feat(terraform): sync provider 0.3.0 from mirror and cut 0.4.0
The provider's release gate in project-releaser publishes only when the topmost released heading in terraform/provider/CHANGELOG.md moves past the tag the mirror already carries. That heading has been 0.2.2 since 2026-05-13, so every stable release since has correctly decided there was nothing to publish and the registry has gone stale. Two things were blocking a release: 1. The mirror shipped 0.3.0 out-of-band on 2026-07-13 (pricing_base_model, BerriAI/terraform-provider-litellm#47) after the source move, so that code exists only in the mirror. The publish rsyncs monorepo -> mirror with --delete, so publishing without this port would have deleted a released feature from the registry. 2. Nothing here declared a new version. Port #47 verbatim (resource_model.go and resource_model_crud.go are now byte-identical to the mirror's released files), backfill the 0.3.0 changelog entry it shipped under, and cut 0.4.0 covering the changes made here since the source move. 0.3.0 is not reusable as the next version -- the mirror holds that tag and the publish workflow's tag guard rejects it. |
||
|
Yuneng Jiang
|
a5b6177226
|
chore(deps): bump grpc and golang.org/x modules in the terraform provider
The vendored provider pinned google.golang.org/grpc v1.79.2 alongside a set of golang.org/x modules that govulncheck reports as reachable from plugin.Serve. Raising grpc to v1.82.1 and golang.org/x/text to v0.39.0 pulls the remainder up through minimal version selection and leaves govulncheck reporting no findings Only go.mod and go.sum move here, no provider source is touched. gofmt, go vet, go build and go test all pass at the new versions |
||
|
Yassin Kortam
|
33eda22386
|
fix(docker): honor USE_DDTRACE in the componentized gateway and backend images (#35490)
The componentized images exec uvicorn directly, so ddtrace-run never wraps the interpreter. USE_DDTRACE is not inert there; the proxy lifespan still runs patch_all and litellm's own manual spans still emit. What never gets installed is ddtrace's ASGI TraceMiddleware: starlette builds its middleware stack lazily on the first __call__, which is the lifespan scope, so patching from inside the lifespan body is already too late and no root request span is ever created. Route both entrypoints through a shared docker/component_entrypoint.sh that mirrors the monolith's prod_entrypoint.sh contract, including the DD_TRACE_OPENAI_ENABLED=False export that keeps ddtrace's openai integration from double-reporting calls litellm instruments itself. Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com> |
||
|
Yassin Kortam
|
a643dd0820
|
feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592)
* feat(proxy): push-based OTLP billable-request metering for enterprise deployments
Adds opt-in, license-gated metering that counts 2xx HTTP requests to LLM
inference, MCP, and A2A endpoints and exports them over mutual TLS to a global
OpenTelemetry Collector for request-based billing.
A pure ASGI middleware (BillableRequestMetricsMiddleware) classifies each
request by route and records one count per 2xx response via an injected
recorder. The recorder (BillingMetricsRecorder) owns a dedicated OTEL meter
provider and an OTLP/gRPC exporter authenticated with client certificates, kept
isolated from the global meter provider so a customer's own OTEL metrics are
untouched. The recorder is built only when a valid LITELLM_LICENSE is present
and the cert material is configured; otherwise the middleware is a transparent
pass-through.
Deployment identity rides on the mTLS client certificate rather than the
payload, so the secret license key is never sent as an attribute or header; only
the license org id travels as a resource attribute for cross-checking.
Resolves LIT-4089
* fix(proxy): align billable-request metering with the global collector
- switch the exporter to OTLP/HTTP with a TLS client certificate. The
collector front end terminates mutual TLS and validates the client cert
against our CA; server verification uses the system trust store, so the
CA env var is now an optional override for private collectors
- resolve the metrics recorder on the first request via a factory instead
of at import time, so deployments that provide the license and cert env
vars through the YAML config's environment_variables export correctly
- close the metering bypass: classify /images/edits, /images/variations,
/v1/messages, /v1/videos, video remix, /v1/ocr and Gemini generateContent
as billable, and gate LLM routes to POST so GET reads (list videos, fetch
a response) do not bill. Verified live: the collector count matches the
UI usage page successful_requests exactly, with failures excluded on both
sides
* fix(proxy): wrap enterprise billing import in try-except per code-quality gate
The check_unsafe_enterprise_import gate requires every import from an
enterprise-pathed module to be guarded. Annotate the factory with the
middleware's BillingRecorder protocol so no enterprise type import is
needed at type-check time
* chore: satisfy strict lint gates in billing modules
- builtin generics per UP006 (dict/tuple instead of typing.Dict/Tuple)
- noqa the deliberate blind catch that keeps metering from breaking startup
- sort proxy_server import blocks split by the guarded enterprise import
* fix(proxy): bill provider passthrough, search, and rag routes
Route-inventory audit against LiteLLMRoutes.llm_api_routes found more
SpendLogs-producing surfaces the classifier missed: provider passthrough
(/bedrock, /vertex-ai, /cohere and the rest of mapped_pass_through_routes),
/v1/search and vector-store search, and the rag ingest/query routes. All are
counted by the dashboard usage page, so missing them undercounts billing.
The passthrough prefix list is read from LiteLLMRoutes so new providers are
picked up without touching this module. /langfuse is excluded: it forwards
observability traffic and writes no SpendLogs row. Known limitation recorded
in the PR: /v1/realtime is a websocket flow the HTTP middleware does not see
* fix(proxy): bill MCP and A2A requests by protocol transport routes only
The billable-request classifier matched the whole /v1/mcp prefix, so
management and discovery reads such as GET /v1/mcp/tools and GET
/v1/mcp/server counted as billable MCP requests, while real MCP tool
calls on the /{server}/mcp and /toolset/{name}/mcp aliases were missed
because their route handlers rewrite the ASGI scope only after this
middleware has already classified the original path. Classify MCP by the
concrete transport surface (the /mcp streamable-HTTP and SSE sub-app plus
the single-segment server and toolset aliases) and exclude the /v1/mcp
management API. Apply the same shape to A2A, which had the identical
issue: only the /message/send invoke route bills, not /v1/a2a/discover or
the .well-known agent-card reads.
* fix(proxy): harden billable-request classification and recorder lifecycle
Exact-match Anthropic /v1/messages so OpenAI Assistants thread-message
routes no longer bill, add Google Interactions create routes, guard
recorder.record() so a broken exporter can never fail a served request,
lock lazy recorder resolution against concurrent first requests, and
disable metering on empty-string env config instead of accepting a
blank endpoint
* chore(ui): regenerate eslint metrics after staging merge
* docs(proxy): state the lower-bound billing contract in middleware comments
* fix(proxy): bill mcp-rest tool calls and bare a2a agent invokes
POST /mcp-rest/tools/call executes a tool and fires the same MCP spend
logging as the /mcp transport, and POST /a2a/{agent_id} is the JSON-RPC
invoke route whose method (message/send or message/stream) travels in
the body; both returned 2xx without being recorded
* fix(proxy): flush billable-request counts on proxy shutdown
PeriodicExportingMetricReader buffers up to one export interval of
counts; without a final flush every restart silently dropped them. The
factory registers the recorder it builds and proxy_shutdown_event pops
and flushes it, bounded by a 5s timeout so a dead collector cannot
stall shutdown
* fix(proxy): stop billing bare a2a task RPCs and close the shutdown race
POST /a2a/{agent_id} multiplexes JSON-RPC methods off the request body. Only
message/send and message/stream write a SpendLogs row; tasks/get, tasks/cancel
and the pushNotificationConfig RPCs are forwarded upstream and write none.
Classifying the bare path as billable counted those task RPCs and pushed the
metric above the dashboard's successful-request count. Since a path-only
classifier cannot read the body, the bare route no longer bills; the explicit
/message/send routes still do. Missing a bare-path invoke undercounts, which is
the only direction this metric is allowed to drift. The /mcp transport keeps
billing every method because its list path logs a SpendLogs row too.
The billing middleware also sat outside InFlightRequestsMiddleware, and it
records after the inner app returns. A request could therefore be counted as
drained while its record() had not yet run, letting proxy_shutdown_event flush
and stop the exporter underneath it. Registering it before the in-flight
tracker nests it inside, so wait_for_drain covers the record
* test(proxy): stub the OTLP exporter in the recorder-build test
test_premium_with_full_config_builds_recorder built a real MeterProvider, so
the shutdown flush resolved collector.example and opened a TLS connection from
a unit test. The exporter is now stubbed, and a getaddrinfo spy asserts nothing
resolves the collector host so the stub cannot be quietly dropped later
* fix(helm): truncate the helm.sh/chart label to 63 bytes
Kubernetes caps a label value at 63 bytes and .Chart.Version is unbounded. CI
publishes branch builds as 0.0.0-branch-<branch>-<sha>, so helm.sh/chart
rendered as a 64 byte value and the API server rejected every labeled resource
with "must be no more than 63 bytes", including the migrations Job. The
litellm-helm chart already guards this through a litellm.chart helper; this
adds the same helper here.
Swept the rest of the chart for label and name values built from unbounded
input. .Chart.Version appeared only in this label. The remaining candidates all
derive from .Release.Name, which helm itself caps at 53 characters, so they
cannot overflow; three of them are selector labels feeding immutable Deployment
matchLabels, where adding trunc would risk churn for no gain. They are left
alone deliberately.
Verified with a new helm-unittest suite, tests/chart_label_tests.yaml, which
overrides chart.version per test:
helm unittest -f 'tests/*.yaml' helm/litellm # 13 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
The truncation cases fail against the previous helper. Reproduced the original
overflow by rendering with the real branch version and measuring the label:
helm template rel helm/litellm -f helm/litellm/tests/values/required.yaml \
| grep helm.sh/chart # 64 bytes before, 63 after
* feat(proxy): accept inline PEM for the billing-metrics mTLS credentials
LITELLM_BILLING_METRICS_CLIENT_CERT, _CLIENT_KEY and _CA_CERT took a filesystem
path. ECS injects Secrets Manager values as environment content and cannot mount
them as files, so a licensed deployment there could not turn metering on.
Each variable now takes either a path or the PEM itself. Inline PEM, detected by
the "-----BEGIN" prefix, is written once when the recorder is built into a 0700
temp dir as a 0600 file, and the config points at that path. The OTLP exporter
still only ever sees paths. A write failure disables metering through the
existing failure-as-None path rather than raising, and path-valued variables are
passed through untouched, so nothing changes for deployments that mount files.
The mixed case works too: mount the CA, inject the client credentials
* feat(helm): add first-class billingMetrics values to the componentized chart
Turning enterprise billable-request metering on meant hand-rolling the env vars
and the cert volume through gateway.extraEnv and gateway.volumes. This adds a
top-level billingMetrics block, off by default, consumed only by the gateway
since that is the component serving billable traffic.
When enabled it renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths
and mounts secretName read-only at /etc/litellm/billing-mtls. caSecretName is
optional and only needed for private collectors whose server certificate is not
on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and
adds the CA env var. exportIntervalMs is passed through only when set.
Enabling without secretName or with an empty endpoint fails the render with a
named message rather than producing a gateway that silently never exports.
The generic gateway.volumes, gateway.volumeMounts and gateway.extraEnv paths are
untouched and still compose with this, so existing overlays keep working.
The chart has no values.schema.json and no README, so there is nothing further to
update. Verified with a new helm-unittest suite:
helm unittest -f 'tests/*.yaml' helm/litellm # 23 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
* feat(terraform): billing-metrics variables for the aws and gcp templates
* feat(helm): add billingMetrics values to the classic chart
The componentized chart just gained a first-class billingMetrics block; this
mirrors it in litellm-helm so enabling enterprise billable-request metering no
longer means hand-rolling the env vars and the cert volume through envVars and
volumes.
When enabled the proxy Deployment renders LITELLM_BILLING_METRICS_ENDPOINT plus
the two cert paths, and mounts secretName read-only at /etc/litellm/billing-mtls.
secretName defaults to litellm-billing-metrics-mtls, the conventional name, so
enabling the block is enough once that Secret exists. caSecretName is optional
and only needed for private collectors whose server certificate is not on the
public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the
CA env var. exportIntervalMs is passed through only when set.
The env entries render after envVars and extraEnvVars, so a user-supplied
LITELLM_BILLING_METRICS_ENDPOINT cannot silently redirect the export under
Kubernetes last-wins duplicate-env semantics; this is the same ordering the
migrations Job relies on for DISABLE_SCHEMA_UPDATE.
Enabling with an emptied secretName or endpoint fails the render with a named
message rather than producing a proxy that silently never exports.
The generic volumes, volumeMounts, envVars and extraEnvVars paths are untouched
and still compose with this, so existing overlays keep working. The chart has no
values.schema.json; README parameters and a setup section are updated.
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 68 passed (54 + 14 new)
helm lint helm/litellm-helm # 0 failed
* test(helm): pin that the migrations job never mounts the billing cert
The componentized chart's suite asserts the backend Deployment stays clear of the
billing wiring, since only the gateway serves billable traffic. The classic chart
has no backend, but it does have a second pod: the migrations Job, which renders
its own env from envVars and extraEnvVars. Nothing today wires the billing
include into it, and nothing stopped a future edit from doing so.
Asserts absence of the env, and that the Job grows no volumes or volumeMounts at
all. Both are notExists rather than notContains because the Job renders neither
key by default, so a notContains would fail on an unknown path instead of
checking the absence it looks like it is checking.
* fix(helm): meter the backend too, it serves the MCP transport
Scoping billingMetrics to the gateway was wrong. Applying each component's own
route allowlist to the proxy app shows the split is 75 billable routes on the
gateway and one on the backend: /{mcp_server_name}/mcp, the named-server MCP
transport, which writes a SpendLogs row on success. Metering only the gateway
would have silently dropped every MCP transport call from the counter, an
undercount proportional to a customer's MCP traffic.
The backend deployment now renders the same env and mounts the same read-only
cert secret. The migrations job still gets neither; it runs prisma and serves no
traffic, and a test pins that.
helm unittest -f 'tests/*.yaml' helm/litellm # 25 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
This also aligns the chart with the terraform templates, which inject the
credentials into both components.
* fix(proxy): never log billing credential values when they fail to resolve
Accepting inline PEM turned the cert env vars into secret-bearing values, but
the disable warning still echoed them. A value that is neither a readable path
nor `-----BEGIN`-prefixed PEM, for example a key with a preamble or a malformed
secret, fell through to the path branch and was written to the proxy logs
verbatim, exposing the client certificate or private key to anyone who can read
them.
The warning now names the offending environment variables and tells the operator
what a valid value looks like, without ever printing one
* Revert "fix(helm): truncate the helm.sh/chart label to 63 bytes"
This reverts commit
|
||
|
Yassin Kortam
|
4e3c437631
|
feat(deploy): make coordination redis a first-class chart and terraform surface (#32662)
Render a general_settings.coordination_redis block into the litellm-helm proxy config when the bundled Redis is enabled, gated on a new redis.coordination.enabled value and skipped when the user already supplies their own block. Sentinel deployments render sentinel_nodes and service_name rather than a host/port pair. Also fixes litellm.redis.serviceName, which gated its sentinel branch on standalone architecture. The bundled Redis subchart only serves sentinel in replication mode, and renders no master Service there, so REDIS_HOST pointed at a Service that never existed for every sentinel user. Documents the coordination redis in the componentized chart and in the terraform modules, whose existing REDIS_* exports now feed it directly. Adds helm-unittest coverage for both charts' redis wiring, which had none |
||
|
Yassin Kortam
|
ce2582e9d0
|
feat(terraform): vendor terraform-provider-litellm as source of truth with endpoint drift CI (#32241)
* feat(terraform): vendor terraform-provider-litellm as source of truth with endpoint drift CI * fix(terraform): address review feedback on vendored provider Replace deprecated io/ioutil with io. Remove the unused org/team CRUD client methods so the endpoint audit only tracks live call sites (54 -> 46). Redact request/response logs by parsing the JSON and recursively masking sensitive fields, which fixes the nested-object leak in the old credential_values regex, with a regex fallback for non-JSON payloads; covered by new unit tests. Docs: stop showing api_key inside vector store litellm_params and document that Sensitive attributes still persist in plaintext state, recommending litellm_credential_name and an encrypted state backend. * fix(terraform): stop persisting server-returned litellm_params into vector store state The vector store Read wrote litellm_params straight back from the API response into state. The proxy redacts secrets in those responses, so the readback overwrote user config with redaction sentinels and caused perpetual diffs, and against a server that returns raw values it would persist secrets into a non-Sensitive attribute. Read now preserves the config value like the credential and model resources do, litellm_params is marked Sensitive, and a regression test pins that a server-returned api_key never lands in state * fix(terraform): send role on team member update and stop persisting server env into MCP state The team member update payload omitted role, and the proxy leaves role unchanged when the field is absent, so a role downgrade reported as applied by Terraform never took effect on the proxy. The update now always sends the configured role (the attribute is Required). The MCP server resource wrote env straight back from API responses into a non-Sensitive attribute, pulling admin-visible secrets into state and, for sanitized responses, blanking user config. Read now preserves the config value, env is marked Sensitive, and the docs warn against passing secrets via args. Regression tests cover both fixes and fail against the previous behavior. |
||
|
Yassin Kortam
|
38b28b96ff
|
fix(terraform/gcp): abandon SQL user on destroy (#29855)
google_sql_user.app issues DROP ROLE on destroy, which Postgres refuses because the role owns every table the migrations job created (75 objects). The previous deletion_policy=ABANDON on google_sql_database keeps the DB intact through destroy, so the role still owns its objects. Set the same policy on the user; the instance deletion takes both the database and the role with it anyway. |
||
|
Yassin Kortam
|
43c10370ee
|
fix(terraform/gcp): prompt for image_registry in DeployStack one-click (#29852)
* fix(terraform/gcp): prompt for image_registry in DeployStack one-click The four litellm-* images live on GHCR and Cloud Run rejects ghcr.io URIs at apply time, so every deploy has to point image_registry at an Artifact Registry remote repo. The DeployStack installer didn't surface image_registry as a prompt, so a click-through user landed on the ghcr.io/berriai default and the apply failed ~20 min in, after Cloud SQL had already provisioned. Add image_registry to custom_settings with a PROJECT_ID-placeholder default and a description that flags the ghcr.io rejection so the failure happens at the prompt, not after billing the slow path. TUTORIAL.md is reworded to tell the user what to enter at the new prompt instead of "edit terraform.tfvars before applying". * fix(terraform/gcp): generalize image_registry default to any region Per Greptile feedback on #29852, the prior default hardcoded us-central1 and would silently produce a Cloud Run-incompatible image path for any deployment in another region. The user would substitute PROJECT_ID, miss the region segment, and reproduce the original late-apply failure. Use REGION as a second placeholder and tighten the prompt copy so both substitutions are mandatory. * fix(terraform/gcp): make destroy work without manual intervention Three Cloud Run v2 services and the migrations Cloud Run v2 job all default to deletion_protection=true at the provider level, which has no data-safety value on stateless resources and blocks terraform destroy with an error that can only be unstuck with a tfvars edit + apply roundtrip. Wire deletion_protection=false directly on all four; the operator-facing tripwire that matters is cloudsql_deletion_protection, which guards the only resource that actually holds data. The litellm Cloud SQL database also drops cleanly only if every connection is closed first. Cloud Run services and the migrations job hold connections open until they're torn down, so destroy races and fails with "database is being accessed by other users". Setting deletion_policy=ABANDON on the database resource lets terraform skip the explicit drop; the Cloud SQL instance deletion takes the database with it anyway. Together these turn destroy into a single command, matching the AWS stack's behavior. |
||
|
Yassin Kortam
|
1cff02f50e
|
refactor: convert AWS and GCP Terraform stacks into reusable modules … (#28103)
* refactor: convert AWS and GCP Terraform stacks into reusable modules with examples/default entry point
- Remove `provider` blocks from both AWS and GCP stack roots so the modules
can be consumed with `count`, `for_each`, `depends_on`, assumed-role or
aliased providers — patterns that are forbidden when a module owns its own
provider configuration
- Add `examples/default/` thin-root wrappers for both stacks that wire the
provider (AWS) / providers (google + google-beta) and call the module with
a curated variable surface, preserving the one-command deploy experience
- Move `terraform.tfvars.example` files into `examples/default/` alongside
the new roots; update example comments to reflect the curated variable surface
- Thread `local.tags` (containing `litellm:stack`, `managed-by`, and
`var.tags`) explicitly onto every taggable AWS resource since the module no
longer controls the provider's `default_tags`; GCP resource labels already
flow through the module's `labels` input
- Add `examples/default/variables.tf` and `outputs.tf` for both stacks,
exposing the most-used knobs and re-exporting all module outputs
- Commit provider lock files for both examples so `terraform init` is
reproducible without a network fetch
- Update top-level and per-stack READMEs to document the module-first design,
the `for_each` multi-tenant pattern, and the `examples/default/` quick-start path
* docs(terraform): address review — state-migration guide, tag dedupe, for_each note
- Add 'Migrating an existing deployment' section to AWS & GCP READMEs
documenting the required terraform state mv step (resource addresses now
gain a module.litellm. prefix under the examples/default root)
- Remove redundant managed-by tag from the AWS example providers.tf;
reserve default_tags there for org-wide tags only
- Document the for_each single-provider limitation for GCP (no
configuration_aliases) in the README and example main.tf
Resolves LIT-3504
* docs(terraform/gcp): note expected SSL cert replacement in state-migration guide
The managed SSL cert is named with a hash of lb_domains, so TLS-enabled
stacks that migrated from the old un-hashed name will see one
create_before_destroy cert replacement after terraform state mv — not a
clean 'No changes'. Document that this single replacement is expected and
safe.
* docs(terraform): drop state-migration guides
The AWS/GCP stacks have never been published, so there are no existing
deployments to migrate from the old root-module layout. Remove the
'Migrating an existing deployment' sections from both READMEs.
* docs(terraform): call out image-registry override required for GCP 1-click
The GCP stack's default image_registry points at ghcr.io, which Cloud
Run won't authenticate against, so any real deploy (HCP Terraform
no-code or otherwise) must override it. Document that as a hard
requirement on the GCP README rather than a side note, and add a
top-level HCP Terraform 1-click section enumerating the required
inputs per stack and the migration-task caveat for HCP-hosted runners.
* feat(terraform/aws): mount proxy_config from S3 and wire OpenTelemetry v2
proxy_config
Drop the inline LITELLM_PROXY_CONFIG_B64 env var. Upload the YAML to S3
at config/litellm-config.yaml; gateway and backend container entrypoints
download it to /tmp/litellm-config.yaml via boto3 before exec'ing
uvicorn. The S3 object etag is wired into the task definition so a
config edit produces a new task-def revision and a rolling redeploy. The
existing s3_access policy already grants the task role s3:GetObject on
this bucket, so no IAM changes were needed for the mount itself.
OpenTelemetry v2
New variables otel_endpoint, otel_exporter, otel_service_name, and
otel_headers_secret_arn. Setting otel_endpoint to a non-empty value adds
LITELLM_OTEL_V2=true plus OTEL_EXPORTER / OTEL_ENDPOINT /
OTEL_SERVICE_NAME / OTEL_ENVIRONMENT_NAME to the shared env block; an
optional Secrets Manager ARN backs OTEL_HEADERS for collectors that need
an auth header. Execution role auto-gains GetSecretValue on that ARN.
Empty endpoint = nothing added, so existing deployments are unchanged.
* feat(terraform/gcp): add DeployStack one-click installer
Wires up a Cloud Shell "Open in Cloud Shell" badge backed by the
GoogleCloudPlatform DeployStack flow so examples/default can be
installed from a click in the README without a local terraform setup.
- examples/default/deploystack.json drives project/region collection
plus prompts for tenant, env, image_tag, and allow_plaintext_lb.
Complex inputs (proxy_config, *_extra_secrets, lb_domains) and
sensitive vars (litellm_master_key, litellm_license, ui_password)
stay tfvars / env only so they never land in a committed file.
- examples/default/TUTORIAL.md is a Cloud Shell walkthrough that
enables required APIs, creates the GHCR-passthrough Artifact
Registry repo, optionally exports the TF_VAR_* secrets, runs
`deploystack install`, and shows how to fetch the master key plus
migrate from plaintext LB to TLS.
- Renames var.project to var.project_id across the module and the
examples/default wrapper to match the variable DeployStack injects
from `collect_project: true`. Breaking rename for anyone with a
`project = ...` line in terraform.tfvars; the fix is one line.
* feat(terraform/gcp): mount proxy_config from GCS and wire OpenTelemetry v2
proxy_config
Drop the inline LITELLM_PROXY_CONFIG_B64 env var and the python-decode
startup fragment. Upload the YAML to a dedicated GCS bucket as
config.yaml, then mount it read-only into the gateway and backend at
/etc/litellm via Cloud Run v2's gcsfuse volume. CONFIG_FILE_PATH points
at the mount; an md5 of the YAML rides along as PROXY_CONFIG_HASH so a
config-only edit forces a new Cloud Run revision (gcsfuse only surfaces
new objects on container restart, so without the hash an updated
proxy_config would sit in the bucket unread).
The config bucket is separate from the data-plane bucket so the runtime
SA can hold objectViewer here (read-only at runtime) while keeping
objectAdmin on the data-plane bucket. Both bucket and IAM binding are
gated on proxy_config != {}; an empty config skips bucket creation and
mounts nothing.
OpenTelemetry v2
LITELLM_OTEL_V2=true is now wired into shared_env_kv unconditionally so
both the gateway and backend boot with the integration enabled. It's
dormant until otel_endpoint is non-empty; setting it injects
OTEL_EXPORTER / OTEL_ENDPOINT / OTEL_ENVIRONMENT_NAME plus a
per-component OTEL_SERVICE_NAME (\${tenant}-litellm-\${env}-{gateway,backend})
so spans land tagged with the right hop. otel_headers_secret takes a
Secret Manager resource ID for OTEL_HEADERS (collector auth); the
runtime SA auto-gains roles/secretmanager.secretAccessor on it.
otel_capture_message_content defaults to no_content matching the litellm
default. Any OTEL_* key set in *_extra_env wins over the defaults so
Cloud Run doesn't reject the apply on the duplicate-env-name check.
* refactor(terraform): make AWS and GCP stacks behave identically
Bring both modules to the same surface and the same runtime behavior so
swapping clouds (or reading either README) is symmetric.
Labels and tags. GCP previously stamped var.labels onto only the two GCS
buckets, leaving Cloud Run, Cloud SQL, Memorystore, Secret Manager, and
the LB resources unlabeled; the variable description claimed full
coverage. Now the module computes local.labels (litellm-stack +
managed-by + var.labels, mirroring AWS's local.tags) and threads it onto
every label-supporting resource: Cloud Run services and the migrations
job, Cloud SQL writer and reader (via user_labels), Memorystore, Secret
Manager entries (master_key, license, ui_password, db_password), both
GCS buckets, the global LB address, and the http/https forwarding rules.
GCP keys use 'litellm-stack' instead of AWS's 'litellm:stack' because
GCP label keys forbid colons; var.labels now defaults to {}.
OpenTelemetry v2 is opt-in on both stacks. AWS already gated everything
on otel_endpoint; GCP previously stamped LITELLM_OTEL_V2=true into
shared_env unconditionally and only ungated the OTEL_* block. Both
stacks now do the same thing: leave otel_endpoint empty and nothing
OTel-related lands in the container env; set it and gateway and backend
get LITELLM_OTEL_V2=true plus OTEL_EXPORTER, OTEL_ENDPOINT,
OTEL_ENVIRONMENT_NAME, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT,
and a per-component OTEL_SERVICE_NAME (${tenant}-litellm-${env}-gateway
or -backend) so spans land tagged with the right hop. AWS picks up the
richer GCP surface: otel_environment_name (defaults to var.env),
otel_capture_message_content (defaults to no_content), and *_extra_env
override filtering so a caller-set OTEL_* key wins over the default for
that service (ECS allows duplicates, but the filter gives the same
predictable last-wins shape Cloud Run enforces). var.otel_service_name
on AWS is gone, replaced by the per-component naming.
uvicorn workers. GCP gains gateway_num_workers, matching AWS; threads
into the gateway args as --workers ${var.gateway_num_workers}.
Docs reflect the parity: each README's OTel section, the GCP 'Using as
a module' Labels paragraph, and a new feature-parity table in the
top-level README that lays out the AWS/GCP input mapping side by side.
* fix(terraform/aws): expose skip_final_snapshot through the default example
The example wrapper already exposed `s3_force_destroy` so ephemeral / CI
stacks could destroy the S3 bucket without manual cleanup, but the matching
Aurora knob (`skip_final_snapshot`) was hidden behind the module surface.
That meant a `terraform destroy` on a trial stack still produced a
`<cluster>-final-<short-sha>` snapshot, with no opt-out short of editing the
module call.
Adds `var.skip_final_snapshot` to the example (default `false`, preserving
the data-loss tripwire) and threads it through to the module input,
mirroring the existing `s3_force_destroy` pattern. Documented alongside it
in the tfvars example.
Verified by deploying the example end-to-end against a clean AWS account
(VPC + Aurora w/ IAM auth + Redis + ALB + 3 ECS services), confirming all
services reach steady state and the data plane serves traffic, then running
`terraform destroy` with `skip_final_snapshot = true` to a clean teardown
(93 destroyed, no Aurora snapshot left behind, no leftover billable
resources).
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: yassin-berriai <yassin.kortam@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Yassin Kortam
|
3d5a9ede05
|
feat: add Terraform stacks for deploying LiteLLM on AWS and GCP (#27673)
- Add AWS ECS Fargate stack with Aurora Postgres (IAM auth), ElastiCache Redis, S3, ALB with path-based routing to gateway/backend/ui components, Application Auto Scaling, and automated DB bootstrap + prisma migration via local-exec provisioners - Add GCP Cloud Run stack with Cloud SQL Postgres (password auth), Memorystore Redis, GCS, external HTTPS load balancer with serverless NEGs and URL map routing, and automated prisma migration via Cloud Run Job - Both stacks support typed proxy_config input mirroring the helm chart's gateway.config.proxy_config, per-component extra env vars, and Secret Manager references for provider API keys - Gateway/backend services depend on terraform_data.migration so they never start before the schema is in place, eliminating crash-loop windows on first apply - AWS stack uses IAM database authentication with a one-shot Fargate bootstrap task that creates and grants the rds_iam role to the application user; GCP stack uses password auth assembled at container startup to avoid Cloud SQL Auth Proxy sidecar complexity - Add .gitignore rules for Terraform state files, plan files, tfvars inputs, provider binaries, and crash logs while explicitly keeping .terraform.lock.hcl for provider version pinning - Include terraform.tfvars.example files, provider lock files, and comprehensive README documentation covering architecture, TLS setup, image pull strategies, and quick-start instructions for both stacks Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |