mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
Mirror of https://github.com/BerriAI/litellm.git
3165951e6d
* feat(otel v2): send a key's or team's whole trace to its own destination A key or team that configures its own Langfuse, Arize, Weave or New Relic credentials used to get a single detached span in its account while the rest of the request trace stayed on the operator's backend, so neither side held a complete trace. Resolve the destination during auth, forward every span of the request to it, and hold the same request back from the operator's exporter for that backend, so the tenant gets the tree the operator would have seen and the operator gets nothing for that request. Also let a credential-mandatory preset build without the operator's own env credentials. Without that, a proxy whose teams each bring their own account fell back to the legacy integration and never ran a line of the v2 path. * fix(otel v2): validate tenant destinations and match each backend's own endpoint Review round on the tenant destination routing. - A key/team Langfuse host is user-supplied input, so it goes through the proxy's SSRF guard. A private address is refused, the operator keeps the trace, and the warning names user_url_allowed_hosts. The operator's own LANGFUSE_HOST is not checked. - Arize and Weave destinations now resolve their endpoint and transport through the backend's own config, so an ARIZE_HTTP_ENDPOINT collector and a self-hosted WANDB_HOST are honoured instead of the cloud default. - A half-configured backend no longer resolves: several dynamic header builders gate each credential separately, so an api key with no space id produced a non-empty but unusable header set that suppressed the operator's exporter. - A callback_type of "failure" no longer takes over the trace. The destination is resolved during auth, before the outcome is known. - The fan-out cache evicts without shutting the processor down, matching ArizePhoenixLogger: a concurrent on_end may still hold it. - The stdout placeholder is identified by what it does rather than by equality with an import-time default, so an operator's OTEL_EXPORTER_OTLP_* collector survives the credential-less path. * refactor(otel v2): reuse the proxy's own destination allowlist for tenant hosts A tenant-supplied Langfuse host is the same threat as a URL-valued `model`, so it now goes through `is_url_destination_allowed_by_host` against `provider_url_destination_allowed_hosts` instead of a second, DNS-based check of its own. The DNS lookup would have blocked the asyncio auth path on a hostname the caller picked, and its cached verdicts could blackhole a real host after one resolver blip. Evicting a destination processor now retires it to drain rather than shutting it down, since `on_end` hands a processor back and exports outside the lock. The retirees are capped so they cannot accumulate a thread each. `credential_gated_exporters` tells the synthesized stdout placeholder from a real exporter by transport rather than by the literal kind `console`, so an unrecognized kind is not mistaken for a configured collector, and an exporter the operator did configure survives. That also stops a weave test's env writes from making this look like a real OTLP exporter later in the same CI worker. * fix(otel v2): read the tenant's stored callback config the way the sibling parser does Three divergences between the destination resolver and `convert_key_logging_metadata_to_callback`, which read the same stored config: - A key whose callbacks are disabled stores an empty list, and `or` treated that as "the key configured nothing", so the request inherited the team's destination. The sibling parser treats an empty list as configured. - Two entries naming one backend now merge their `callback_vars` last-wins, matching the sibling, instead of the resolver taking the first entry and the per-request tracer routing taking the last. - `credential_gated_exporters` dropped any exporter whose kind had no transport, which also dropped an `in_memory` exporter the operator asked for. The placeholder is the spec with every field still at its default, so that is what the predicate now says. Arize's `allow_missing_credentials` branch was unreachable: `get_arize_config` resolves every credential with `os.environ.get` and always supplies an endpoint, so it never raises. Dropped it and corrected the protocol docstring. * fix(otel v2): keep the destination merge immutable The per-backend var merge seeded a plain dict and the gated exporter list a plain list, both of which the LIT budget counts. Wrap the merge in MappingProxyType and hand the exporters back as a tuple. * fix(otel v2): scope the fan-out to its own backend and close shed processors off the export path Three problems in the fan-out, two of them in the eviction added last round: - Every v2 logger carries its own provider and emits its own copy of a gen-AI span, so a proxy running two of them handed the tenant the same model call twice. A provider now forwards only destinations for the backend it speaks for; the tenant's own backend always has a logger, since naming it in the key or team config is what builds one. Reproduced live against a self-hosted Langfuse on an arize-only proxy and on the bare `otel` callback. - Eviction could close a processor another thread was still exporting through, which drops that span. Exports are now counted, and a retired processor is closed only once its count reaches zero. - That close ran inside `on_end`, where `shutdown` flushes over the network, so one unreachable tenant collector stalled every other tenant's spans. It now runs on a short-lived thread, which also retires the retiree cap: a retiree drains as soon as its export finishes. * fix(otel v2): deliver tenant destinations from the published global provider Scoping the fan-out by callback name in the previous commit left every backend that is not the canonical logger with a one-span trace: only the published global provider sees the FastAPI server span, the auth span and the post-call database spans, so an arize-only proxy handed a team's Langfuse just the model call. Attach the fan-out once, to that provider, and let it forward every destination. An overridden backend now skips per-request tracer routing outright rather than only clearing its credential headers, since a key or team otel_service_name was still enough to detach the model call onto a second provider. The destination carries that service name as a resource attribute instead. Shed processors drain on a two-thread pool rather than a thread each, so a tenant cycling its destination config cannot spawn threads as fast as it sends requests. * fix(otel v2): drain shed destination processors on daemon workers A ThreadPoolExecutor joins its workers at interpreter exit, so one unreachable tenant collector would hold the whole proxy open for its export timeout on the way down. Two long-lived daemon workers off a queue keep the thread count bounded without blocking shutdown. * fix(otel v2): give the fan-out its own drain pool instead of a lazy singleton functools.lru_cache does not hold a lock across the call it caches, so concurrent first evictions each finish building a queue and start its workers, and every queue but the winner is abandoned with two daemon threads blocked on it forever. * fix(otel v2): close no destination processor under a span still in flight The fan-out now refuses new work once shutdown starts and waits out the spans already being forwarded, so teardown neither drops a trace mid-forward nor hands the next caller an exporter nothing will ever close. The wait is bounded so a dead collector cannot hold the proxy open. * fix(otel v2): retire the drain workers with the fan-out that started them A proxy that rebuilds its telemetry builds another fan-out, so workers that outlive the one that started them are two more threads per reload. Shutdown now retires them once everything queued is closed, and a processor shed afterwards is closed inline rather than queued to nobody. * fix(otel v2): guard the fan-out's closed state with the lock that gates it An Event read on its own leaves room for shutdown to run in the gap. A cache miss then inserted a live exporter into a map that had been cleared, and a shed processor landed behind sentinels every drain worker had exited on. The drain pool takes its queue by injection so both interleavings are reachable from a test without patching. * feat(otel v2): let a tenant destination export alongside the operator's own Override stays the default: a key or team destination replaces the operator's exporter for that backend. Operators running one org-wide backend across every team set litellm_settings.otel_tenant_destination_mode to additive, and the same trace lands in both places. A team that names the operator's own project is still written once, since the fan-out skips a destination the operator's exporter is already sending that span to. * fix(otel v2): let a straggling export close its own destination processor Shutdown waits out the exports in flight, but the wait has to be bounded or a tenant collector that stops answering holds the proxy open on the way down. Past the bound it closed everything anyway, which is the case it was written to avoid: a processor closed under the span it is carrying loses that span. Keep the bound and retire the stragglers instead. The thread still exporting one closes it through the drain as soon as its export returns, so teardown stays bounded and no span is dropped mid-forward. * fix(otel v2): identify a destination account by its credentials, not its header names Under additive the fan-out skips a destination the operator's own exporter already writes to, so the same account is not written twice. It compared header names as well as values, and one account answers to more than one spelling: the operator's Arize exporter sends space_id where a team destination sends arize-space-id, so every span landed in the operator's own space twice. The credentials are the identity. Compare those and leave the spelling to each backend. * fix(otel v2): keep the credential's role in a destination's account identity Comparing values alone folds two accounts together whenever they hold the same strings in different roles, and the second team would then get no trace at all. Compare the credential under a normalized name instead, and fold the one alias that actually exists: Arize's space_id and arize-space-id. * fix(otel v2): build one destination processor per destination, not per racing span Building outside the cache lock meant a cold cache met by a burst of concurrent requests constructed an exporter per thread, kept one, and handed the rest to the drain, so a batch worker and a connection pool per losing thread sat in a queue two workers service. Build under the lock that reads the cache. Opening an exporter connects to nothing, so the lock is held for a constructor, once per destination, and the race it was avoiding stops existing. * fix(otel v2): bound the teardown that closes a destination, not the one that never blocks The five-second bound guarded the wait for spans still inside on_end, but a batching processor's on_end only queues the span and returns, so that counter is empty and the bound engaged against nothing. The blocking half was the serial close, which flushes over the network and joins the SDK's own worker thread with no timeout of its own, so a single tenant collector that answers and never finishes held process teardown open for as long as it liked. Hand every close to the drain, whose workers are daemons, and give the whole teardown one deadline. * fix(otel v2): preserve operator spans on destination failure * fix(otel v2): anchor destinations off the published provider, refuse headerless tenant transports set_tracer_provider keeps the first provider it is handed, so a process whose OTel global was claimed before the proxy published (auto-instrumentation, a legacy logger) had no fan-out on the global and auth anchored no destination. Auth now reads the fan-out off the registered logger's own provider. A destination whose protocol maps to a headerless exporter kind is no longer buildable: the console fallback would drop the tenant's credentials and print the spans to stdout while the operator's exporter stood down for them. * fix(otel): anchor tenant fan-out to the published provider A legacy v1 logger can occupy proxy_server.open_telemetry_logger, in which case the proxy publishes with registered=None and the fan-out lands on a v2 logger taken from _in_memory_loggers. Reading the registered slot found no v2 logger and the OTel global belonged to v1, so auth refused every tenant destination. * fix(otel): preserve registered provider fallback * test(otel): cover pre-publish provider fallback * fix(otel): attach fan-out on fallback provider * fix(otel): serialize first fan-out attach * fix(otel): keep the operator's database endpoint out of tenant traces A database span forwarded to a key or team destination carried the proxy's own Postgres host, port and schema, and on failure the Prisma error text naming them. The fan-out now hands tenants a view of each database span without those keys, its events or its status text, while the operator's own copy is untouched and model endpoints such as server.address on the LLM span still travel * fix(otel): keep relabelled spans in the fan-out and honour disabled callbacks for destinations A key or team otel_service_name used to move a backend's span onto a second provider even when another backend had a destination, so the fan-out never saw the model call and the tenant's trace lost it. A service name alone now stays on the published provider whenever the request has a destination; credential and project routing to a tenant's own account is unchanged Destinations now skip a backend the request disabled dynamically, reading the x-litellm-disable-callbacks header and the key's litellm_disabled_callbacks with the same precedence and premium gate dispatch applies, so a disabled backend is neither delivered to nor withheld from the operator * test(otel): project routing survives a sibling backend destination * docs(otel): state why a disabled backend still routes its own span * fix(otel): keep a degraded backend's spans off a collector another v2 logger already serves * test(otel): a credentialed preset beside another v2 logger keeps every exporter * fix(otel): keep credentialless fallback on base path * test(otel): cover legacy callback carrier rejection * fix(otel): preserve valid exporter beside gated preset * fix(otel): avoid console export without operator destination * refactor(otel): share the console placeholder check with the presets * fix(otel): bound shed destination processors waiting on a dead collector * fix(otel): preserve explicit console exporters * fix(otel): avoid mutable field-set construction * fix(otel): close drain saturation race * fix(otel): drop captured request headers from tenant spans * test(otel v2): give the newrelic dispatch tests operator credentials, since a credential-less preset now falls back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): rebuild an anchored destination's processor past drain saturation A destination deliverable() accepted at auth can be evicted by other tenants' auths before its request's spans end, and that eviction is what tips the drain over. The saturation gate then refused the rebuild at on_end, and with the operator's exporter already stood down for that backend the span went nowhere. The gate now applies only while a request decides whether to anchor * fix(otel): hold destination eviction while the drain is saturated An anchored destination evicted by other tenants' auths is rebuilt on its next span, and that rebuild evicted another anchored one, so with more destinations in flight than the cache holds every span cost one more processor, one more batch thread and one more close queued behind a collector that never answers. Eviction now holds while the drain is saturated, so the cache keeps one entry per destination in flight and trims back to its cap on the next hit or build once the drain has room * fix(otel): keep the proxy's own error text out of tenant traces A tenant destination received every span the request produced, error text included, so a Prisma failure during auth handed a team admin's collector the operator's Postgres endpoint, and the exception event on any failed span carried a stack trace naming the proxy's install paths. Spans the tenant's own call produced (the model call, MCP, guardrails) keep their error text. Every other span keeps the failure without the prose: its type, its provider error code and its status code, with the message, the events and the status description dropped. Stack traces come off every span, attribute and event alike. A destination's resource attributes now merge onto the span's resource instead of rebuilding one per span, which was re-running resource detection on every export. * fix(otel): redact tenant URL query parameters * fix(otel): close final tenant routing gaps * fix(otel): refresh destinations for stateful MCP messages --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .cargo | ||
| .circleci | ||
| .devcontainer | ||
| .githooks | ||
| .github | ||
| .semgrep/rules | ||
| backend | ||
| ci_cd | ||
| cookbook | ||
| db_scripts | ||
| docker | ||
| enterprise | ||
| examples | ||
| gateway | ||
| helm | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm-rust | ||
| migrations | ||
| packaging/homebrew | ||
| scripts | ||
| terraform | ||
| tests | ||
| ui | ||
| .dockerignore | ||
| .env.example | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitguardian.yaml | ||
| .gitignore | ||
| .npmrc | ||
| AGENTS.md | ||
| ARCHITECTURE.md | ||
| basedpyright-code-budget.json | ||
| CLAUDE.md | ||
| codecov.yaml | ||
| CONTRIBUTING.md | ||
| cosign.pub | ||
| docker-compose.hardened.yml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GEMINI.md | ||
| LICENSE | ||
| license_cache.json | ||
| Makefile | ||
| mcp_servers.json | ||
| model_prices_and_context_window.json | ||
| model_prices_and_context_window.schema.json | ||
| osv-scanner.toml | ||
| package-lock.json | ||
| package.json | ||
| policy_templates.json | ||
| prometheus.yml | ||
| provider_endpoints_support.json | ||
| proxy_server_config.yaml | ||
| pyproject.toml | ||
| pyrightconfig.json | ||
| qa_sticky_session.sh | ||
| README.md | ||
| render.yaml | ||
| router_plugins.json | ||
| ruff-strict-budget.json | ||
| ruff-strict.toml | ||
| ruff-tests.toml | ||
| ruff.toml | ||
| rust-toolchain.toml | ||
| schema.prisma | ||
| security.md | ||
| taplo.toml | ||
| test-quality-budget.json | ||
| type-discipline-budget.json | ||
| uv.lock | ||
| whitelisted_bedrock_models.txt | ||
🚅 LiteLLM
LiteLLM AI Gateway
Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website
What is LiteLLM
LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.
Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
Why LiteLLM
Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:
- Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
- Drop-in OpenAI compatibility — swap providers without rewriting your code
- Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
- 8ms P95 latency at 1k RPS (benchmarks)
OSS Adopters
Netflix |
Features
LLMs - Call 100+ LLMs (Python SDK + AI Gateway)
All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.
Python SDK
uv add litellm
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
AI Gateway (Proxy Server)
Getting Started - E2E Tutorial - Setup virtual keys, make your first request
uv tool install 'litellm[proxy]'
litellm --model gpt-4o
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Agents - Invoke A2A Agents (Python SDK + AI Gateway)
Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
Python SDK - A2A Protocol
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
AI Gateway (Proxy Server)
Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent
Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)
import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4
base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"} # LiteLLM Virtual Key
async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
config = ClientConfig(
httpx_client=http_client,
streaming=False,
supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
)
client = ClientFactory(config).create(agent_card)
request = SendMessageRequest(
message=Message(
message_id=uuid4().hex,
role=Role.ROLE_USER,
parts=[Part(text="Hello!")],
)
)
async for event in client.send_message(request):
populated = event.ListFields()
if populated and populated[0][0].name in ("message", "msg"):
print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))
MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)
Python SDK - MCP Bridge
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Load MCP tools in OpenAI format
tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
# Use with any LiteLLM model
response = await litellm.acompletion(
model="gpt-4o",
messages=[{"role": "user", "content": "What's 3 + 5?"}],
tools=tools
)
AI Gateway - MCP Gateway
Step 1. Add your MCP Server to the AI Gateway
Step 2. Call MCP tools via /chat/completions
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize the latest open PR"}],
"tools": [{
"type": "mcp",
"server_url": "litellm_proxy/mcp/github",
"server_label": "github_mcp",
"require_approval": "never"
}]
}'
Use with Cursor IDE
{
"mcpServers": {
"LiteLLM": {
"url": "http://localhost:4000/mcp/",
"headers": {
"x-litellm-api-key": "Bearer sk-1234"
}
}
}
}
Supported Providers (Website Supported Models | Docs)
Get Started
You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
| LiteLLM AI Gateway | LiteLLM Python SDK | |
|---|---|---|
| Use Case | Central service (LLM Gateway) to access multiple LLMs | Use LiteLLM directly in your Python code |
| Who Uses It? | Gen AI Enablement / ML Platform Teams | Developers building LLM projects |
| Key Features | Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management | Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.) |
Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Deploy on AWS or GCP with Terraform
Run the LiteLLM proxy as a production-ready componentized stack (gateway, backend, UI on separate services; managed Postgres + Redis + object store) using the published Terraform modules. Both modules are on the public Terraform Registry — no auth needed.
AWS — ECS Fargate + Aurora + ElastiCache + ALB
git clone https://github.com/BerriAI/litellm.git
cd litellm/terraform/litellm/aws/examples/default
cp terraform.tfvars.example terraform.tfvars # edit region/tenant/env
terraform init && terraform apply
Or call the module from your own root config:
# main.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.60" }
}
}
provider "aws" {
region = "us-west-2"
}
module "litellm" {
source = "BerriAI/litellm/aws"
version = "~> 1.89"
region = "us-west-2"
azs = ["us-west-2a", "us-west-2b"]
tenant = "acme"
env = "prod"
# Production: provide an ACM cert. Without one, set allow_plaintext_alb = true
# (dev/trial only).
# acm_certificate_arn = "arn:aws:acm:us-west-2:111122223333:certificate/..."
allow_plaintext_alb = true
}
output "litellm_url" {
value = module.litellm.alb_dns_name
}
terraform init
terraform apply
Provider API keys live in AWS Secrets Manager; reference ARNs via gateway_extra_secrets. Full input list and architecture diagram on the registry page.
GCP — Cloud Run + Cloud SQL + Memorystore + HTTPS LB
Real 1-click. Opens Cloud Shell, clones this repo, and walks you through terraform apply via a built-in DeployStack tutorial — pick the project, the tutorial sets up the Artifact Registry remote repo, writes terraform.tfvars from your answers, and runs apply.
To call the module from your own config instead, Cloud Run can't pull from ghcr.io directly, so first set up a one-time Artifact Registry remote repo backed by GHCR:
gcloud artifacts repositories create litellm \
--location=us-central1 \
--repository-format=docker \
--mode=remote-repository \
--remote-docker-repo=https://ghcr.io \
--project=my-gcp-project
Then:
# main.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
google = { source = "hashicorp/google", version = "~> 6.10" }
google-beta = { source = "hashicorp/google-beta", version = "~> 6.10" }
}
}
provider "google" { project = "my-gcp-project"; region = "us-central1" }
provider "google-beta" { project = "my-gcp-project"; region = "us-central1" }
module "litellm" {
source = "BerriAI/litellm/google"
version = "~> 1.89"
project_id = "my-gcp-project"
region = "us-central1"
tenant = "acme"
env = "prod"
# Replace my-gcp-project with your GCP project ID (same value as project_id above).
image_registry = "us-central1-docker.pkg.dev/my-gcp-project/litellm/berriai"
# Production: provide DNS already pointing at the LB IP for Google-managed certs.
# Without one, set allow_plaintext_lb = true (dev/trial only).
# lb_domains = ["proxy.example.com"]
allow_plaintext_lb = true
}
output "litellm_url" {
value = module.litellm.load_balancer_url
}
terraform init
terraform apply
Provider API keys live in Secret Manager; reference resource IDs (e.g. projects/my-gcp-project/secrets/openai-api-key) via gateway_extra_secrets. Full input list and architecture diagram on the registry page.
Both stacks include
- The full componentized split (gateway / backend / UI as independent services)
- Managed Postgres (writer + reader) and Redis
- Versioned object store for proxy state + file uploads
- An auto-generated
LITELLM_MASTER_KEYin your cloud's secret manager - A one-off migration job that runs
prisma migrate deploybefore the proxy starts - The same
proxy_configsurface as the Helm chart — pass YAML as a typed map
The Terraform modules live at terraform/litellm/aws/ and terraform/litellm/gcp/ in this repo; the registry entries are read-only mirrors updated on each release.
Run in Developer Mode
Services
- Setup .env file in root
- Run dependent services
docker-compose up db prometheus
Backend
- Run
make bootstrap - Start proxy backend:
uv run python litellm/proxy/proxy_cli.py
Frontend
- Navigate to
ui/litellm-dashboard(dependencies were already installed w/make bootstrap) - Start dashboard:
npm run dev
Verify Docker Image Signatures
All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.
Verify using the pinned commit hash (recommended):
A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Verify using a release tag (convenience):
Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).
Enterprise
For companies that need better security, user management and professional support
Get an Enterprise License Talk to founders
This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On
Contributing
We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
Quick Start for Contributors
This requires uv to be installed.
git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev # Install development dependencies
make format # Format your code
make lint # Run all linting checks
make test-unit # Run unit tests
make format-check # Check formatting only
For detailed contributing guidelines, see CONTRIBUTING.md.
📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
Our automated checks include:
- Black for code formatting
- Ruff for linting and code quality
- MyPy for type checking
- Circular import detection
- Import safety checks
All these checks must pass before your PR can be merged.
Support / talk with founders
- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai