7c7304c22b
* feat(providers): add ZeroGPU as an OpenAI-compatible provider Registers zerogpu through the JSON provider registry so zerogpu/<model> routes to https://api.zerogpu.ai/v1/chat/completions with ZEROGPU_API_KEY and an optional ZEROGPU_API_BASE. Adds LlmProviders.ZEROGPU, the constants.py provider and endpoint lists, cost map entries for the 10 ZeroGPU text generation models, the endpoint matrix, the Add Model form fields, the README row and the dashboard provider entry * feat(zerogpu): add capability flags verified against the live API Live calls to https://api.zerogpu.ai/v1/chat/completions confirmed tools, tool_choice and json_schema response_format on 8 of the 10 catalog text models, cached token reporting on deepseek-v4.1-flash, gpt-4.1-mini, gpt-5.4-nano and gpt-5.6-luna, and reasoning_effort on gpt-5.6-luna. LFM2.5-1.2B-Instruct answers stream requests with plain JSON instead of SSE and LFM2.5-1.2B-Thinking returns 500 on every request, so both stay out of the cost map until the API serves them correctly * feat(zerogpu): price LFM2.5-1.2B-Instruct Adds zerogpu/LFM2.5-1.2B-Instruct to the cost map with the catalog prices and context window. It gets no capability flags because the live API ignores tools, response_format and max_completion_tokens for this model, so LiteLLM drops tool params for it instead of sending them. LFM2.5-1.2B-Thinking stays out of the cost map * fix(zerogpu): reject transcription requests before calling the upstream API ZeroGPU only serves chat completions, so drop it from the OpenAI transcription transport set. A zerogpu transcription request now fails locally instead of hitting an /audio/transcriptions endpoint that does not exist * test(zerogpu): cover the responses and messages bridges Both endpoints reach ZeroGPU through chat completions, so assert the request each one sends upstream and the reply it returns * refactor(zerogpu): move zerogpu out of the provider list tails New upstream providers append to the end of openai_compatible_endpoints, openai_compatible_providers and providers.json, so an entry at the tail conflicts with every one of them. Placing zerogpu next to long-stable entries keeps those merges clean. No behavior change * fix(zerogpu): price cached input for qwen3-30b-a3b-fp8 The ZeroGPU model catalog (https://docs.zerogpu.ai/docs/model-catalog, checked 2026-10-09) lists cached input for qwen3-30b-a3b-fp8 at $0.05 per 1M tokens. Without cache_read_input_token_cost, cached tokens were billed at the full $0.10 input rate * revert(zerogpu): leave the audio transcription provider set unchanged Per review, keep OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS as upstream defines it. This PR only covers chat completions, and ZeroGPU now serves /audio/transcriptions anyway, so the local refusal and its test are removed * fix(zerogpu): add max_tokens so trim_messages can size ZeroGPU history trim_messages reads max_tokens as the fallback for max_input_tokens and evaluates it eagerly, so without it the lookup raised, got swallowed, and the full history came back untrimmed. max_tokens matches each model's maxTokens in ZeroGPU's models API (https://api-dashboard.zerogpu.ai/api/models, checked 2026-10-10) * test(zerogpu): record mocked requests on a recorder object The mock handler appended to a list the caller passed in, which AGENTS.md forbids. A recorder now owns the captured requests as a tuple * fix(zerogpu): cap max_tokens at each model's output limit Each cap is the max_output_tokens ZeroGPU stores for the model in its production models table (read 2026-10-20), the same column its /v1/models serves as max_length. That gives 65536 for deepseek-v4.1-flash, 32768 for gpt-4.1-mini and gpt-oss-120b, 128000 for gpt-5.4-nano and gpt-5.6-luna, 8192 for LFM2.5-1.2B-Instruct and llama-3.1-8b-instruct-fast, and 16384 for qwen3-30b-a3b-fp8. glm-5.3-flash is stored as 32786, which reads as a typo for 32768, so that row uses 32768 trim_messages sizes history from max_input_tokens, so lowering max_tokens leaves it unaffected * fix(zerogpu): flag prompt caching on glm-5.3-flash Repeating the same ~4.3K-token prompt against ZeroGPU on 2026-10-20 returned cached_tokens of 4320 on some glm-5.3-flash calls. gpt-oss-120b and llama-3.1-8b-instruct-fast returned 0 on all nine calls, and qwen3-30b-a3b-fp8 never returned cached_tokens, so they stay unflagged. Their cached prices stay as ZeroGPU publishes them, so any cache hit is still billed at the right rate * fix(zerogpu): set gpt-5.4-nano context window to 272000 ZeroGPU accepts a 271001-token gpt-5.4-nano prompt and rejects 273000 tokens as over the context window (checked 2026-10-21), so max_input_tokens is 272000 rather than the 400000 its catalog lists * fix(zerogpu): list zerogpu models for zerogpu/* wildcards zerogpu had no models_by_provider entry, so get_provider_models("zerogpu") returned None and a zerogpu/* wildcard listed no models. zerogpu_models is now filled from the cost map rows like openinfer_models and registered in models_by_provider * fix(zerogpu): flag gpt-5.4-nano as a reasoning model ZeroGPU honors reasoning_effort on gpt-5.4-nano (checked 2026-10-21). Without it the model spends 0 reasoning tokens, and with low or high it spends 32 or more. LiteLLM only treated it as a reasoning model by falling back to OpenAI's bare gpt-5.4-nano row, so get_model_info and /model/info reported supports_reasoning as null. The row now carries the flag itself * fix(zerogpu): flag deepseek, glm, gpt-oss and qwen3 as reasoning models ZeroGPU's /v1/models marks these four as reasoning models and accepts reasoning_effort on all of them (checked 2026-10-21), but LiteLLM rejected the param with UnsupportedParamsError because their rows had no supports_reasoning flag. glm reports reasoning_tokens under reasoning_effort, and gpt-oss and qwen3 spend hidden reasoning tokens on every reply. The gpt-oss test case now expects reasoning_effort to reach ZeroGPU * fix(zerogpu): set gpt-oss-120b context window to 128000 ZeroGPU routes gpt-oss-120b across several upstream backends, and at least one of them returns 413 past a 128000-token window, so max_input_tokens uses that as the safe floor. The backends reached on 2026-10-21 accepted up to about 131K prompt tokens, so this only gives up a little headroom on the larger ones * feat(zerogpu): call ZeroGPU's Responses API natively ZeroGPU serves /v1/responses for all nine catalog models, so LiteLLM no longer bridges Responses requests through chat completions. Native calls return ZeroGPU's reasoning output items for glm, gpt-oss and qwen3, which the bridge dropped. Checked live on 2026-10-22 through litellm.responses for every model, non-streaming and streaming, plus a gpt-oss tool call. LFM2.5-1.2B-Instruct streams correctly now that ZeroGPU's text_generation lane emits Responses SSE * fix(zerogpu): stop inferring the zerogpu provider from api_base alone Listing https://api.zerogpu.ai/v1 in openai_compatible_endpoints rerouted bare model names on that api_base from openai to zerogpu. A bare gpt-5.6-luna call with verbosity used to return 200 and started raising UnsupportedParamsError. Drop the entry and its api_base-alone test, so zerogpu is only selected through the zerogpu/ prefix * fix(zerogpu): price gpt-6-luna, llama-3.2-3b, LFM2.5-1.2B-Thinking and qwen3.6 A zerogpu/* wildcard served these four chat models with no cost-map row, so their calls logged $0 spend. Add rows at ZeroGPU's catalog prices from https://api-dashboard.zerogpu.ai/api/models, with context windows and output caps from ZeroGPU's models table. LFM2.5-1.2B-Thinking has no cached price, like LFM2.5-1.2B-Instruct. gpt-6-luna is flagged supports_reasoning because LiteLLM already routes gpt-6 names as reasoning models. Checked live on 2026-10-28, all four now report nonzero response_cost * fix(zerogpu): drop the retired max_tokens field from the zerogpu rows Upstream #48471 retired max_tokens from the cost map in favor of max_input_tokens and max_output_tokens, and test_no_row_carries_the_retired_max_tokens_field failed on all 13 zerogpu rows. Every row already carries the same value in max_output_tokens, so nothing is lost. The zerogpu limits test now checks that each row's output limit fits its context window * chore(zerogpu): regenerate provider endpoint support for zerogpu Upstream #48619 generates provider_endpoints_support.json from the cost map and checks it in CI, and the hand-written zerogpu entry no longer matched. Regenerated with scripts/generate_provider_endpoints_support.py. Only the zerogpu entry changed, and it now matches what the generator emits for other JSON-registry providers such as openinfer * fix(zerogpu): flag tools, json_schema and prompt caching on gpt-6-luna ZeroGPU serves function calling, tool_choice, json_schema response_format and prompt caching on gpt-6-luna, so set supports_function_calling, supports_tool_choice, supports_response_schema and supports_prompt_caching on its row in both cost maps. Checked live on 2026-10-28, a weather tool call came back with the right arguments, json_schema output validated, and repeated prompts reported 3260 of 3263 tokens cached * fix(zerogpu): set qwen3.6-35b-a3b context to 262144 and flag its tools and reasoning ZeroGPU accepts up to 262,144 input tokens on qwen3.6-35b-a3b and serves tools, tool_choice, json_schema response_format and reasoning_effort on it. Raise max_input_tokens from 16384 and set supports_function_calling, supports_tool_choice, supports_response_schema and supports_reasoning in both cost maps. Before this, LiteLLM rejected tools and reasoning_effort for the model. Checked live on 2026-10-28, a 209,905-token prompt succeeded, an oversized one was refused with a 262144-token limit, and tool calls, json_schema output and reasoning_effort all worked * fix(zerogpu): flag vision on gpt-6-luna and qwen3.6-35b-a3b ZeroGPU reads image inputs on gpt-6-luna and qwen3.6-35b-a3b, so set supports_vision on both rows in both cost maps. Checked live on 2026-10-28, both models named the animal in an image_url photo, and the image showed up in their prompt token counts |
||
|---|---|---|
| .agents/skills/database-access | ||
| .cargo | ||
| .circleci | ||
| .devcontainer | ||
| .githooks | ||
| .github | ||
| .semgrep/rules | ||
| backend | ||
| ci_cd | ||
| cookbook | ||
| db_scripts | ||
| docker | ||
| enterprise | ||
| examples | ||
| gateway | ||
| helm | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm-rust | ||
| litellm_cli | ||
| migrations | ||
| packaging | ||
| scripts | ||
| terraform | ||
| tests | ||
| ui | ||
| vscode-extension | ||
| .dockerignore | ||
| .env.example | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitguardian.yaml | ||
| .gitignore | ||
| .grype.yaml | ||
| .npmrc | ||
| AGENTS.md | ||
| ARCHITECTURE.md | ||
| codecov.yaml | ||
| CONTRIBUTING.md | ||
| cosign.pub | ||
| docker-compose.hardened.yml | ||
| docker-compose.liteadmin.yml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GEMINI.md | ||
| LICENSE | ||
| license_cache.json | ||
| Makefile | ||
| mcp_servers.json | ||
| model_prices_and_context_window.json | ||
| osv-scanner.toml | ||
| package-lock.json | ||
| package.json | ||
| policy_templates.json | ||
| prometheus.yml | ||
| provider_endpoints_support.json | ||
| proxy_server_config.yaml | ||
| pyproject.toml | ||
| pyrightconfig.json | ||
| qa_sticky_session.sh | ||
| README.md | ||
| render.yaml | ||
| router_plugins.json | ||
| ruff-strict.toml | ||
| ruff-tests.toml | ||
| ruff.toml | ||
| rust-toolchain.toml | ||
| schema.prisma | ||
| SECURITY.md | ||
| taplo.toml | ||
| uv.lock | ||
| whitelisted_bedrock_models.txt | ||
🚅 LiteLLM
LiteLLM AI Gateway
Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website
What is LiteLLM
LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.
Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
Why LiteLLM
Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:
- Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
- Drop-in OpenAI compatibility — swap providers without rewriting your code
- Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
- 8ms P95 latency at 1k RPS (benchmarks)
OSS Adopters
Netflix |
Features
LLMs - Call 100+ LLMs (Python SDK + AI Gateway)
All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.
Python SDK
uv add litellm
An independent litellm-core distribution provides the Python SDK with the same
import litellm API and runtime dependencies. It has no optional extras, CLI entry
points, or bundled dashboard. Install one SDK distribution per environment because
litellm and litellm-core own overlapping Python files. Use litellm for the
proxy, CLI, and optional extras
Build and install core from this checkout while its release integration is pending:
python scripts/build_core_distribution.py --out-dir dist/core
python -m pip install dist/core/litellm_core-*.whl
The builder requires Git, uv and the Rust build toolchain. It reads the release version
from the root pyproject.toml, leaves source files unchanged, and produces a wheel
and a self-contained sdist in dist/core. Run installation in a fresh environment
without litellm
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
AI Gateway (Proxy Server)
Getting Started - E2E Tutorial - Setup virtual keys, make your first request
uv tool install 'litellm[proxy]'
litellm --model gpt-4o
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Agents - Invoke A2A Agents (Python SDK + AI Gateway)
Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
Python SDK - A2A Protocol
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
AI Gateway (Proxy Server)
Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent
Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)
import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4
base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer <your-master-key>"} # LiteLLM master key or a virtual key
async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
config = ClientConfig(
httpx_client=http_client,
streaming=False,
supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
)
client = ClientFactory(config).create(agent_card)
request = SendMessageRequest(
message=Message(
message_id=uuid4().hex,
role=Role.ROLE_USER,
parts=[Part(text="Hello!")],
)
)
async for event in client.send_message(request):
populated = event.ListFields()
if populated and populated[0][0].name in ("message", "msg"):
print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))
MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)
Python SDK - MCP Bridge
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Load MCP tools in OpenAI format
tools = await mcp_client.load_mcp_tools(session=session, format="openai")
# Use with any LiteLLM model
response = await litellm.acompletion(
model="gpt-4o",
messages=[{"role": "user", "content": "What's 3 + 5?"}],
tools=tools
)
AI Gateway - MCP Gateway
Step 1. Add your MCP Server to the AI Gateway
Step 2. Call MCP tools via /chat/completions
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Authorization: Bearer <your-master-key>' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize the latest open PR"}],
"tools": [{
"type": "mcp",
"server_url": "litellm_proxy/mcp/github",
"server_label": "github_mcp",
"require_approval": "never"
}]
}'
Use with Cursor IDE
{
"mcpServers": {
"LiteLLM": {
"url": "http://localhost:4000/mcp/",
"headers": {
"x-litellm-api-key": "Bearer <your-master-key>"
}
}
}
}
For MCP OAuth, an upstream may advertise dynamic client registration but refuse requests with HTTP 401 or 403. If the provider requires a pre-registered OAuth app, configure its credentials.client_id and, when required, credentials.client_secret on the MCP server. This skips dynamic registration in the gateway sign-in flow. The provider must approve the app for MCP access; reaching its authorization page does not establish that login or tool calls will succeed
Agents - Run Claude Code, Codex, OpenCode, pi or Deep Agents on any model (Python SDK)
Python SDK - Agents
import litellm
from litellm import Harness, sandbox
result = litellm.agent(
Harness.CLAUDE_CODE, # or Harness.CODEX, Harness.OPENCODE, Harness.PI, Harness.DEEPAGENTS
"Find why tests/test_router.py is flaky and fix it.",
sandbox=sandbox.local("./repo"),
model="litellm_proxy/claude-sonnet-4-5", # a model group on your AI Gateway
)
print(result.text, result.cost, [f.path for f in result.files])
Set LITELLM_PROXY_API_BASE and LITELLM_PROXY_API_KEY and every model call the agent makes goes through your AI Gateway, tagged harness,claude_code. Drop the litellm_proxy/ prefix to call a provider directly. Install starlette uvicorn plus the agent's CLI (claude, codex, opencode or pi), or deepagents langchain-litellm for Deep Agents.
Supported Providers (Website Supported Models | Docs)
Get Started
You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
| LiteLLM AI Gateway | LiteLLM Python SDK | |
|---|---|---|
| Use Case | Central service (LLM Gateway) to access multiple LLMs | Use LiteLLM directly in your Python code |
| Who Uses It? | Gen AI Enablement / ML Platform Teams | Developers building LLM projects |
| Key Features | Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management | Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.) |
Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Deploy on AWS or GCP with Terraform
Run the LiteLLM proxy as a production-ready componentized stack (gateway, backend, UI on separate services; managed Postgres + Redis + object store) using the published Terraform modules. Both modules are on the public Terraform Registry — no auth needed.
AWS — ECS Fargate + Aurora + ElastiCache + ALB
git clone https://github.com/BerriAI/litellm.git
cd litellm/terraform/litellm/aws/examples/default
cp terraform.tfvars.example terraform.tfvars # edit region/tenant/env
terraform init && terraform apply
Or call the module from your own root config:
# main.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.60" }
}
}
provider "aws" {
region = "us-west-2"
}
module "litellm" {
source = "BerriAI/litellm/aws"
version = "~> 1.89"
region = "us-west-2"
azs = ["us-west-2a", "us-west-2b"]
tenant = "acme"
env = "prod"
# Production: provide an ACM cert. Without one, set allow_plaintext_alb = true
# (dev/trial only).
# acm_certificate_arn = "arn:aws:acm:us-west-2:111122223333:certificate/..."
allow_plaintext_alb = true
}
output "litellm_url" {
value = module.litellm.alb_dns_name
}
terraform init
terraform apply
Provider API keys live in AWS Secrets Manager; reference ARNs via gateway_extra_secrets. Full input list and architecture diagram on the registry page.
GCP — Cloud Run + Cloud SQL + Memorystore + HTTPS LB
Real 1-click. Opens Cloud Shell, clones the terraform-google-litellm module mirror at the latest stable release (a small download instead of the full litellm repo), and walks you through terraform apply via a built-in DeployStack tutorial. Pick the project, the tutorial sets up the Artifact Registry remote repo, writes terraform.tfvars from your answers, and runs apply
To call the module from your own config instead, Cloud Run can't pull from ghcr.io directly, so first set up a one-time Artifact Registry remote repo backed by GHCR:
gcloud artifacts repositories create litellm \
--location=us-central1 \
--repository-format=docker \
--mode=remote-repository \
--remote-docker-repo=https://ghcr.io \
--project=my-gcp-project
Then:
# main.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
google = { source = "hashicorp/google", version = "~> 6.10" }
google-beta = { source = "hashicorp/google-beta", version = "~> 6.10" }
}
}
provider "google" { project = "my-gcp-project"; region = "us-central1" }
provider "google-beta" { project = "my-gcp-project"; region = "us-central1" }
module "litellm" {
source = "BerriAI/litellm/google"
version = "~> 1.89"
project_id = "my-gcp-project"
region = "us-central1"
tenant = "acme"
env = "prod"
# Replace my-gcp-project with your GCP project ID (same value as project_id above).
image_registry = "us-central1-docker.pkg.dev/my-gcp-project/litellm/berriai"
# Production: provide DNS already pointing at the LB IP for Google-managed certs.
# Without one, set allow_plaintext_lb = true (dev/trial only).
# lb_domains = ["proxy.example.com"]
allow_plaintext_lb = true
}
output "litellm_url" {
value = module.litellm.load_balancer_url
}
terraform init
terraform apply
Provider API keys live in Secret Manager; reference resource IDs (e.g. projects/my-gcp-project/secrets/openai-api-key) via gateway_extra_secrets. Full input list and architecture diagram on the registry page.
Both stacks include
- The full componentized split (gateway / backend / UI as independent services)
- Managed Postgres (writer + reader) and Redis
- Versioned object store for proxy state + file uploads
- An auto-generated
LITELLM_MASTER_KEYin your cloud's secret manager - A one-off migration job that runs
prisma migrate deploybefore the proxy starts - The same
proxy_configsurface as the Helm chart — pass YAML as a typed map
The Terraform modules live at terraform/litellm/aws/ and terraform/litellm/gcp/ in this repo; the registry entries are read-only mirrors updated on each release.
Run in Developer Mode
Services
- Setup .env file in root
- Run dependent services
docker-compose up db prometheus
Backend
- Run
make bootstrap - Start proxy backend:
uv run python litellm/proxy/proxy_cli.py
Frontend
- Navigate to
ui/litellm-dashboard(dependencies were already installed w/make bootstrap) - Start dashboard:
npm run dev
Verify Docker Image Signatures
All LiteLLM Docker images published to GHCR are signed with cosign. Releases are signed with the key introduced in commit e82a3c6.
Verify using the pinned commit hash (recommended):
A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the published signing key:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Verify using a release tag (convenience):
Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).
Enterprise
For companies that need better security, user management and professional support
Get an Enterprise License Talk to founders
This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On
Contributing
We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
If you want to work closely with our team, check out LiteLLM OSS Champions, our program for trusted OSS contributors with their own Slack channel and a voice in the roadmap
Quick Start for Contributors
This requires uv to be installed.
git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev # Install development dependencies
make format # Format your code
make lint # Run all linting checks
make test-unit # Run unit tests
make format-check # Check formatting only
For detailed contributing guidelines, see CONTRIBUTING.md.
📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
Our automated checks include:
- Ruff for formatting, linting, and code quality
- basedpyright for type checking
- Circular import detection
- Import safety checks
All these checks must pass before your PR can be merged.
Support / talk with founders
- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai