0ca29c7d84
* feat(harness): add pi harness for litellm.agent()
Adds Harness.PI, which drives the pi coding agent CLI (`pi --mode json`) once per turn
against the per-session model endpoint. The endpoint is declared as a `litellm`
openai-completions provider in the models.json of a LiteLLM-owned PI_CODING_AGENT_DIR, so
the user's own pi settings, auth, extensions and MCP servers are never read, and the token
travels only through an env var referenced as `$LITELLM_HARNESS_TOKEN`
pi exits 0 after a failed provider call, so a turn fails when the last assistant message
ends with stopReason error/aborted. Permission modes map to a `--tools` allowlist,
disable_tools to `--exclude-tools`, and sessions resume with `--session` from a persisted
session dir. build_instructions and turn_prompt move from the OpenCode config into the
shared harness utils so both configs use them
Unit tests run on JSONL recorded from pi 1.1.0
* test(harness): align pi harness tests with the other CLI harness suites
Mirror the OpenCode suite's layout, fakes and helpers, and add the cases every CLI harness
covers: turn before start, early close kills the process, persist failure, endpoint model
fallback, long JSONL lines, skills without SKILL.md and a missing endpoint. Add an
endpoint_requests.jsonl fixture recording what pi sends to the session endpoint, and
re-record the turn fixtures in one run so their ids agree
* fix(harness): parametrize harness e2e only over harnesses with a model group
harness_params() iterated every Harness member, so Harness.TOOL_LOOP, which has no entry in
BINARIES or DEFAULT_MODEL_GROUPS, raised KeyError and the whole suite failed to collect
* feat(harness): accept pi config and MCP servers through PiOptions.config
PiOptions gains `config`, like the other harness options. `mcpServers` is written to the
private agent dir's mcp.json and every other key to its settings.json. Keys that would reroute
model calls, bypass permissions=, or load code as the host user (defaultProvider, defaultModel,
defaultTools, defaultProjectTrust, extensions, packages, ...) raise OptionsMismatch
A plain-name `--tools` allowlist drops MCP tools in pi 1.1.0, so when servers are configured
the allowlist also gets `mcp__*` plus the tool each exposure is reached through (`codemode` by
default, `tool_search` for `deferred`). disable_tools still excludes individual MCP tools
* refactor(harness): give harness options immutable mapping defaults
`field(default_factory=dict)` left every config/env default typed as
`dict[Unknown, Unknown]`, which basedpyright reports as
reportUnknownVariableType under strict mode: 9 errors across the five
options classes, two of them added by PiOptions. A shared module-level
`MappingProxyType({})` default is accepted by frozen dataclasses, matches
the no-mutation rule the rest of the harness follows, and drops the file
to zero errors so the rule shrinks against the merge base instead of
growing.
* fix(harness): reject managed env keys and a missing model in the pi harness
PiOptions.env could name PI_CODING_AGENT_DIR or LITELLM_HARNESS_TOKEN and
the value was dropped in silence, because transform_session_setup merges
the managed keys last. The ordering stays as defense in depth, but
validate_environment now rejects those keys up front the way the Claude
Code config does, so a bad option is an error rather than a setting that
appears to apply and does not.
transform_turn_request also stringified the model it resolved, so a call
with neither model= nor an endpoint model would have put the literal
"None" on pi's --model. Both call sites now go through session_model(),
which raises the same ValueError transform_session_setup already raised.
* test(harness): escape the regex dot in the pi skill manifest match
* fix(harness): keep harness options importable on Python 3.11
Python 3.11 rejects an unhashable dataclass default, and MappingProxyType
only became hashable in 3.12, so the shared MappingProxyType({}) defaults
made importing litellm.harness.options raise ValueError and broke every
harness on 3.11. The defaults now come from typed default factories that
still return read-only mappings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(harness): address review feedback on the pi harness tests
Drop the endpoint-request fixture test, which only read a recorded file and
could not fail when PiHarnessConfig changed, along with its now unused
fixture. Remove the section-divider comments and a comment that restated
MANAGED_ENV_KEYS, and type the test helpers and fakes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(harness): cover the last uncovered pi harness lines
Assert that a tool_execution_end event without a result yields an empty
ToolResult output, and that a PiOptions.config value JSON cannot encode
raises TypeError. The Python 3.11 default check now asserts on the list of
unhashable defaults instead of relying on hash() raising, which the
test-quality gate flagged as an assertion-free test
* test(harness): annotate harness option tests and stop rebinding helper params
Mark the option test locals Final and check read-only defaults through the
MutableMapping interface instead of writing into them. In the pi suite,
type the parametrized test arguments, and have started() and parse_all()
bind new names instead of reassigning their sandbox and state parameters
* test(harness): test harness option defaults by constructing the options
Replace the field-introspection check with one that builds each options
class twice and asserts every mapping default is empty, read-only and not
shared between instances. Sharing one MappingProxyType across instances is
what broke class creation on Python 3.11, and that sharing is observable on
every Python version, so the test fails on the old code without inspecting
dataclass declarations
* test(harness): stop mutating test helper inputs and assert whole values
make_ctx took model and endpoint out of **kwargs with pop, which mutated
its own parameter; they are explicit parameters now, with a frozen default
endpoint. assistant_end builds its message in one expression instead of
adding errorMessage afterwards. Two docstrings that explained an assertion
became that assertion's message, and the options test compares whole
values so a failure names the offending fields
* ci(unit): run the pi harness unit tests in the llms-h-to-z shard
main now lists every tests/unit/llms provider directory per shard, so the new
tests/unit/llms/pi directory was claimed by no shard and assert_ci_coverage
failed on both the shard and the test-file checks
* test(harness): mark pi test locals Final and freeze the test output model
Every single-assignment local and module constant in the pi suite is now
Final. The name-mapping loop became one comprehension compared as a whole
list, test_start_persists_sessions_dir takes the sandbox by index instead of
binding _ twice, and the Answer output model is frozen like every other
pydantic model
* fix(harness): send PiOptions.thinking to the gateway as reasoning_effort
pi clamps --thinking to "off" for a model that does not declare reasoning,
and the generated models.json only declared {"id": model}, so every
thinking level was dropped and the gateway never saw reasoning_effort.
xhigh and max are also only offered when thinkingLevelMap names them. The
model now declares reasoning plus that map whenever a level other than
"off" is requested, and nothing when thinking is unset, so pi does not send
its default effort on every call
The pi tests also drop **kwargs from make_ctx and started in favour of
explicit parameters, and annotate their helpers and the fake sandbox with
read-only types; the fake records calls by rebinding tuples instead of
appending to lists
* docs(harness): list pi among the harnesses in the module and agent() docstrings
* fix(harness): keep pi calls provider-agnostic and harness option defaults copyable
Declare supportsStore false so pi stops sending store, take contextWindow and maxTokens from the model map, treat a length stop or a silent exit as an error, and give every options class plain dict defaults again so deepcopy, pickle and asdict work
* fix(harness): look up pi gateway groups as litellm_proxy and ask pi for cache_control marks
A gateway session's model is the bare group name, so the model map lookup
now passes custom_llm_provider=litellm_proxy, the way the SDK prices it.
That keeps an unmapped group from printing get_llm_provider's provider
list banner into the caller's stdout.
pi only adds Anthropic cache_control marks for OpenRouter Anthropic models,
so every Claude call through the harness missed the prompt cache. Models
the map says support prompt caching now declare cacheControlFormat
anthropic, which LiteLLM translates for each provider.
* fix(harness): cap pi's maxTokens at its own default and fork sessions saved in another directory
pi sends maxTokens as max_completion_tokens on every call, and the gateway reserves that many
tokens against the key's TPM limit, so passing the model's full output cap (128000 for
claude-haiku-5-5) made keys with lower limits answer 429. maxTokens is now the lower of the
model's cap and pi's own 16384 default.
agent_resume() into another sandbox directory stopped at pi's interactive fork prompt. The
native session id now carries the directory pi saved the session in, and a resume from a
different directory passes --fork, so the conversation continues in the new sandbox.
* fix(harness): leave pi room for the thinking budget and price cached prompt tokens
pi's maxTokens now adds LiteLLM's thinking budget for the requested level to pi's 16384 default, within the model's output cap, so Anthropic budget thinking at max no longer answers 400. Harness turns now pass cache read and write tokens to cost_per_token, so cached prompts are priced at their cache rates on every harness.
* test(harness): cover cached prompt cost on the gateway and non-stream routes
* test(harness): mark the cost test constants Final
* test(harness): type the cached cost test helpers without Any
* fix(harness): price OpenAI cache write tokens in harness cost
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
38 KiB
🚅 LiteLLM
LiteLLM AI Gateway
Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website
What is LiteLLM
LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.
Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
Why LiteLLM
Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:
- Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
- Drop-in OpenAI compatibility — swap providers without rewriting your code
- Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
- 8ms P95 latency at 1k RPS (benchmarks)
OSS Adopters
Netflix |
Features
LLMs - Call 100+ LLMs (Python SDK + AI Gateway)
All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.
Python SDK
uv add litellm
An independent litellm-core distribution provides the Python SDK with the same
import litellm API and runtime dependencies. It has no optional extras, CLI entry
points, or bundled dashboard. Install one SDK distribution per environment because
litellm and litellm-core own overlapping Python files. Use litellm for the
proxy, CLI, and optional extras
Build and install core from this checkout while its release integration is pending:
python scripts/build_core_distribution.py --out-dir dist/core
python -m pip install dist/core/litellm_core-*.whl
The builder requires Git, uv and the Rust build toolchain. It reads the release version
from the root pyproject.toml, leaves source files unchanged, and produces a wheel
and a self-contained sdist in dist/core. Run installation in a fresh environment
without litellm
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
AI Gateway (Proxy Server)
Getting Started - E2E Tutorial - Setup virtual keys, make your first request
uv tool install 'litellm[proxy]'
litellm --model gpt-4o
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Agents - Invoke A2A Agents (Python SDK + AI Gateway)
Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
Python SDK - A2A Protocol
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
AI Gateway (Proxy Server)
Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent
Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)
import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4
base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer <your-master-key>"} # LiteLLM master key or a virtual key
async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
config = ClientConfig(
httpx_client=http_client,
streaming=False,
supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
)
client = ClientFactory(config).create(agent_card)
request = SendMessageRequest(
message=Message(
message_id=uuid4().hex,
role=Role.ROLE_USER,
parts=[Part(text="Hello!")],
)
)
async for event in client.send_message(request):
populated = event.ListFields()
if populated and populated[0][0].name in ("message", "msg"):
print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))
MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)
Python SDK - MCP Bridge
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Load MCP tools in OpenAI format
tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
# Use with any LiteLLM model
response = await litellm.acompletion(
model="gpt-4o",
messages=[{"role": "user", "content": "What's 3 + 5?"}],
tools=tools
)
AI Gateway - MCP Gateway
Step 1. Add your MCP Server to the AI Gateway
Step 2. Call MCP tools via /chat/completions
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Authorization: Bearer <your-master-key>' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize the latest open PR"}],
"tools": [{
"type": "mcp",
"server_url": "litellm_proxy/mcp/github",
"server_label": "github_mcp",
"require_approval": "never"
}]
}'
Use with Cursor IDE
{
"mcpServers": {
"LiteLLM": {
"url": "http://localhost:4000/mcp/",
"headers": {
"x-litellm-api-key": "Bearer <your-master-key>"
}
}
}
}
For MCP OAuth, an upstream may advertise dynamic client registration but refuse requests with HTTP 401 or 403. If the provider requires a pre-registered OAuth app, configure its credentials.client_id and, when required, credentials.client_secret on the MCP server. This skips dynamic registration in the gateway sign-in flow. The provider must approve the app for MCP access; reaching its authorization page does not establish that login or tool calls will succeed
Agents - Run Claude Code, Codex, OpenCode, pi or Deep Agents on any model (Python SDK)
Python SDK - Agents
import litellm
from litellm import Harness, sandbox
result = litellm.agent(
Harness.CLAUDE_CODE, # or Harness.CODEX, Harness.OPENCODE, Harness.PI, Harness.DEEPAGENTS
"Find why tests/test_router.py is flaky and fix it.",
sandbox=sandbox.local("./repo"),
model="litellm_proxy/claude-sonnet-4-5", # a model group on your AI Gateway
)
print(result.text, result.cost, [f.path for f in result.files])
Set LITELLM_PROXY_API_BASE and LITELLM_PROXY_API_KEY and every model call the agent makes goes through your AI Gateway, tagged harness,claude_code. Drop the litellm_proxy/ prefix to call a provider directly. Install starlette uvicorn plus the agent's CLI (claude, codex, opencode or pi), or deepagents langchain-litellm for Deep Agents.
Supported Providers (Website Supported Models | Docs)
Get Started
You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
| LiteLLM AI Gateway | LiteLLM Python SDK | |
|---|---|---|
| Use Case | Central service (LLM Gateway) to access multiple LLMs | Use LiteLLM directly in your Python code |
| Who Uses It? | Gen AI Enablement / ML Platform Teams | Developers building LLM projects |
| Key Features | Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management | Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.) |
Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Deploy on AWS or GCP with Terraform
Run the LiteLLM proxy as a production-ready componentized stack (gateway, backend, UI on separate services; managed Postgres + Redis + object store) using the published Terraform modules. Both modules are on the public Terraform Registry — no auth needed.
AWS — ECS Fargate + Aurora + ElastiCache + ALB
git clone https://github.com/BerriAI/litellm.git
cd litellm/terraform/litellm/aws/examples/default
cp terraform.tfvars.example terraform.tfvars # edit region/tenant/env
terraform init && terraform apply
Or call the module from your own root config:
# main.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.60" }
}
}
provider "aws" {
region = "us-west-2"
}
module "litellm" {
source = "BerriAI/litellm/aws"
version = "~> 1.89"
region = "us-west-2"
azs = ["us-west-2a", "us-west-2b"]
tenant = "acme"
env = "prod"
# Production: provide an ACM cert. Without one, set allow_plaintext_alb = true
# (dev/trial only).
# acm_certificate_arn = "arn:aws:acm:us-west-2:111122223333:certificate/..."
allow_plaintext_alb = true
}
output "litellm_url" {
value = module.litellm.alb_dns_name
}
terraform init
terraform apply
Provider API keys live in AWS Secrets Manager; reference ARNs via gateway_extra_secrets. Full input list and architecture diagram on the registry page.
GCP — Cloud Run + Cloud SQL + Memorystore + HTTPS LB
Real 1-click. Opens Cloud Shell, clones the terraform-google-litellm module mirror at the latest stable release (a small download instead of the full litellm repo), and walks you through terraform apply via a built-in DeployStack tutorial. Pick the project, the tutorial sets up the Artifact Registry remote repo, writes terraform.tfvars from your answers, and runs apply
To call the module from your own config instead, Cloud Run can't pull from ghcr.io directly, so first set up a one-time Artifact Registry remote repo backed by GHCR:
gcloud artifacts repositories create litellm \
--location=us-central1 \
--repository-format=docker \
--mode=remote-repository \
--remote-docker-repo=https://ghcr.io \
--project=my-gcp-project
Then:
# main.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
google = { source = "hashicorp/google", version = "~> 6.10" }
google-beta = { source = "hashicorp/google-beta", version = "~> 6.10" }
}
}
provider "google" { project = "my-gcp-project"; region = "us-central1" }
provider "google-beta" { project = "my-gcp-project"; region = "us-central1" }
module "litellm" {
source = "BerriAI/litellm/google"
version = "~> 1.89"
project_id = "my-gcp-project"
region = "us-central1"
tenant = "acme"
env = "prod"
# Replace my-gcp-project with your GCP project ID (same value as project_id above).
image_registry = "us-central1-docker.pkg.dev/my-gcp-project/litellm/berriai"
# Production: provide DNS already pointing at the LB IP for Google-managed certs.
# Without one, set allow_plaintext_lb = true (dev/trial only).
# lb_domains = ["proxy.example.com"]
allow_plaintext_lb = true
}
output "litellm_url" {
value = module.litellm.load_balancer_url
}
terraform init
terraform apply
Provider API keys live in Secret Manager; reference resource IDs (e.g. projects/my-gcp-project/secrets/openai-api-key) via gateway_extra_secrets. Full input list and architecture diagram on the registry page.
Both stacks include
- The full componentized split (gateway / backend / UI as independent services)
- Managed Postgres (writer + reader) and Redis
- Versioned object store for proxy state + file uploads
- An auto-generated
LITELLM_MASTER_KEYin your cloud's secret manager - A one-off migration job that runs
prisma migrate deploybefore the proxy starts - The same
proxy_configsurface as the Helm chart — pass YAML as a typed map
The Terraform modules live at terraform/litellm/aws/ and terraform/litellm/gcp/ in this repo; the registry entries are read-only mirrors updated on each release.
Run in Developer Mode
Services
- Setup .env file in root
- Run dependent services
docker-compose up db prometheus
Backend
- Run
make bootstrap - Start proxy backend:
uv run python litellm/proxy/proxy_cli.py
Frontend
- Navigate to
ui/litellm-dashboard(dependencies were already installed w/make bootstrap) - Start dashboard:
npm run dev
Verify Docker Image Signatures
All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.
Verify using the pinned commit hash (recommended):
A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Verify using a release tag (convenience):
Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).
Enterprise
For companies that need better security, user management and professional support
Get an Enterprise License Talk to founders
This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On
Contributing
We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
Quick Start for Contributors
This requires uv to be installed.
git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev # Install development dependencies
make format # Format your code
make lint # Run all linting checks
make test-unit # Run unit tests
make format-check # Check formatting only
For detailed contributing guidelines, see CONTRIBUTING.md.
📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
Our automated checks include:
- Ruff for formatting, linting, and code quality
- basedpyright for type checking
- Circular import detection
- Import safety checks
All these checks must pass before your PR can be merged.
Support / talk with founders
- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai