Find a file
Mateo Wang 75faaeab64
feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777)
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response

Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.

* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests

* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing

* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown

Moves the cost map lookup off the event loop

* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty

* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing

A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.

* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates

* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment

A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.

* test(count_tokens): use the shared integration port allocator in the fallback wire test

* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today

* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings

Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it

* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body

A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.

* fix(count_tokens): answer the provider's error whenever a framed estimate raises

A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.

* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate

A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.

* fix(count_tokens): look up framing under the deployment's custom_llm_provider

* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error

A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.

* fix(count_tokens): keep one tokenizer download per key running and load the local route per request

* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible

* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error

Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500

* feat(proxy): count tokens with token_count_model in its pinned region

token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable

* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys

* fix(proxy): keep the upstream error envelope when adding the region hint

* test(proxy): annotate the token count setting echo test

* ci(cost map): require non-blank token_count_model strings and test the region rules

* fix(count_tokens): answer the token_count_model's error even when framing is set

* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate

* fix(count_tokens): estimate Google contents the route translated to messages

* fix(count_tokens): keep the provider error for Google contents with non-text parts

* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock

* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags

* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer

* test(bedrock): keep the Kimi K3 image check in-process

* fix(count_tokens): join the region hint to an upstream message that already ends with a period

Also pins that a blank token_count_model in config falls through to the cost map default

* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts

The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts

The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool

A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning

* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them

A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.

* test(count_tokens): annotate new token counter test locals with Final

* fix(count_tokens): keep shipped count regions inside the deployment's geography

A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added

* fix(count_tokens): resolve count settings off the event loop
2026-10-17 17:11:27 -07:00
.cargo feat(rust): add litellm-db and litellm-db-testing workspace scaffolding (#43504) 2026-09-27 18:48:45 -07:00
.circleci ci: run mcp interactions and redis cluster iam integration tests only in circleci (#46780) 2026-10-15 16:53:36 -07:00
.devcontainer build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
.githooks security(proxy): keep team callback credentials out of the stored request body (#43217) 2026-09-28 11:16:58 -07:00
.github test(mcp): replace the live Linear MCP OAuth e2e tests with integration coverage (#47154) 2026-10-17 14:29:59 -07:00
.semgrep/rules security: remove .claude/settings.json and add semgrep rule to prevent re-adding 2026-03-25 11:57:43 -07:00
backend fix(backend): retain Prometheus metrics during startup (#47129) 2026-10-17 17:10:42 +00:00
ci_cd ci(cost map): remove the generated model prices JSON schema (#47064) 2026-10-17 05:46:31 +00:00
cookbook fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344) 2026-10-16 00:53:52 +00:00
db_scripts fix(vector-stores): keep proxy context out of provider vector store metadata (#47021) 2026-10-17 12:51:47 -07:00
docker feat(docker): stop, start, and uninstall a quickstart install with litellm-gateway (#46486) 2026-10-15 10:08:06 -07:00
enterprise fix(batches): preserve spend across malformed output and metadata failures (#47184) 2026-10-17 17:10:22 -07:00
examples chore: litellm oss staging (#31185) 2026-06-26 09:17:44 -07:00
gateway build(docker): cache the Rust extension build in its own stage (#47033) 2026-10-16 20:49:53 -07:00
helm feat(helm): add db.useIAMAuth and db.useAzureEntraAuth token auth to litellm-helm (#46534) 2026-10-15 15:33:51 -07:00
litellm feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) 2026-10-17 17:11:27 -07:00
litellm-proxy-extras fix(proxy): unset DB environment variables removed from the row on every worker by row revision (#46709) 2026-10-17 03:51:05 +00:00
litellm-rust feat(openinfer): add OpenInfer as an OpenAI-compatible provider (#41034) 2026-10-17 23:22:19 +00:00
litellm_cli perf(cli): start lite without initializing the litellm SDK (#46803) 2026-10-17 04:53:32 +00:00
migrations build(docker): cache the Rust extension build in its own stage (#47033) 2026-10-16 20:49:53 -07:00
packaging fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
scripts test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
terraform feat(terraform)!: default the component images to the module release tag (#46960) 2026-10-17 10:23:21 -07:00
tests feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) 2026-10-17 17:11:27 -07:00
ui feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) 2026-10-17 17:11:27 -07:00
vscode-extension fix(deps): bump source-map-js, smol-toml, mako, multidict and werkzeug for OSV advisories (#44728) 2026-10-06 00:57:04 +00:00
.dockerignore build(docker): build the Admin UI from source in a build-platform-pinned stage (#31130) 2026-06-25 23:41:08 -07:00
.env.example docs: stop advertising sk-1234 as the master key in shipped configs and examples 2026-09-19 12:59:48 -07:00
.git-blame-ignore-revs chore: ignore the mechanical lint and typing sweeps in git blame 2026-08-06 11:39:34 +00:00
.gitattributes feat(ui): generate dashboard API types from the proxy OpenAPI spec (#29816) 2026-06-05 17:20:01 -07:00
.gitguardian.yaml fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
.gitignore chore: ignore .mypy_cache (#45628) 2026-10-09 18:26:19 +00:00
.grype.yaml ci(image-scan): ignore zlib CVE-2026-85091 until Wolfi ships the fix 2026-09-15 19:23:21 -07:00
.npmrc [Fix] CI/Tooling: Correct min-release-age value in .npmrc files 2026-04-29 19:49:27 -07:00
AGENTS.md docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI (#46796) 2026-10-16 14:57:49 -07:00
ARCHITECTURE.md refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
codecov.yaml test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
CONTRIBUTING.md chore(deps): drop unused pytest-postgresql dev dependency (#44056) 2026-10-01 18:53:54 +00:00
cosign.pub [Infra] Add release workflow and cosign public key 2026-03-31 14:30:27 -07:00
docker-compose.hardened.yml build(docker): drop the no-op PROXY_EXTRAS_SOURCE switch from the non-root image (#44097) 2026-10-01 17:06:08 -07:00
docker-compose.liteadmin.yml feat(enterprise): bundle LiteAdmin Slack with native gateway login (#44444) 2026-10-03 17:28:18 -07:00
docker-compose.yml feat: add read-replica routing for Prisma DB via DATABASE_URL_READ_REPLICA (#27493) 2026-05-08 21:05:50 -07:00
Dockerfile refactor(lens): connect LiteLLM to the independent Lens service (#45529) 2026-10-09 16:17:05 -07:00
GEMINI.md chore: consolidate CLAUDE.md into AGENTS.md 2026-09-19 02:30:35 +00:00
LICENSE refactor: creating enterprise folder 2024-02-15 12:54:13 -08:00
license_cache.json Add granian as a ASGI compliant web server. Provider better throughput stability, (#26027) 2026-05-21 19:08:37 -07:00
Makefile test(rust-bridge): select native routes per test without LITELLM_RUST (#46620) 2026-10-15 09:12:08 -07:00
mcp_servers.json Add ScrapeGraph MCP server configuration (#18923) 2026-01-11 21:57:46 +05:30
model_prices_and_context_window.json feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) 2026-10-17 17:11:27 -07:00
model_prices_and_context_window.schema.json feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) 2026-10-17 17:11:27 -07:00
osv-scanner.toml build(deps): bump mlflow to 3.17.0 and drop GHSA-h7x2-h6g9-p789 suppression (#45777) 2026-10-13 22:24:17 -07:00
package-lock.json chore(deps): refresh dependency locks 2026-05-04 11:36:18 -07:00
package.json chore(deps): refresh dependency locks 2026-05-04 11:36:18 -07:00
policy_templates.json fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
prometheus.yml build(docker-compose.yml): add prometheus scraper to docker compose 2024-07-24 10:09:23 -07:00
provider_endpoints_support.json feat(openinfer): add OpenInfer as an OpenAI-compatible provider (#41034) 2026-10-17 23:22:19 +00:00
proxy_server_config.yaml test(ci): refresh qualified retired OpenAI fixtures (#43938) 2026-09-30 16:10:18 -07:00
pyproject.toml perf(cli): start lite without initializing the litellm SDK (#46803) 2026-10-17 04:53:32 +00:00
pyrightconfig.json perf(cli): start lite without initializing the litellm SDK (#46803) 2026-10-17 04:53:32 +00:00
qa_sticky_session.sh fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
README.md feat: add Nace Drex decisions provider (#46523) 2026-10-16 18:07:50 -07:00
render.yaml feat(proxy)!: refuse to start with an unset, empty, or publicly known master key 2026-09-19 13:44:00 -07:00
router_plugins.json feat(router): add router plugin reference catalog (#33746) 2026-07-17 18:46:20 +00:00
ruff-strict.toml feat(guardrails): add Airia guardrail integration (#40784) 2026-10-14 15:45:47 -07:00
ruff-tests.toml test: gate the test tree on fifteen assertion and handler rules it already satisfies (#38361) 2026-08-26 16:05:34 -07:00
ruff.toml ci(lint): gate every rule on its merge-base count and cap Anys at a fixed total (#40193) 2026-10-08 05:39:29 +00:00
rust-toolchain.toml fix(ci): pin workflow toolchain dependencies 2026-09-02 12:16:25 -07:00
schema.prisma fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys (#45743) 2026-10-16 12:00:08 +00:00
security.md docs(security): point readers to the security announcements mailing list signup (#43713) 2026-09-29 13:15:37 +00:00
taplo.toml fix(agentcore): simplify agentcore streaming (#17141) 2026-01-19 05:20:24 -08:00
uv.lock build(deps): bump claude-agent-sdk to 0.2.165 to unblock osv-scan (#46971) 2026-10-16 14:05:32 -07:00
whitelisted_bedrock_models.txt fix(bedrock): add the bare Pegasus 1.5 row and take GPT-5.x context windows from the model cards (#45621) 2026-10-09 11:06:54 -07:00

🚅 LiteLLM

LiteLLM AI Gateway

Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.

Deploy to Render Deploy on Railway Deploy on AWS Deploy on GCP

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website

PyPI Version GitHub Stars Y Combinator W23 Whatsapp Discord Slack CodSpeed

LiteLLM AI Gateway

What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers


Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

  • Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
  • Drop-in OpenAI compatibility — swap providers without rewriting your code
  • Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
  • 8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters

Stripe image Google ADK Greptile OpenHands

Netflix

OpenAI Agents SDK

Features

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

uv add litellm

An independent litellm-core distribution provides the Python SDK with the same import litellm API and runtime dependencies. It has no optional extras, CLI entry points, or bundled dashboard. Install one SDK distribution per environment because litellm and litellm-core own overlapping Python files. Use litellm for the proxy, CLI, and optional extras

Build and install core from this checkout while its release integration is pending:

python scripts/build_core_distribution.py --out-dir dist/core
python -m pip install dist/core/litellm_core-*.whl

The builder requires Git, uv and the Rust build toolchain. It reads the release version from the root pyproject.toml, leaves source files unchanged, and produces a wheel and a self-contained sdist in dist/core. Run installation in a fresh environment without litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o
import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

Agents - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent

Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)

import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer <your-master-key>"}    # LiteLLM master key or a virtual key

async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
    resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    config = ClientConfig(
        httpx_client=http_client,
        streaming=False,
        supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
    )
    client = ClientFactory(config).create(agent_card)

    request = SendMessageRequest(
        message=Message(
            message_id=uuid4().hex,
            role=Role.ROLE_USER,
            parts=[Part(text="Hello!")],
        )
    )
    async for event in client.send_message(request):
        populated = event.ListFields()
        if populated and populated[0][0].name in ("message", "msg"):
            print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer <your-master-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer <your-master-key>"
      }
    }
  }
}

For MCP OAuth, an upstream may advertise dynamic client registration but refuse requests with HTTP 401 or 403. If the provider requires a pre-registered OAuth app, configure its credentials.client_id and, when required, credentials.client_secret on the MCP server. This skips dynamic registration in the gateway sign-in flow. The provider must approve the app for MCP access; reaching its authorization page does not establish that login or tool calls will succeed

Docs: MCP Gateway

Agents - Run Claude Code, Codex, OpenCode or Deep Agents on any model (Python SDK)

Python SDK - Agents

import litellm
from litellm import Harness, sandbox

result = litellm.agent(
    Harness.CLAUDE_CODE,  # or Harness.CODEX, Harness.OPENCODE, Harness.DEEPAGENTS
    "Find why tests/test_router.py is flaky and fix it.",
    sandbox=sandbox.local("./repo"),
    model="litellm_proxy/claude-sonnet-4-5",  # a model group on your AI Gateway
)

print(result.text, result.cost, [f.path for f in result.files])

Set LITELLM_PROXY_API_BASE and LITELLM_PROXY_API_KEY and every model call the agent makes goes through your AI Gateway, tagged harness,claude_code. Drop the litellm_proxy/ prefix to call a provider directly. Install starlette uvicorn plus the agent's CLI (claude, codex or opencode), or deepagents langchain-litellm for Deep Agents.

Docs: Agent Harnesses

Supported Providers (Website Supported Models | Docs)

Provider /chat/completions /messages /responses /embeddings /image/generations /audio/transcriptions /audio/speech /moderations /batches /rerank
Abliteration (abliteration) ✅
AI/ML API (aiml) ✅ ✅ ✅ ✅ ✅
AI21 (ai21) ✅ ✅ ✅
AI21 Chat (ai21_chat) ✅ ✅ ✅
Aleph Alpha ✅ ✅ ✅
Amazon Nova ✅ ✅ ✅
Anthropic (anthropic) ✅ ✅ ✅ ✅
Anthropic Text (anthropic_text) ✅ ✅ ✅ ✅
Anyscale ✅ ✅ ✅
AssemblyAI (assemblyai) ✅ ✅ ✅ ✅
Auto Router (auto_router) ✅ ✅ ✅
AWS - Bedrock (bedrock) ✅ ✅ ✅ ✅ ✅
AWS - Sagemaker (sagemaker) ✅ ✅ ✅ ✅
Azure (azure) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
Azure AI (azure_ai) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
Azure Text (azure_text) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Baseten (baseten) ✅ ✅ ✅
Bytez (bytez) ✅ ✅ ✅
Cerebras (cerebras) ✅ ✅ ✅
Clarifai (clarifai) ✅ ✅ ✅
Cloudflare AI Workers (cloudflare) ✅ ✅ ✅
Codestral (codestral) ✅ ✅ ✅
Cognition (cognition) ✅ ✅ ✅
Cohere (cohere) ✅ ✅ ✅ ✅ ✅
Cohere Chat (cohere_chat) ✅ ✅ ✅
CometAPI (cometapi) ✅ ✅ ✅ ✅
CompactifAI (compactifai) ✅ ✅ ✅
CoralBricks (coralbricks) ✅ ✅ ✅
Custom (custom) ✅ ✅ ✅
Custom OpenAI (custom_openai) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Dashscope (dashscope) ✅ ✅ ✅ ✅ ✅
Databricks (databricks) ✅ ✅ ✅
DataRobot (datarobot) ✅ ✅ ✅
Deepgram (deepgram) ✅ ✅ ✅ ✅
DeepInfra (deepinfra) ✅ ✅ ✅
Deepseek (deepseek) ✅ ✅ ✅
Eden AI (edenai) ✅ ✅ ✅ ✅ ✅ ✅ ✅
ElevenLabs (elevenlabs) ✅ ✅ ✅ ✅ ✅
Empower (empower) ✅ ✅ ✅
Fal AI (fal_ai) ✅ ✅ ✅ ✅
Featherless AI (featherless_ai) ✅ ✅ ✅
Fireworks AI (fireworks_ai) ✅ ✅ ✅
FriendliAI (friendliai) ✅ ✅ ✅
Galadriel (galadriel) ✅ ✅ ✅
GitHub Copilot (github_copilot) ✅ ✅ ✅ ✅
GitHub Models (github) ✅ ✅ ✅
Google - PaLM ✅ ✅ ✅
Google - Vertex AI (vertex_ai) ✅ ✅ ✅ ✅ ✅
Google AI Studio - Gemini (gemini) ✅ ✅ ✅
GradientAI (gradient_ai) ✅ ✅ ✅
Groq AI (groq) ✅ ✅ ✅
Heroku (heroku) ✅ ✅ ✅
Hosted VLLM (hosted_vllm) ✅ ✅ ✅
Huggingface (huggingface) ✅ ✅ ✅ ✅ ✅
Hyperbolic (hyperbolic) ✅ ✅ ✅
IBM - Watsonx.ai (watsonx) ✅ ✅ ✅ ✅
Infinity (infinity) ✅
Jina AI (jina_ai) ✅
Lambda AI (lambda_ai) ✅ ✅ ✅
Lemonade (lemonade) ✅ ✅ ✅
LiteLLM Proxy (litellm_proxy) ✅ ✅ ✅ ✅ ✅
Llamafile (llamafile) ✅ ✅ ✅
LM Studio (lm_studio) ✅ ✅ ✅
Maritalk (maritalk) ✅ ✅ ✅
Meta - Llama API (meta_llama) ✅ ✅ ✅
Mistral AI API (mistral) ✅ ✅ ✅ ✅
ModelScope (modelscope) ✅ ✅ ✅ ✅
Moonshot (moonshot) ✅ ✅ ✅
Morph (morph) ✅ ✅ ✅
Nace (nace)
Nebius AI Studio (nebius) ✅ ✅ ✅ ✅
NLP Cloud (nlp_cloud) ✅ ✅ ✅
Novita AI (novita) ✅ ✅ ✅
Nscale (nscale) ✅ ✅ ✅
Nvidia NIM (nvidia_nim) ✅ ✅ ✅
OCI (oci) ✅ ✅ ✅
Ollama (ollama) ✅ ✅ ✅ ✅
Ollama Chat (ollama_chat) ✅ ✅ ✅
Oobabooga (oobabooga) ✅ ✅ ✅ ✅ ✅ ✅ ✅
OpenAI (openai) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
OpenAI-like (openai_like) ✅
OpenRouter (openrouter) ✅ ✅ ✅
OVHCloud AI Endpoints (ovhcloud) ✅ ✅ ✅
Perplexity AI (perplexity) ✅ ✅ ✅
Petals (petals) ✅ ✅ ✅
Pinstripes (pinstripes) ✅ ✅ ✅
Predibase (predibase) ✅ ✅ ✅
Qianwen AI Platform (qwen_ai_platform) ✅ ✅ ✅ ✅ ✅ ✅
QwenCloud (qwencloud) ✅ ✅ ✅ ✅ ✅ ✅
Recraft (recraft) ✅
Replicate (replicate) ✅ ✅ ✅
Sagemaker Chat (sagemaker_chat) ✅ ✅ ✅
Sail (sail) ✅ ✅ ✅
Sambanova (sambanova) ✅ ✅ ✅
ScaleDown (scaledown) ✅
Snowflake (snowflake) ✅ ✅ ✅
Strands Decider (strands_decider)
Text Completion Codestral (text-completion-codestral) ✅ ✅ ✅
Text Completion OpenAI (text-completion-openai) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Together AI (together_ai) ✅ ✅ ✅
Topaz (topaz) ✅ ✅ ✅
Triton (triton) ✅ ✅ ✅
Typesafe Decisions API (typesafe)
V0 (v0) ✅ ✅ ✅
Vercel AI Gateway (vercel_ai_gateway) ✅ ✅ ✅
VLLM (vllm) ✅ ✅ ✅
Volcengine (volcengine) ✅ ✅ ✅
VoyageAI by MongoDB (voyage) ✅
WandB Inference (wandb) ✅ ✅ ✅
Watsonx Text (watsonx_text) ✅ ✅ ✅
xAI (xai) ✅ ✅ ✅
Xinference (xinference) ✅

Read the Docs


Get Started

You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:

LiteLLM AI Gateway LiteLLM Python SDK
Use Case Central service (LLM Gateway) to access multiple LLMs Use LiteLLM directly in your Python code
Who Uses It? Gen AI Enablement / ML Platform Teams Developers building LLM projects
Key Features Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)

Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

Deploy on AWS or GCP with Terraform

Run the LiteLLM proxy as a production-ready componentized stack (gateway, backend, UI on separate services; managed Postgres + Redis + object store) using the published Terraform modules. Both modules are on the public Terraform Registry — no auth needed.

AWS — ECS Fargate + Aurora + ElastiCache + ALB

Launch in AWS CloudShell — opens an in-browser shell, already authenticated to your AWS account. Once inside, run:

git clone https://github.com/BerriAI/litellm.git
cd litellm/terraform/litellm/aws/examples/default
cp terraform.tfvars.example terraform.tfvars   # edit region/tenant/env
terraform init && terraform apply

Module page →

Or call the module from your own root config:

# main.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

provider "aws" {
  region = "us-west-2"
}

module "litellm" {
  source  = "BerriAI/litellm/aws"
  version = "~> 1.89"

  region = "us-west-2"
  azs    = ["us-west-2a", "us-west-2b"]
  tenant = "acme"
  env    = "prod"

  # Production: provide an ACM cert. Without one, set allow_plaintext_alb = true
  # (dev/trial only).
  # acm_certificate_arn = "arn:aws:acm:us-west-2:111122223333:certificate/..."
  allow_plaintext_alb = true
}

output "litellm_url" {
  value = module.litellm.alb_dns_name
}
terraform init
terraform apply

Provider API keys live in AWS Secrets Manager; reference ARNs via gateway_extra_secrets. Full input list and architecture diagram on the registry page.

GCP — Cloud Run + Cloud SQL + Memorystore + HTTPS LB

Open in Cloud Shell

Real 1-click. Opens Cloud Shell, clones the terraform-google-litellm module mirror at the latest stable release (a small download instead of the full litellm repo), and walks you through terraform apply via a built-in DeployStack tutorial. Pick the project, the tutorial sets up the Artifact Registry remote repo, writes terraform.tfvars from your answers, and runs apply

Module page →

To call the module from your own config instead, Cloud Run can't pull from ghcr.io directly, so first set up a one-time Artifact Registry remote repo backed by GHCR:

gcloud artifacts repositories create litellm \
  --location=us-central1 \
  --repository-format=docker \
  --mode=remote-repository \
  --remote-docker-repo=https://ghcr.io \
  --project=my-gcp-project

Then:

# main.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    google      = { source = "hashicorp/google",      version = "~> 6.10" }
    google-beta = { source = "hashicorp/google-beta", version = "~> 6.10" }
  }
}

provider "google"      { project = "my-gcp-project"; region = "us-central1" }
provider "google-beta" { project = "my-gcp-project"; region = "us-central1" }

module "litellm" {
  source  = "BerriAI/litellm/google"
  version = "~> 1.89"

  project_id = "my-gcp-project"
  region     = "us-central1"
  tenant     = "acme"
  env        = "prod"

  # Replace my-gcp-project with your GCP project ID (same value as project_id above).
  image_registry = "us-central1-docker.pkg.dev/my-gcp-project/litellm/berriai"

  # Production: provide DNS already pointing at the LB IP for Google-managed certs.
  # Without one, set allow_plaintext_lb = true (dev/trial only).
  # lb_domains         = ["proxy.example.com"]
  allow_plaintext_lb = true
}

output "litellm_url" {
  value = module.litellm.load_balancer_url
}
terraform init
terraform apply

Provider API keys live in Secret Manager; reference resource IDs (e.g. projects/my-gcp-project/secrets/openai-api-key) via gateway_extra_secrets. Full input list and architecture diagram on the registry page.

Both stacks include

  • The full componentized split (gateway / backend / UI as independent services)
  • Managed Postgres (writer + reader) and Redis
  • Versioned object store for proxy state + file uploads
  • An auto-generated LITELLM_MASTER_KEY in your cloud's secret manager
  • A one-off migration job that runs prisma migrate deploy before the proxy starts
  • The same proxy_config surface as the Helm chart — pass YAML as a typed map

The Terraform modules live at terraform/litellm/aws/ and terraform/litellm/gcp/ in this repo; the registry entries are read-only mirrors updated on each release.

Run in Developer Mode

Services

  1. Setup .env file in root
  2. Run dependent services docker-compose up db prometheus

Backend

  1. Run make bootstrap
  2. Start proxy backend: uv run python litellm/proxy/proxy_cli.py

Frontend

  1. Navigate to ui/litellm-dashboard (dependencies were already installed w/ make bootstrap)
  2. Start dashboard: npm run dev

Verify Docker Image Signatures

All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).


Enterprise

For companies that need better security, user management and professional support

Get an Enterprise License Talk to founders

This covers:

  • ✅ Features under the LiteLLM Commercial License:
  • ✅ Feature Prioritization
  • ✅ Custom Integrations
  • ✅ Professional Support - Dedicated discord + slack
  • ✅ Custom SLAs
  • ✅ Secure access with Single Sign-On

Contributing

We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.

Quick Start for Contributors

This requires uv to be installed.

git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only

For detailed contributing guidelines, see CONTRIBUTING.md.

📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.

Code Quality / Linting

LiteLLM follows the Google Python Style Guide.

Our automated checks include:

  • Ruff for formatting, linting, and code quality
  • basedpyright for type checking
  • Circular import detection
  • Import safety checks

All these checks must pass before your PR can be merged.

Support / talk with founders

Contributors