scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102
litellm/pyproject.toml
Krrish Dholakia cf90445574
feat(cli): add lite up/down to ambiently route Claude Code through the proxy (#33231) 
* feat(cli): add `lite up`/`lite down` to ambiently route Claude Code through the proxy

Patches ~/.claude/settings.json in place (env.ANTHROPIC_BASE_URL + apiKeyHelper
via `lite auth print-token`) so any `claude` session started afterward, from
any terminal, routes through the local LiteLLM proxy with no wrapper command
needed, unlike the existing `lite claude` subprocess-exec approach. Backs up
the original file first and restores it on Ctrl-C/SIGTERM, or via `lite down`
after an unclean exit. Cursor is not supported: no equivalent file-based config
to patch.

* feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy (#33249)

* feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy

Lets a customer try litellm's complexity_router against models they already
have on their existing, unmodified production proxy, with no config.yaml
edits and no new infra. lite autoroute configure discovers accessible
models via /model_group/info and walks through tier assignment (plus
optional LLM classifier / semantic matching / adaptive selection); every
referenced model becomes its own litellm_proxy/<name> deployment forwarding
back to the real proxy with the real key, so every actual call, routed
completions, classifier calls, embedding calls, still lands on their real
proxy. lite autoroute up launches that generated config as an ephemeral
local proxy, patches ~/.claude/settings.json to point Claude Code at it, and
streams routing decisions live; Ctrl-C/SIGTERM (or lite autoroute down
after an unclean exit) restores everything.

Also adds lite model-groups list (a thin CLI wrapper over the existing
ModelGroupsManagementClient), and generalizes up.py's settings-backup/restore
helpers to take explicit paths so this feature can reuse them instead of
duplicating the logic.

Depends on litellm_lite_up_down (#33231) for that generalization.

* feat(cli): allow multiple models per autoroute tier

complexity_router already supports a pool of models per tier (randomly
picked per request; adaptive mode specifically needs a pool to choose
within), but the configure wizard only ever let you assign one. Tiers are
now a tuple of model names; the wizard prompt accepts comma-separated
indices to pick more than one per tier.

* feat(cli): fuzzy model picker and auto-route Claude Code to autorouter

Numbered-index selection didn't scale past a handful of models, so switch
the tier picker to InquirerPy's fzf-style fuzzy search. Also set
ANTHROPIC_DEFAULT_{SONNET,HAIKU,OPUS}_MODEL to "autorouter" in Claude
Code's settings, since Router resolves auto-router deployments by literal
model name with no wildcard support, so a "*" catch-all model_name would
never match real traffic.

* feat(cli): allow installing lite CLI from source via LITELLM_CLI_REF

Lets testers try an unreleased branch's CLI changes with the same
curl-piped installer, instead of waiting for a PyPI release.

* fix(ci): modernize type hints to clear ruff strict-rule budget

* fix(ci): bump httplib2 and setuptools to patched versions

Clears osv-scan findings for PYSEC-2026-3444 and PYSEC-2026-3447.

* fix(cli): write autoroute's secret-bearing files with mode 0600

commands.py wrote config.yaml (embeds the real proxy key) and Claude
Code's settings.json (embeds the ephemeral proxy's master key) with
plain open(), landing at the umask-derived default (commonly 0644)
until a later chmod call caught up. That window, and the missed case
where settings.json already exists (chmod never ran at all there),
left a credential-bearing file readable by another local account.

secure_create() fixes the mode via fchmod on the fd before any
content is written, covering both the brand-new-file and
already-exists cases, and commands.py/wizard.py now route their
sensitive writes through it.

* docs(cli): warn that a stale Claude Code session can leak to a squatted port

lite autoroute up's master key is embedded statically (unlike lite up's
apiKeyHelper, resolved per request), so a Claude Code session still
running after teardown keeps sending it, along with prompt content, to
a now-unbound loopback port that another local account can bind. This
is the same one-time-patch tradeoff lite up already accepts, just with
a static secret instead of a re-resolved one -- document it in the
README's Caveats section and surface it in the teardown message itself.

* fix(cli): address greptile review feedback on autoroute PR

- terminate the ephemeral proxy child process when its health check
  fails, instead of leaking an orphaned, unrecoverable process bound
  to the port
- replace bare assert isinstance checks (no-ops under python -O) with
  click.ClickException in the model-groups list and configure wizard
  code paths
- close launch_proxy's log file handle once the child process has
  inherited its fd, instead of leaking it
- add build_generated_proxy_config to config.py's __all__

* fix(cli): close TOCTOU window in lite up's settings backup write

write_backup wrote the backup (which can embed the original
apiKeyHelper/settings content) with plain open() + a chmod call after
the fact -- the same permissive-until-corrected window already fixed
for autoroute's config.yaml and Claude settings writes, and missed
entirely when the backup file already exists with broader permissions.

Moves secure_create (atomic-enough 0600 via fchmod before any content
is written) to up.py, the module both lite up and lite autoroute
share, and has autoroute/process.py import it from there instead of
keeping its own copy.

* fix(cli): refuse autoroute up when a stale backup exists from a crash

The pid-record check only catches a still-live duplicate process; a
SIGKILL'd `up` leaves no live pid but does leave AUTOROUTE_BACKUP_PATH
behind. Without this guard, a fresh `up` overwrote that backup with
the currently-patched Claude settings instead of the true originals,
so `down`/Ctrl-C would restore the wrong content permanently. up.py's
`lite up` already guards the analogous case; mirror it here.

* fix(cli): bind the ephemeral autoroute proxy to loopback only

proxy_cli.py defaults --host to 0.0.0.0 when not passed explicitly.
launch_proxy never passed it, so the ephemeral proxy -- despite every
base_url in this module being built from 127.0.0.1 -- was actually
reachable from other hosts on the network, including its
unauthenticated-until-config-lands routes before the master key is
wired in.

* docs(cli): show curl install for the autoroute QA flow

Points readers at scripts/install-cli.sh's curl one-liner instead of
assuming uv/pip is already set up, and documents the LITELLM_CLI_REF
override for trying an unreleased branch or commit.

* fix(cli): surface a clean error on an empty or corrupt autoroute config

A configure run killed between secure_create's O_TRUNC and the write
completing leaves an empty config.yaml on disk. The next up read that
via yaml.safe_load (None) into the generated-config TypeAdapter
uncaught, surfacing a raw pydantic.ValidationError instead of pointing
the user back at `lite autoroute configure`.

* fix(cli): bind lite up's apiKeyHelper to the proxy it was started against

_ensure_fresh_login only checked token freshness, not which proxy the
cached token belonged to, and resolve_api_key_helper built a bare
`lite auth print-token` command with no --base-url. A user logged into
proxy A who ran `up --base-url proxy-b` (or LITELLM_PROXY_URL=proxy-b)
would silently get proxy A's real token wired into Claude Code's
apiKeyHelper; since apiKeyHelper is invoked bare, print-token's
existing origin check never engaged, so proxy B -- attacker-controlled
or not -- received every subsequent request's Authorization header
carrying proxy A's credential.

_ensure_fresh_login now requires the cached token's base_url to match
before treating it as usable, forcing a fresh login for the selected
proxy otherwise. resolve_api_key_helper now takes that base_url and
threads it through as an explicit --base-url, so print-token's
existing (but previously unreachable in the apiKeyHelper flow)
base_url_explicit check actually enforces the match at request time
too.

* fix(cli): surface clean errors instead of raw tracebacks in lite up/down

load_json_or_empty and read_backup both delegate to pydantic's
validate_json, which raises ValidationError on invalid JSON or a
non-object root -- neither up() nor down() caught it, so a corrupt
settings or backup file surfaced an unformatted Python traceback
instead of a clean CLI error. Both now convert to UpError, and down()
(previously uncaught entirely) and up()'s teardown path now handle it.

restore_claude_settings also gained a parent.mkdir guard before
rewriting CLAUDE_SETTINGS_PATH: if ~/.claude/ was removed while `lite
up` was running, the restore would crash before deleting the backup
file, permanently stranding it and breaking every future `lite down`.

* docs(cli): call out env-var auth for autoroute commands

* fix(cli): clean up leaked proxy and surface clean errors in autoroute

Three related gaps, all following an UpError getting raised somewhere
that wasn't catching it yet:

- up() left the just-launched ephemeral proxy running with no pid
  record if load_json_or_empty/write_backup/secure_create raised after
  the health check passed, mirroring the existing ProcessLaunchError
  cleanup for the health-check-failure branch.
- _teardown() didn't catch restore_claude_settings raising UpError
  (e.g. a corrupt backup at stop time), which would otherwise escape
  to Click as an unhandled error in the normal-exit path, or print
  "Error in atexit" in the atexit path. up.py's own _restore_once
  handles the identical case the same way.
- read_pid_record let a corrupt PID file surface a raw
  pydantic.ValidationError instead of a clean message, and did so in
  down(), the command specifically meant for crash recovery. down()
  now clears an unreadable pid record and continues cleanup instead of
  aborting, since a corrupt pid file must never block the one command
  meant to recover from exactly this kind of crash.

* docs(cli): warn against running lite up and lite autoroute up together
2026-07-15 21:46:02 -07:00

349 lines
11 KiB
TOML
Raw Blame History

[project]
name = "litellm"
version = "1.94.0"
description = "Library to easily interface with LLM API providers"
readme = "README.md"
requires-python = ">=3.10, <3.15"
license = "MIT"
license-files = ["LICENSE"]
authors = [
{ name = "BerriAI" },
]
dependencies = [
# Ranges (not exact pins) so SDK consumers can coexist with their other
# deps. Reproducibility for our Docker/CI comes from `uv.lock`.
# When changing a floor, verify it installs + imports on every supported
# Python with: `uv pip install --resolution=lowest-direct .`
"fastuuid>=0.14.0,<1.0",
"httpx>=0.28.0,<1.0",
"openai>=2.20.0,<3.0.0",
"python-dotenv>=1.0.0,<2.0",
"tiktoken>=0.8.0,<1.0",
"importlib-metadata>=8.0.0,<9.0",
"tokenizers>=0.21.0,<1.0",
"click>=8.0.0,<9.0",
"jinja2>=3.1.6,<4.0",
"aiohttp>=3.10,<4.0",
"pydantic>=2.10.0,<3.0.0",
"jsonschema>=4.0.0,<5.0",
]
[project.urls]
Homepage = "https://litellm.ai"
Repository = "https://github.com/BerriAI/litellm"
Documentation = "https://docs.litellm.ai"
# Optional extras use compatible ranges (like the core SDK above) so downstream
# consumers can coexist with other packages and pick up security patches without
# forking. Reproducibility for our Docker/CI comes from `uv.lock` (images install
# via `uv sync --frozen`). A few deps stay exact-pinned: litellm's own
# sub-packages and the opentelemetry trio move in lockstep, and grpcio is
# supply-chain-pinned to a vetted, aged release.
[project.optional-dependencies]
proxy = [
"gunicorn>=23.0.0,<24.0",
"uvicorn>=0.33.0,<1.0",
"granian>=2.7.4,<3.0",
"uvloop>=0.21.0,<1.0; sys_platform != 'win32'",
"fastapi>=0.136.3,<1.0",
"starlette>=1.0.1,<2.0",
"backoff>=2.2.1,<3.0",
"pyyaml>=6.0.3,<7.0",
"rq>=2.7.0,<3.0",
"orjson>=3.11.6,<4.0",
"apscheduler>=3.11.2,<4.0",
"fastapi-sso>=0.19.0,<1.0",
"PyJWT>=2.13.0,<3.0",
"python-multipart>=0.0.27,<1.0",
"cryptography>=48.0.1,<49.0",
"pynacl>=1.6.2,<2.0",
"websockets>=15.0.1,<16.0",
"boto3>=1.43.1,<2.0",
"azure-identity>=1.25.2,<2.0",
"azure-storage-blob>=12.28.0,<13.0",
"mcp>=1.26.0,<2.0",
"litellm-proxy-extras==0.4.77",
"litellm-enterprise==0.1.50",
"RestrictedPython>=8.1,<9.0",
"rich>=13.9.4,<14.0",
"InquirerPy>=0.3.4,<1.0",
"polars>=1.38.1,<2.0",
"soundfile>=0.12.1,<1.0",
"pyroscope-io>=0.8.16,<1.0; sys_platform != 'win32'",
"pydantic-settings>=2.14.1,<3.0",
"expression>=5.6.0,<6.0",
]
# Thin client install for the `lite` CLI on developer laptops. The CLI's heavy
# imports (fastapi, cryptography, ...) are all guarded, so it runs on the base
# SDK plus just these four; none of the server runtime in `proxy` is pulled in.
cli = [
"rich>=13.9.4,<14.0",
"pyyaml>=6.0.3,<7.0",
"requests>=2.32.0,<3.0",
"InquirerPy>=0.3.4,<1.0",
]
extra_proxy = [
"prisma>=0.11.0,<1.0",
"azure-identity>=1.25.2,<2.0",
"azure-keyvault-secrets>=4.10.0,<5.0",
# Not in PyPI proxy extra.
"google-cloud-kms>=2.24.2,<3.0",
"google-cloud-iam>=2.19.1,<3.0",
# Not in PyPI proxy extra.
"resend>=2.23.0,<3.0",
"redisvl>=0.4.1,<1.0; python_version < '3.14'",
"a2a-sdk>=1.1.0,<2.0",
]
utils = [
# Not in Docker or PyPI proxy extra.
"numpydoc>=1.8.0,<2.0",
]
caching = ["diskcache>=5.6.3,<6.0"]
semantic-router = [
"semantic-router>=0.1.15,<1.0; python_version < '3.14'",
"aurelio-sdk>=0.0.19,<1.0; python_version < '3.14'",
]
mlflow = ["mlflow>=3.11.1,<4.0"]
grpc = [
# Newest non-yanked release older than the 30-day cutoff.
"grpcio==1.78.0",
]
stt-nvidia-riva = [
# NVIDIA Riva STT provider (gRPC). These are imported lazily inside the
# provider handler so litellm core remains usable without them.
"nvidia-riva-client>=2.15.0",
"soundfile>=0.12.1",
"audioread>=3.0.1",
"numpy>=1.26.0",
]
google = ["google-cloud-aiplatform>=1.133.0,<2.0"]
proxy-runtime = [
# Historically bundled in the proxy Docker images via requirements.txt.
# Keep these in a dedicated extra so uv-based images preserve the same
# feature surface without forcing the base SDK install to grow.
"google-cloud-aiplatform>=1.133.0,<2.0",
"google-genai>=1.37.0,<2.0",
"anthropic[vertex]>=0.84.0,<1.0",
"grpcio==1.78.0",
"prometheus-client>=0.20.0,<1.0",
"langfuse>=2.59.7,<3.0",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"ddtrace>=4.8.2,<5.0",
"sentry-sdk>=2.21.0,<3.0",
"mangum>=0.17.0,<1.0",
"azure-ai-contentsafety>=1.0.0,<2.0",
"azure-storage-file-datalake>=12.20.0,<13.0",
"pypdf>=6.12.0,<7.0; python_version < '3.14'",
"llm-sandbox>=0.3.39,<1.0",
"detect-secrets>=1.5.0,<2.0",
]
[project.scripts]
litellm = "litellm:run_server"
lite = "litellm.proxy.client.cli:cli"
litellm-proxy = "litellm.proxy.client.cli:cli"
[dependency-groups]
dev = [
"diff-cover==9.7.2",
"flake8==7.3.0",
"basedpyright==1.39.7",
"pytest==9.0.3",
"pytest-mock==3.15.1",
"pytest-asyncio==1.3.0",
"pytest-postgresql==7.0.2",
# pytest-postgresql imports psycopg v3 during pytest startup. Keep the base
# package and the binary wheel in the default dev environment so local
# pytest works without requiring a system libpq install.
"psycopg==3.3.3",
"psycopg-binary==3.3.3",
"pytest-xdist==3.8.0",
"requests-mock==1.12.1",
"responses==0.26.0",
"respx==0.22.0",
"ruff==0.15.3",
"types-requests==2.32.4.20260107",
"types-setuptools==75.8.0.20250225",
"types-redis==4.6.0.20241004",
"types-PyYAML==6.0.12.20250915",
"botocore-stubs==1.43.14",
"types-boto3[bedrock,bedrock-agent,bedrock-runtime,kms,s3,sagemaker-runtime,sts]==1.43.30",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"langfuse==2.59.7",
"fastapi-offline==1.7.6",
"fakeredis==2.34.1",
"pytest-rerunfailures==15.1",
"pytest-cov==5.0.0",
"parameterized==0.9.0",
"openapi-core==0.22.0; python_version < '3.14'",
"pytest-timeout==2.4.0",
"vcrpy==8.2.1",
"pytest-recording==0.13.4",
]
e2e-dev = [
"playwright==1.61.0",
"websockets>=15.0.1,<16.0",
]
proxy-dev = [
"prisma==0.11.0",
"hypercorn==0.17.3",
"prometheus-client==0.20.0",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"azure-identity==1.25.2",
"a2a-sdk==1.1.0",
]
ci = [
# These are lazily imported at call sites; keep them out of core deps to
# avoid bloating the base SDK install (google-generativeai pulls grpcio +
# protobuf, Pillow is a compiled C extension).
"tenacity==8.5.0",
"google-generativeai==0.8.6",
"Pillow==12.3.0",
# Azure batch E2E tests still import psycopg2 directly.
"psycopg2-binary==2.9.11",
"pytest-codspeed==4.3.0",
"pytest-retry==1.7.0",
"pyarrow==23.0.1",
"langchain==1.3.9",
"lunary==1.4.36; python_version == '3.10'",
"lunary==1.4.37; python_version >= '3.11'",
"logfire==4.6.0",
"traceloop-sdk==0.33.12",
"detect-secrets==1.5.0",
"PyGithub==2.8.1",
"aiodynamo==24.7",
"argon2-cffi==25.1.0",
"assemblyai==0.52.4",
"jsonlines==4.0.0",
"anthropic==0.84.0",
"blockbuster==1.5.26",
"beautifulsoup4==4.14.3",
"pylint==4.0.5",
"langchain-mcp-adapters==0.2.1",
"langchain-openai==1.1.14",
"langgraph>=1.2.4,<1.3.0",
"langgraph-prebuilt>=1.1.0,<1.3.0",
"claude-agent-sdk==0.1.44",
]
healthcheck = [
"httpx==0.28.1",
"pyyaml==6.0.3",
]
[build-system]
requires = ["maturin==1.9.4"]
build-backend = "maturin"
[tool.maturin]
manifest-path = "litellm-rust/crates/python-bridge/Cargo.toml"
module-name = "litellm.rust_bridge._native"
python-source = "."
bindings = "pyo3"
include = ["litellm/proxy/_experimental/out/**"]
exclude = [
"litellm/proxy/enterprise",
"litellm/proxy/enterprise/**",
"**/__pycache__",
"**/__pycache__/**",
"**/.pytest_cache",
"**/.pytest_cache/**",
"**/.ruff_cache",
"**/.ruff_cache/**",
]
[tool.uv]
constraint-dependencies = [
"tornado>=6.5.6",
"aiohttp>=3.14.1,<4.0",
"packaging>=24.0",
"soupsieve>=2.8.4",
"httplib2>=0.32.0",
"setuptools>=83.0.0",
]
override-dependencies = [
# a2a-sdk 1.x requires packaging>=24.0; lunary 1.4.x still caps at <24.0.
"packaging>=24.0",
]
default-groups = ["dev"]
required-version = ">=0.10.9"
exclude-newer = "3 days"
[tool.uv.sources]
litellm-proxy-extras = { workspace = true }
litellm-enterprise = { workspace = true }
[tool.uv.workspace]
members = ["enterprise", "litellm-proxy-extras"]
[tool.isort]
profile = "black"
[tool.commitizen]
version = "1.94.0"
version_files = [
"pyproject.toml:^version",
]
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "session"
markers = [
"asyncio: mark test as an asyncio test",
"limit_leaks: mark test with memory limit for leak detection (e.g., '40 MB')",
"no_parallel: mark test to run sequentially (not in parallel) - typically for memory measurement tests",
]
filterwarnings = [
# Suppress Pydantic serializer warnings from mock server responses (non-critical for memory tests)
# These occur because the mock server returns a simplified response format
"ignore:Pydantic serializer warnings:UserWarning",
"ignore::UserWarning:pydantic.main",
# Suppress pytest-asyncio event loop deprecation warning (handled automatically by pytest-asyncio)
"ignore::DeprecationWarning:pytest_asyncio.plugin",
]
[tool.mutmut]
# Mutation-testing scope. Driven by the manually-triggered workflow at
# .github/workflows/mutation-test.yml. mutmut is not part of the project's
# default install; it is pulled in via `uv run --with mutmut==<version>` in CI.
# `also_copy = ["litellm/"]` is required because mutmut runs in a `mutants/`
# sandbox and the test conftest imports from across the litellm package.
paths_to_mutate = [
"litellm/proxy/management_endpoints/",
]
tests_dir = [
"tests/test_litellm/proxy/management_endpoints/",
"tests/proxy_behavior/management/",
]
also_copy = [
"litellm/",
]
# Run the test suite once before mutation to gather line coverage, then skip
# mutating lines no test exercises. Those mutants would survive regardless
# (no test hits the line to kill them), so generating them wastes hours of CI.
# The score now reads as "mutation score over covered code" — pair with a
# line-coverage number when reporting.
mutate_only_covered_lines = true
# Disable rerun/parallel plugins for mutation runs:
# - pytest-retry triggers an `INTERNALERROR: no option named 'filtered_exceptions'`
# when invoked via mutmut's in-process `pytest.main()` call.
# - rerunning a "failed" test on a mutant would mask which mutants are killed
# vs. survive, so reruns are wrong for mutation testing regardless.
# - xdist is unnecessary inside mutmut (mutmut handles its own parallelism).
pytest_add_cli_args = [
"-p", "no:retry",
"-p", "no:rerunfailures",
"-p", "no:xdist",
]
[tool.coverage.run]
source = ["litellm"]
relative_files = true
Reference in a new issue View git blame Copy permalink