scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 36
litellm/pyproject.toml
yuneng-jiang 9146667f80
fix(ci): stop the mutation report publishing a score it never measured (#37825) 
* fix(ci): stop the mutation report publishing a score it never measured

Run 32475268575 was the first dispatch of this workflow since May. Every setup
step passed and mutmut generated all 48 mutant files, so the suspected
zero-mutants bug is not what stops it. It dies in the stats phase, where mutmut
times the configured test set once up front. That set included
tests/proxy_behavior/management/, a behaviour tier that talks to a real
seeded database, so the run ended having mutated nothing.

Narrow tests_dir to the unit tier that maps to paths_to_mutate. Run 32476663383
proved a Postgres service is not enough on its own: with a schema but no seed
rows the same test fails on a foreign key instead, and a mutation score is only
meaningful against the tests that claim to cover the mutated code.

The second half is the one that matters. With no results at all,
mutation_report.py printed "No surviving mutants, the test suite caught every
mutation" and exited 0, so a run that mutated nothing published a perfect score.
It now separates no survivors from no results, says which it got, and exits 1.

* fix(ci): count mutmut's multi-word verdicts as results

The verdict capture was `\w+`, so it matched only single-word statuses. mutmut's
status_by_exit_code table has four that are not: `no tests`, `not checked`,
`caught by type check` and `check was interrupted by user`. A finished run made
entirely of those parsed as zero results, which is exactly the state this script
now treats as an unfinished run, so it would have failed a run that had in fact
completed.

The regression test asserting `reported == 2` on a three-verdict fixture was
codifying that, and now asserts 3. A second test walks all four multi-word
statuses and checks the report does not call the run unfinished.

Caught by Greptile on #37825.

* fix(ci): keep the saml tests out of the mutmut stats phase

Run 32477695014 got past the database blocker and ran 208 of the configured
tests, then ended on one error: test_saml_sso.py builds an x509 certificate in
a fixture, and inside mutmut's mutants/ sandbox cryptography's hash classes are
imported under a second identity, so .sign() rejects the SHA256 instance with
"Algorithm must be a registered hash algorithm".

That is a property of the sandbox, not of the tests or the code being mutated,
and one erroring test ends the stats phase before a single mutant runs.

* fix(ci): only claim a clean sweep when something was shown to be killed

`mutmut results` skips killed mutants by design, so its silence means either
that everything was killed or that nothing ran. Counting the verdicts it does
print cannot tell those apart, which left the report still able to say the suite
caught every mutation on a run whose mutants were all `no tests` or
`not checked`.

The clean-sweep sentence is now gated on mutmut-cicd-stats.json reporting a
non-zero killed count, which is the only signal that positively distinguishes
the two. Without it the report says so in as many words and main returns 1. A
run with zero kills and a stats file says that too.

The test asserting a non-killed run was not called unfinished was codifying the
same confusion; it is replaced by three that pin each branch.

Caught by Greptile on #37825.

* fix(ci): treat stats that count survivors the report never listed as untrusted

clean_sweep_is_provable passed on any positive kill count, so a stats file
reporting 48 killed and 3 survived, next to a `mutmut results` that listed no
survivors, still published a clean sweep. The two sources contradict each other
there, and neither one is worth believing. It now requires the stats file to
agree that nothing survived, and the report says which disagreement it found.

* fix(ci): refuse a clean sweep while mutants never reached the tests

A run can end with kills, no survivors, and a pile of mutants marked no tests,
skipped, suspicious, timeout or segfault. Those never got put in front of the
suite, so "caught every mutation" says more than the run measured. The verdict
now names which of them it found and withholds the pass, and the status list
those five come from is one constant the summary and the verdict share.

* fix(ci): read anything that is not a kill or a survivor as unresolved

The unresolved statuses were a list of five, so a run ending in a status the
reporter had never met, "check was interrupted by user" among them, still
counted as a clean sweep. The rule is now the other way round: killed, survived
and total are the keys with a meaning here, and every other non-zero count is a
mutant that did not reach the tests, whatever mutmut chose to call it.
2026-08-21 20:15:52 -07:00

381 lines
13 KiB
TOML
Raw Blame History

[project]
name = "litellm"
version = "1.99.0"
description = "Library to easily interface with LLM API providers"
readme = "README.md"
requires-python = ">=3.10, <3.15"
license = "MIT"
license-files = ["LICENSE"]
authors = [
{ name = "BerriAI" },
]
dependencies = [
# Ranges (not exact pins) so SDK consumers can coexist with their other
# deps. Reproducibility for our Docker/CI comes from `uv.lock`.
# When changing a floor, verify it installs + imports on every supported
# Python with: `uv pip install --resolution=lowest-direct .`
"fastuuid>=0.14.0,<1.0",
"httpx>=0.28.0,<1.0",
"openai>=2.20.0,<3.0.0",
"python-dotenv>=1.0.0,<2.0",
"tiktoken>=0.8.0,<1.0",
"importlib-metadata>=8.0.0,<9.0",
"tokenizers>=0.21.0,<1.0",
"click>=8.0.0,<9.0",
"jinja2>=3.1.6,<4.0",
"aiohttp>=3.14.2,<4.0",
"pydantic>=2.10.0,<3.0.0",
"pydantic-settings>=2.14.1,<3.0",
"jsonschema>=4.0.0,<5.0",
"boto3>=1.43.1,<2.0",
]
[project.urls]
Homepage = "https://litellm.ai"
Repository = "https://github.com/BerriAI/litellm"
Documentation = "https://docs.litellm.ai"
# Optional extras use compatible ranges (like the core SDK above) so downstream
# consumers can coexist with other packages and pick up security patches without
# forking. Reproducibility for our Docker/CI comes from `uv.lock` (images install
# via `uv sync --frozen`). A few deps stay exact-pinned: litellm's own
# sub-packages and the opentelemetry trio move in lockstep, and grpcio is
# supply-chain-pinned to a vetted, aged release.
[project.optional-dependencies]
proxy = [
"gunicorn>=23.0.0,<24.0",
"uvicorn>=0.33.0,<1.0",
"granian>=2.7.4,<3.0",
"uvloop>=0.21.0,<1.0; sys_platform != 'win32'",
"fastapi>=0.136.3,<1.0",
"starlette>=1.0.1,<2.0",
"backoff>=2.2.1,<3.0",
"pyyaml>=6.0.3,<7.0",
"rq>=2.7.0,<3.0",
"orjson>=3.11.6,<4.0",
# redis-py's C response parser. It arrives with redis (via rq) either way; naming
# it here is what makes redis-py select _HiredisParser instead of the Python one.
"hiredis>=3.0.0,<4.0",
"apscheduler>=3.11.2,<4.0",
"fastapi-sso>=0.19.0,<1.0",
"PyJWT>=2.13.0,<3.0",
"python-multipart>=0.0.27,<1.0",
"cryptography>=49.0.0,<51.0",
"pynacl>=1.6.2,<2.0",
"websockets>=15.0.1,<16.0",
"boto3>=1.43.1,<2.0",
"azure-identity>=1.25.2,<2.0",
"azure-storage-blob>=12.28.0,<13.0",
"mcp>=1.28.1,<2.0",
"litellm-proxy-extras==0.4.88",
"litellm-enterprise==0.1.58",
"RestrictedPython>=8.1,<9.0",
"rich>=13.9.4,<14.0",
"InquirerPy>=0.3.4,<1.0",
"polars>=1.38.1,<2.0",
"soundfile>=0.12.1,<1.0",
"pyroscope-io>=0.8.16,<1.0; sys_platform != 'win32'",
"expression>=5.6.0,<6.0",
]
# Thin client install for the `lite` CLI on developer laptops. The CLI's heavy
# imports are all guarded, so it runs on the base SDK plus just these five, and
# none of the server runtime in `proxy` is pulled in. On Linux,
# keyring reaches the Secret Service through secretstorage, which brings
# cryptography with it.
cli = [
"rich>=13.9.4,<14.0",
"pyyaml>=6.0.3,<7.0",
"requests>=2.32.0,<3.0",
"InquirerPy>=0.3.4,<1.0",
"keyring>=25.6.0,<26.0",
]
extra_proxy = [
"prisma>=0.11.0,<1.0",
"azure-identity>=1.25.2,<2.0",
"azure-keyvault-secrets>=4.10.0,<5.0",
# Not in PyPI proxy extra.
"google-cloud-kms>=2.24.2,<3.0",
"google-cloud-iam>=2.19.1,<3.0",
# Not in PyPI proxy extra.
"resend>=2.23.0,<3.0",
"redisvl>=0.4.1,<1.0",
"a2a-sdk>=1.1.0,<2.0",
]
utils = [
# Not in Docker or PyPI proxy extra.
"numpydoc>=1.8.0,<2.0",
]
caching = ["diskcache>=5.6.3,<6.0"]
# SAML SSO for the admin UI. python3-saml pulls in xmlsec/lxml, whose wheels
# bundle the native libxmlsec1/libxml2 libraries, so no system packages are
# required. Kept out of the base `proxy` extra so it stays optional.
saml = ["python3-saml>=1.16.0,<2.0"]
semantic-router = [
"semantic-router>=0.1.15,<1.0; python_version < '3.14'",
"aurelio-sdk>=0.0.19,<1.0; python_version < '3.14'",
]
mlflow = ["mlflow>=3.11.1,<4.0"]
grpc = [
# Newest non-yanked release older than the 30-day cutoff.
"grpcio==1.78.0",
]
stt-nvidia-riva = [
# NVIDIA Riva STT provider (gRPC). These are imported lazily inside the
# provider handler so litellm core remains usable without them.
"nvidia-riva-client>=2.15.0",
"soundfile>=0.12.1",
"audioread>=3.0.1",
"numpy>=1.26.0",
]
google = ["google-cloud-aiplatform>=1.133.0,<2.0"]
bedrock-realtime = [
# Bedrock Nova Sonic realtime (speech-to-speech) uses the
# InvokeModelWithBidirectionalStream API, which boto3 cannot do. This
# experimental AWS SDK (with its smithy-* deps, pulled transitively)
# provides the bidirectional stream; imported lazily in the realtime
# handler so litellm core stays usable without it.
"aws-sdk-bedrock-runtime>=0.7.0,<0.8.0; python_version >= '3.12'",
]
proxy-runtime = [
# Historically bundled in the proxy Docker images via requirements.txt.
# Keep these in a dedicated extra so uv-based images preserve the same
# feature surface without forcing the base SDK install to grow.
"google-cloud-aiplatform>=1.133.0,<2.0",
"google-genai>=1.37.0,<2.0",
"anthropic[vertex]>=0.84.0,<1.0",
"grpcio==1.78.0",
"prometheus-client>=0.20.0,<1.0",
"langfuse>=2.59.7,<3.0",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"ddtrace>=4.8.2,<5.0",
"sentry-sdk>=2.21.0,<3.0",
"mangum>=0.17.0,<1.0",
"azure-ai-contentsafety>=1.0.0,<2.0",
"azure-storage-file-datalake>=12.20.0,<13.0",
"pypdf>=6.12.0,<7.0",
"llm-sandbox>=0.3.39,<1.0",
"detect-secrets>=1.5.0,<2.0",
]
[project.scripts]
litellm = "litellm:run_server"
lite = "litellm.proxy.client.cli:cli"
litellm-proxy = "litellm.proxy.client.cli:cli"
[dependency-groups]
dev = [
"diff-cover==9.7.2",
"basedpyright==1.39.7",
"keyring==25.7.0",
"pytest==9.0.3",
"pytest-mock==3.15.1",
"pytest-asyncio==1.3.0",
"pytest-postgresql==7.0.2",
# pytest-postgresql imports psycopg v3 during pytest startup. Keep the base
# package and the binary wheel in the default dev environment so local
# pytest works without requiring a system libpq install.
"psycopg==3.3.3",
"psycopg-binary==3.3.3",
"pytest-xdist==3.8.0",
"requests-mock==1.12.1",
"responses==0.26.0",
"respx==0.22.0",
"ruff==0.15.3",
"types-requests==2.32.4.20260107",
"types-setuptools==75.8.0.20250225",
"types-redis==4.6.0.20241004",
"types-PyYAML==6.0.12.20250915",
"botocore-stubs==1.43.14",
"types-boto3[bedrock,bedrock-agent,bedrock-runtime,kms,s3,sagemaker-runtime,sts]==1.43.30",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"langfuse==2.59.7",
"fastapi-offline==1.7.6",
"fakeredis==2.34.1",
"pytest-rerunfailures==15.1",
"pytest-cov==5.0.0",
"parameterized==0.9.0",
"openapi-core==0.22.0",
"pytest-timeout==2.4.0",
"vcrpy==8.2.1",
"pytest-recording==0.13.4",
]
e2e-dev = [
"playwright==1.61.0",
"websockets>=15.0.1,<16.0",
"locust==2.45.0",
"mcp>=1.28.1,<2.0",
]
proxy-dev = [
"prisma==0.11.0",
"hypercorn==0.17.3",
"prometheus-client==0.20.0",
"opentelemetry-api==1.28.0",
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"azure-identity==1.25.2",
"a2a-sdk==1.1.0",
]
ci = [
# These are lazily imported at call sites; keep them out of core deps to
# avoid bloating the base SDK install (google-generativeai pulls grpcio +
# protobuf, Pillow is a compiled C extension).
"tenacity==8.5.0",
"google-generativeai==0.8.6",
"Pillow==12.3.0",
# Azure batch E2E tests still import psycopg2 directly.
"psycopg2-binary==2.9.11",
"pytest-codspeed==4.3.0",
"pytest-retry==1.7.0",
"pyarrow==23.0.1",
"langchain==1.3.9",
"lunary==1.4.36; python_version == '3.10'",
"lunary==1.4.37; python_version >= '3.11'",
"logfire==4.6.0",
"traceloop-sdk==0.33.12",
"detect-secrets==1.5.0",
"PyGithub==2.8.1",
"aiodynamo==24.7",
"argon2-cffi==25.1.0",
"assemblyai==0.52.4",
"jsonlines==4.0.0",
"anthropic==0.84.0",
"blockbuster==1.5.26",
"beautifulsoup4==4.14.3",
"pylint==4.0.5",
"langchain-mcp-adapters==0.2.1",
"langchain-openai==1.1.14",
"langgraph>=1.2.4,<1.3.0",
"langgraph-prebuilt>=1.1.0,<1.3.0",
"claude-agent-sdk==0.1.44",
]
healthcheck = [
"httpx==0.28.1",
"pyyaml==6.0.3",
]
[build-system]
requires = ["maturin==1.9.4"]
build-backend = "maturin"
[tool.maturin]
manifest-path = "litellm-rust/crates/python-bridge/Cargo.toml"
module-name = "litellm.rust_bridge._native"
python-source = "."
bindings = "pyo3"
include = ["litellm/proxy/_experimental/out/**"]
exclude = [
"litellm/proxy/enterprise",
"litellm/proxy/enterprise/**",
"**/__pycache__",
"**/__pycache__/**",
"**/.pytest_cache",
"**/.pytest_cache/**",
"**/.ruff_cache",
"**/.ruff_cache/**",
]
[tool.uv]
constraint-dependencies = [
"tornado>=6.5.6",
"aiohttp>=3.14.2,<4.0",
"packaging>=24.0",
"soupsieve>=2.8.4",
"httplib2>=0.32.0",
"setuptools>=83.0.0",
]
override-dependencies = [
# a2a-sdk 1.x requires packaging>=24.0; lunary 1.4.x still caps at <24.0.
"packaging>=24.0",
"cryptography>=50.0.0,<51.0",
]
default-groups = ["dev"]
required-version = ">=0.10.9"
exclude-newer = "3 days"
[tool.uv.sources]
litellm-proxy-extras = { workspace = true }
litellm-enterprise = { workspace = true }
[tool.uv.workspace]
members = ["enterprise", "litellm-proxy-extras"]
[tool.isort]
profile = "black"
[tool.commitizen]
version = "1.99.0"
version_files = [
"pyproject.toml:^version",
]
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "session"
markers = [
"asyncio: mark test as an asyncio test",
"limit_leaks: mark test with memory limit for leak detection (e.g., '40 MB')",
"no_parallel: mark test to run sequentially (not in parallel) - typically for memory measurement tests",
]
filterwarnings = [
# Suppress Pydantic serializer warnings from mock server responses (non-critical for memory tests)
# These occur because the mock server returns a simplified response format
"ignore:Pydantic serializer warnings:UserWarning",
"ignore::UserWarning:pydantic.main",
# Suppress pytest-asyncio event loop deprecation warning (handled automatically by pytest-asyncio)
"ignore::DeprecationWarning:pytest_asyncio.plugin",
]
[tool.mutmut]
# Mutation-testing scope. Driven by the manually-triggered workflow at
# .github/workflows/mutation-test.yml. mutmut is not part of the project's
# default install; it is pulled in via `uv run --with mutmut==<version>` in CI.
# `also_copy = ["litellm/"]` is required because mutmut runs in a `mutants/`
# sandbox and the test conftest imports from across the litellm package.
paths_to_mutate = [
"litellm/proxy/management_endpoints/",
]
# Only the unit tier that maps to paths_to_mutate. mutmut times and
# coverage-maps this whole set once before mutating, so a tier that needs a
# seeded database (tests/proxy_behavior/) kills the run before it starts, and
# a mutation score is only meaningful against the tests that claim to cover
# the mutated code anyway.
tests_dir = [
"tests/test_litellm/proxy/management_endpoints/",
]
also_copy = [
"litellm/",
]
# Run the test suite once before mutation to gather line coverage, then skip
# mutating lines no test exercises. Those mutants would survive regardless
# (no test hits the line to kill them), so generating them wastes hours of CI.
# The score now reads as "mutation score over covered code" — pair with a
# line-coverage number when reporting.
mutate_only_covered_lines = true
# Disable rerun/parallel plugins for mutation runs:
# - pytest-retry triggers an `INTERNALERROR: no option named 'filtered_exceptions'`
# when invoked via mutmut's in-process `pytest.main()` call.
# - rerunning a "failed" test on a mutant would mask which mutants are killed
# vs. survive, so reruns are wrong for mutation testing regardless.
# - xdist is unnecessary inside mutmut (mutmut handles its own parallelism).
# test_saml_sso.py cannot run inside mutmut's mutants/ sandbox: the copied tree
# re-imports cryptography's hash classes under a second identity, so x509 .sign()
# rejects the SHA256 instance the fixture builds with "Algorithm must be a
# registered hash algorithm". Nothing to do with mutation coverage, and one
# erroring test is enough to end the stats phase before any mutant runs.
pytest_add_cli_args = [
"-p", "no:retry",
"-p", "no:rerunfailures",
"-p", "no:xdist",
"--ignore=tests/test_litellm/proxy/management_endpoints/test_saml_sso.py",
]
[tool.coverage.run]
source = ["litellm"]
relative_files = true
Reference in a new issue View git blame Copy permalink