mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-06 08:16:43 +00:00
afec9b8ab9
* perf(ci): cache the Rust build the unit shards compile from scratch Every unit shard installs the workspace, and the root package builds through maturin, so each of the eleven jobs compiles litellm-rust/crates/python-bridge in release mode before a single test runs. That step measured 2m40s a shard on 2026-08-21, which is more wall clock than the entire unit tier spends running tests, and none of it was cached: the uv cache covers wheels it downloads, not wheels it builds, and a path dependency whose source moves every commit can never hit that cache anyway. A composite action now exports CARGO_TARGET_DIR to a fixed workspace path and caches it alongside the Cargo registry, keyed on Cargo.lock. Cargo rebuilds only what changed, so a warm job pays for the bridge crate rather than its whole dependency graph. Measured locally, that is 34s cold against 8s warm, including after a Python-only or Rust-only edit. The absolute path matters: uv builds the wheel from its own working directory, so a relative target directory lands the artifacts where nothing can find them again. * perf(ci): cache the Rust build in the other four workflows that sync the workspace code-quality, mcp, documentation and the schema.d.ts check each install the workspace and so each compile the bridge from scratch, measured at 138s, 177s, 163s and 154s on 2026-08-21. The lint job pays the same and is left to #37783, which already owns that file's setup section. * perf(ci): cache the Rust build in the lint job too * fix(ci): cache cargo's own target directory instead of redirecting it uv builds the wheel in place, so cargo already writes to litellm-rust/target, which test-rust.yml has cached all along. Redirecting CARGO_TARGET_DIR bought nothing and cost a GITHUB_ENV write that zizmor rejects as a code-execution path. * chore(ci): raise the job backstop for the added setup step The cargo cache is a fifth bounded setup step, so the base's setup ceiling goes 30m to 35m and every job budget follows: 55 to 60, and proxy-server's 95 to 100. check_workflow_startup_safety enforces exactly this sum, and failed on the first push without it. * perf(ci): cache the Rust build in the four remaining workflows that sync Six workflows were wired; ten install the workspace. The four left out still compile the pyo3 bridge from scratch. test-terraform-provider.yml is the one that matters per PR: its endpoint-drift job triggers on any change under litellm/proxy/**. The other three are a scheduled load check, a manual mutation run, and the staging-push counts publisher, whose gate syncs the project inside scripts/type_check_gate.py rather than in a workflow step, so nothing in the file names the build it pays for.
135 lines
5.3 KiB
YAML
135 lines
5.3 KiB
YAML
name: "Mutation Test (manual)"
|
|
|
|
# Manually-triggered mutation testing. Runs mutmut against the scope
|
|
# configured in [tool.mutmut] in pyproject.toml (currently the
|
|
# litellm/proxy/management_endpoints/ folder). Intended cadence is roughly
|
|
# weekly — clicked from the Actions tab when someone wants a fresh report.
|
|
#
|
|
# Uploads a structured `mutation-report.md` (Meta ACH-style: original +
|
|
# mutated function with `# MUTANT START`/`# MUTANT END` delimiters + the
|
|
# existing tests + a task instruction) as a workflow artifact. Failures
|
|
# do not block anything because nothing depends on this workflow.
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: mutation-test-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
mutation:
|
|
name: Run mutmut
|
|
runs-on: ubuntu-latest
|
|
# Whole-folder mutation against ~15 files / ~7.5k LOC can take hours.
|
|
# 350 minutes is just under the GitHub-hosted job cap of 360 minutes.
|
|
timeout-minutes: 350
|
|
|
|
steps:
|
|
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Set up uv
|
|
uses: ./.github/actions/setup-uv-with-retries
|
|
with:
|
|
version: "0.10.9"
|
|
|
|
- name: Cache uv dependencies
|
|
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
|
|
with:
|
|
path: |
|
|
~/.cache/uv
|
|
.venv
|
|
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
|
|
restore-keys: |
|
|
${{ runner.os }}-uv-
|
|
|
|
- name: Cache the Rust build
|
|
uses: ./.github/actions/cache-cargo-build
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml
|
|
|
|
- name: Cache Prisma binaries
|
|
uses: ./.github/actions/cache-prisma-binaries
|
|
|
|
- name: Generate Prisma client
|
|
run: |
|
|
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
|
|
|
|
# mutmut 3.x runs tests inside a `mutants/` sandbox where it injects
|
|
# mutation trampolines. uv installs the project as editable by default,
|
|
# which puts the original source dir on sys.path via a .pth file and
|
|
# shadows the sandbox copy — so tests would never exercise the mutated
|
|
# code. Reinstalling non-editable removes the .pth shadow.
|
|
- name: Reinstall litellm non-editable (so mutants/ is not shadowed)
|
|
run: |
|
|
uv pip uninstall litellm
|
|
uv pip install . --no-deps
|
|
|
|
# pytest-retry's pytest_configure hook crashes with
|
|
# `INTERNALERROR: no option named 'filtered_exceptions'` when invoked
|
|
# via mutmut's in-process pytest.main() call. The entry-point name
|
|
# doesn't normalize cleanly with `-p no:<name>`, so just remove the
|
|
# package outright. Reruns are wrong for mutation testing anyway —
|
|
# rerunning a "failed" mutant test would mask which mutants are killed.
|
|
- name: Remove pytest plugins that conflict with mutmut
|
|
run: |
|
|
uv pip uninstall pytest-retry || true
|
|
|
|
- name: Run mutmut
|
|
env:
|
|
# Make the mutants/ sandbox win over site-packages on sys.path so the
|
|
# trampolined files are imported instead of the installed copy.
|
|
PYTHONPATH: ${{ github.workspace }}/mutants
|
|
run: |
|
|
set -o pipefail
|
|
mkdir -p mutants
|
|
uv run --no-sync --with mutmut==3.5.0 mutmut run 2>&1 | tee mutmut-run.log
|
|
|
|
# Generate the structured report. The script embeds the enclosing
|
|
# function source for each survivor (via Python AST) and includes the
|
|
# existing test files, so an LLM agent has enough context to write
|
|
# killing tests without further file lookups. Modeled on Meta's ACH
|
|
# prompt template (arXiv 2501.12862).
|
|
- name: Generate detailed mutation report
|
|
if: always()
|
|
run: |
|
|
set +e
|
|
uv run --no-sync --with mutmut==3.5.0 mutmut export-cicd-stats > /dev/null 2>&1
|
|
uv run --no-sync --with mutmut==3.5.0 mutmut results > mutmut-results.txt 2>&1
|
|
uv run --no-sync python scripts/mutation_report.py
|
|
# The full report can be very long for big test files; the run-page
|
|
# summary cuts off at 1 MB. Append the head of the report (summary
|
|
# + survivor list) and link out to the artifact for the full body.
|
|
{
|
|
head -c 900000 mutation-report.md
|
|
echo ""
|
|
echo ""
|
|
echo "_Full report (with embedded function bodies and test files) is in the workflow artifact._"
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
- name: Upload mutmut artifacts
|
|
if: always()
|
|
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
|
|
with:
|
|
name: mutmut-${{ github.run_id }}-${{ github.run_attempt }}
|
|
path: |
|
|
mutation-report.md
|
|
mutmut-results.txt
|
|
mutmut-run.log
|
|
mutants/mutmut-stats.json
|
|
mutants/mutmut-cicd-stats.json
|
|
mutants/litellm/proxy/management_endpoints/**/*.py
|
|
if-no-files-found: warn
|
|
retention-days: 14
|