The step-timeout comment claimed mutmut streams each mutant's result into
mutants/mutmut-stats.json. It does not. That file holds the pre-run test
timings and coverage map (tests_by_mangled_function_name, duration_by_test,
stats_time) written once by save_stats() before mutation starts.
Per-mutant results live in mutants/<source path>.meta. Verified against
mutmut 3.5.0: SourceFileMutationData.register_result() calls save() after
every single result, and export-cicd-stats walks those .meta files to build
mutmut-cicd-stats.json. So the reason the step deadline exists is still
right, an interrupted run keeps the mutants it already scored, but the
comment pointed at the wrong file.
Also upload the .meta files, since they are the partial results the comment
relies on and the artifact could not otherwise show them.
mutmut's gather_coverage() looks each source file's covered lines up by
absolute path, but [tool.coverage.run] sets relative_files = true, so every
lookup misses. With mutate_only_covered_lines = true that leaves no line
eligible for mutation, and the run ends on "Stopping early, because we could
not find any test case for any mutant" after spending 26 minutes collecting
coverage. The last four dispatches all died that way.
Point COVERAGE_RCFILE at a small rc file for mutation runs only, so the
coverage instance mutmut builds stores absolute paths. Scoped to one module
locally this takes the run from 0 mutants to 8 generated and 8 killed.
Also give the mutmut step a deadline inside the job's own. mutmut records
each mutant's verdict to mutants/mutmut-stats.json as it finishes, so a run
that outlasts its budget still scores what it got through, but a cancelled
job skips the report and upload steps and publishes nothing. That is how the
two runs before these four ended.
Ignore mutants/ and .venv-mutmut, which a local run leaves behind untracked.
* perf(ci): cache the Rust build the unit shards compile from scratch
Every unit shard installs the workspace, and the root package builds through
maturin, so each of the eleven jobs compiles litellm-rust/crates/python-bridge
in release mode before a single test runs. That step measured 2m40s a shard on
2026-08-21, which is more wall clock than the entire unit tier spends running
tests, and none of it was cached: the uv cache covers wheels it downloads, not
wheels it builds, and a path dependency whose source moves every commit can
never hit that cache anyway.
A composite action now exports CARGO_TARGET_DIR to a fixed workspace path and
caches it alongside the Cargo registry, keyed on Cargo.lock. Cargo rebuilds only
what changed, so a warm job pays for the bridge crate rather than its whole
dependency graph. Measured locally, that is 34s cold against 8s warm, including
after a Python-only or Rust-only edit.
The absolute path matters: uv builds the wheel from its own working directory,
so a relative target directory lands the artifacts where nothing can find them
again.
* perf(ci): cache the Rust build in the other four workflows that sync the workspace
code-quality, mcp, documentation and the schema.d.ts check each install the
workspace and so each compile the bridge from scratch, measured at 138s, 177s,
163s and 154s on 2026-08-21. The lint job pays the same and is left to #37783,
which already owns that file's setup section.
* perf(ci): cache the Rust build in the lint job too
* fix(ci): cache cargo's own target directory instead of redirecting it
uv builds the wheel in place, so cargo already writes to litellm-rust/target,
which test-rust.yml has cached all along. Redirecting CARGO_TARGET_DIR bought
nothing and cost a GITHUB_ENV write that zizmor rejects as a code-execution
path.
* chore(ci): raise the job backstop for the added setup step
The cargo cache is a fifth bounded setup step, so the base's setup ceiling goes
30m to 35m and every job budget follows: 55 to 60, and proxy-server's 95 to 100.
check_workflow_startup_safety enforces exactly this sum, and failed on the first
push without it.
* perf(ci): cache the Rust build in the four remaining workflows that sync
Six workflows were wired; ten install the workspace. The four left out still
compile the pyo3 bridge from scratch.
test-terraform-provider.yml is the one that matters per PR: its endpoint-drift
job triggers on any change under litellm/proxy/**. The other three are a
scheduled load check, a manual mutation run, and the staging-push counts
publisher, whose gate syncs the project inside scripts/type_check_gate.py
rather than in a workflow step, so nothing in the file names the build it pays
for.
`prisma generate` runs `npm install prisma@<version>` whenever the
prisma-client-py binary cache directory has no CLI entrypoint, pulling ~85 MB
of query and schema engines over the network. Every workflow pointed
PRISMA_BINARY_CACHE_DIR at `${{ runner.temp }}/prisma-cache`, which GitHub
wipes and recreates per job, so that cache was empty on every job of every
run and the download was never avoidable.
The download is normally a few seconds and occasionally minutes. On one
proxy-db run it took 5m18s on a single shard against 3.8s on its eleven
siblings, which pushed the job past its 15 minute timeout and cancelled a
shard whose tests were at 99% and all passing.
Leave PRISMA_BINARY_CACHE_DIR unset so the binaries land in the
prisma-client-py default, which is already keyed by prisma and engine
version, and restore both that path and the @prisma/engines staging cache
through a shared composite action.
Job timeouts also counted setup against the test budget. `timeout-minutes`
now bounds the pytest step, with a separate allowance for checkout,
dependency install, and client generation, so slow setup shows up as a slow
job instead of a cancelled test run.
check_prisma_binary_cache.py guards all three invariants: no workflow
reintroduces the override, every job that generates the client restores the
cache, and the version the action greps out of uv.lock still resolves.
litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.
A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.
SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.
python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.
Resolves LIT-4016
* ci: add manually-triggered mutation testing smoke workflow
Adds a workflow_dispatch-only GitHub Actions workflow that runs mutmut
against a single source/test pair (router_settings_endpoints) to validate
the tooling end-to-end before scaling.
The workflow reinstalls litellm non-editable so the mutants/ sandbox is
not shadowed by the editable .pth on sys.path, and sets PYTHONPATH so
the trampolined sandbox copy wins over site-packages.
mutmut itself is pulled in via uv run --with so it does not appear in
uv.lock or affect the shared dev environment.
Includes a temporary push: trigger scoped to this branch so we can
iterate before the workflow file lands on the default branch — to be
removed before merging (workflow_dispatch only requires the file on the
default branch to surface the manual trigger button).
* ci(mutation): disable rerun and xdist plugins for mutmut runs
mutmut's in-process pytest.main() call hits
`INTERNALERROR: no option named 'filtered_exceptions'` from
pytest-retry's pytest_configure hook. Reruns are also wrong for
mutation testing — a "failed" mutant test that gets retried would
mask which mutants are killed vs. survive. Disable retry,
rerunfailures, and xdist via pytest_add_cli_args in [tool.mutmut].
* ci(mutation): uninstall pytest-retry before mutmut runs
`-p no:retry` (and similar names) didn't match pytest-retry's
entry-point name, so the plugin still loaded and crashed during
mutmut's "Running clean tests" phase. Uninstalling the package is
surgical and doesn't depend on guessing the entry-point name.
* ci(mutation): emit per-survivor diffs to run-page summary + artifact
The previous artifact only contained `mutmut results` text (which in
mutmut 3.x lists survivor names but not the actual mutations). Adds:
- `mutmut export-cicd-stats` to produce mutmut-cicd-stats.json with the
killed/survived/total scoreboard.
- `mutmut show <name>` per surviving mutant to capture each mutation as
a unified diff.
- A `mutmut-report.md` that combines summary + run-progress tail +
per-survivor diffs, written to both the artifact and
$GITHUB_STEP_SUMMARY (visible on the run page, no download needed).
- Corrected artifact paths: stats files live under mutants/, not the
project root.
- The trampolined source file from the sandbox so survivors can be
inspected even outside `mutmut show`.
* ci(mutation): document intended manual weekly cadence in trigger comment
* ci(mutation): generate ACH-style report with embedded function bodies
Replaces the inline bash markdown generation with a Python script that:
- Groups survivors by function (one section per function, function body
shown once per section, surviving mutants nested as subsections)
- Embeds each enclosing function's source via Python AST (so the agent
has full context, not just a 3-line `mutmut show` diff)
- Inlines the existing test file(s) listed in [tool.mutmut].tests_dir
- Writes an ACH-style task description at the bottom following the
prompt template from arXiv 2501.12862
Output goes to mutation-report.md (artifact) and the head of the file
is appended to $GITHUB_STEP_SUMMARY for at-a-glance visibility.
* fix(mutation report): correctly parse function names with leading underscores
mutmut's mutant-name prefix is x_ (single underscore), so a function
named _foo produces mutants x__foo__mutmut_N. The previous regex
\.x__(.+)__mutmut_ ate the function's leading underscore as part of
the prefix. Changed to \.x_(.+)__mutmut_ so leading underscores are
preserved in the captured function name; verified for normal, leading-
underscore, and dunder-method names.
* feat(mutation report): full Meta ACH-style rendering with MUTANT delimiters
For each surviving mutant, parse the mutmut sandbox trampoline file and
render the mutated function as it appears in the source — with the
differing lines wrapped in `# MUTANT START` / `# MUTANT END` comments,
matching the format from Meta's ACH paper (arXiv 2501.12862, Table 1).
Renames the function header back to its original name so the agent sees
the function as it would appear in the file. Falls back to the unified
diff if the trampoline lookup fails.
Handles replace, insert, and delete diff ops; uses difflib's
SequenceMatcher to find the differing line ranges.
The unified diff is preserved in a collapsible <details> block as
secondary context.
* ci(mutation): scope to whole management_endpoints folder, drop temp push trigger
Final scope before merge:
- paths_to_mutate / tests_dir broadened from one file to the entire
management_endpoints source/test folders
- Trigger is now `workflow_dispatch` only — the temporary push: block
used during workflow iteration is removed
- timeout-minutes bumped from 60 to 350 (just under the GH-hosted job
cap of 360); whole-folder mutation against ~15 files / ~7.5k LOC can
take a few hours
- Artifact path for the trampoline files glob-expanded to cover all
files under mutants/litellm/proxy/management_endpoints/
* fix(mutation report): warn when multiple functions in a file share a name
Addresses the Greptile review concern: ast.walk's first-match-wins
behavior could embed the wrong function body when a file defines the
same name in multiple places (e.g., a module-level helper and a class
method). mutmut's mutant identifier does not carry class context, so
we can't always determine which definition was mutated.
find_function_in_file now returns the start line of every matching
definition; render() surfaces a "Note: N functions named X" warning
in the report when there is more than one match. The first match is
still embedded as the body — the warning tells the reader to verify
manually instead of silently using the wrong context.
Smoke-tested against the existing artifact: single-match files render
unchanged.
* Fix mutation report anchors
* Fix mutation report TOC anchors
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
