Timing the gate against three real open pull requests turned up two ways the
test selection went wide. A private module never matched its test file, since
litellm/_redis.py globs test__redis*.py while the file is test_redis.py, and the
miss fell through to the mirror directory. For a module at the top of litellm/
that directory is tests/test_litellm itself, so every mutant would have run the
entire unit suite. The same fallback pulled all of tests/test_litellm/caching
into a caching diff and picked up an unrelated order-dependent s3 test, which
fails on an unmutated tree and aborts the run before a single mutant executes.
Private modules now also try the underscore-stripped spelling, and the directory
fallback stops at the mirror root instead of returning it. Overload stubs are
deduplicated too: they repeat the implementation's name, so a touched signature
asked for the same function three times and burned three slots of --max-functions.
The existing mutation workflow is manual and mutates a whole folder, which takes
hours, so nothing about a pull request's own test quality gets checked before it
merges.
This adds the diff-scoped half, following Google's approach in "State of Mutation
Testing at Google": mutate what the change touched, not the codebase.
scripts/mutation_diff_scope.py reads the diff against the merge base, rewrites
[tool.mutmut] so paths_to_mutate is the changed production files and tests_dir is
the tests mirroring them, then emits mutant-name globs for the functions holding
the changed lines. mutmut trampolines per function and per method, so that is the
smallest unit it can be asked to run.
The new workflow is advisory: results land in the job summary and an artifact,
and nothing blocks a merge. A --max-functions cap bounds the worst case and
prints what it dropped rather than truncating quietly.
The report script now counts only the mutants a run actually executed, since a
diff-scoped run leaves everything else at "not checked" and the old summary
folded those into the score.
