mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
9146667f80
* fix(ci): stop the mutation report publishing a score it never measured Run 32475268575 was the first dispatch of this workflow since May. Every setup step passed and mutmut generated all 48 mutant files, so the suspected zero-mutants bug is not what stops it. It dies in the stats phase, where mutmut times the configured test set once up front. That set included tests/proxy_behavior/management/, a behaviour tier that talks to a real seeded database, so the run ended having mutated nothing. Narrow tests_dir to the unit tier that maps to paths_to_mutate. Run 32476663383 proved a Postgres service is not enough on its own: with a schema but no seed rows the same test fails on a foreign key instead, and a mutation score is only meaningful against the tests that claim to cover the mutated code. The second half is the one that matters. With no results at all, mutation_report.py printed "No surviving mutants, the test suite caught every mutation" and exited 0, so a run that mutated nothing published a perfect score. It now separates no survivors from no results, says which it got, and exits 1. * fix(ci): count mutmut's multi-word verdicts as results The verdict capture was `\w+`, so it matched only single-word statuses. mutmut's status_by_exit_code table has four that are not: `no tests`, `not checked`, `caught by type check` and `check was interrupted by user`. A finished run made entirely of those parsed as zero results, which is exactly the state this script now treats as an unfinished run, so it would have failed a run that had in fact completed. The regression test asserting `reported == 2` on a three-verdict fixture was codifying that, and now asserts 3. A second test walks all four multi-word statuses and checks the report does not call the run unfinished. Caught by Greptile on #37825. * fix(ci): keep the saml tests out of the mutmut stats phase Run 32477695014 got past the database blocker and ran 208 of the configured tests, then ended on one error: test_saml_sso.py builds an x509 certificate in a fixture, and inside mutmut's mutants/ sandbox cryptography's hash classes are imported under a second identity, so .sign() rejects the SHA256 instance with "Algorithm must be a registered hash algorithm". That is a property of the sandbox, not of the tests or the code being mutated, and one erroring test ends the stats phase before a single mutant runs. * fix(ci): only claim a clean sweep when something was shown to be killed `mutmut results` skips killed mutants by design, so its silence means either that everything was killed or that nothing ran. Counting the verdicts it does print cannot tell those apart, which left the report still able to say the suite caught every mutation on a run whose mutants were all `no tests` or `not checked`. The clean-sweep sentence is now gated on mutmut-cicd-stats.json reporting a non-zero killed count, which is the only signal that positively distinguishes the two. Without it the report says so in as many words and main returns 1. A run with zero kills and a stats file says that too. The test asserting a non-killed run was not called unfinished was codifying the same confusion; it is replaced by three that pin each branch. Caught by Greptile on #37825. * fix(ci): treat stats that count survivors the report never listed as untrusted clean_sweep_is_provable passed on any positive kill count, so a stats file reporting 48 killed and 3 survived, next to a `mutmut results` that listed no survivors, still published a clean sweep. The two sources contradict each other there, and neither one is worth believing. It now requires the stats file to agree that nothing survived, and the report says which disagreement it found. * fix(ci): refuse a clean sweep while mutants never reached the tests A run can end with kills, no survivors, and a pile of mutants marked no tests, skipped, suspicious, timeout or segfault. Those never got put in front of the suite, so "caught every mutation" says more than the run measured. The verdict now names which of them it found and withholds the pass, and the status list those five come from is one constant the summary and the verdict share. * fix(ci): read anything that is not a kill or a survivor as unresolved The unresolved statuses were a list of five, so a run ending in a status the reporter had never met, "check was interrupted by user" among them, still counted as a clean sweep. The rule is now the other way round: killed, survived and total are the keys with a meaning here, and every other non-zero count is a mutant that did not reach the tests, whatever mutmut chose to call it.
139 lines
4.8 KiB
Python
139 lines
4.8 KiB
Python
"""Tests for scripts/mutation_report.py.
|
|
|
|
The report is the only thing anyone reads after a mutation run, so the one thing it
|
|
must never do is describe a run that produced nothing as a run that killed everything.
|
|
`render` decides that wording and `get_survivors` supplies the evidence for it, so both
|
|
are tested directly.
|
|
"""
|
|
|
|
import importlib.util
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
_MODULE_PATH = _REPO_ROOT / "scripts" / "mutation_report.py"
|
|
_spec = importlib.util.spec_from_file_location("mutation_report", _MODULE_PATH)
|
|
report = importlib.util.module_from_spec(_spec)
|
|
sys.modules[_spec.name] = report
|
|
_spec.loader.exec_module(report)
|
|
|
|
_CONFIG = {"paths_to_mutate": ["litellm/proxy/management_endpoints/"], "tests_dir": ["tests/"]}
|
|
|
|
|
|
def test_a_run_that_reported_nothing_is_not_a_clean_sweep():
|
|
rendered = report.render(_CONFIG, report.MutmutResults(survivors=(), reported=0), None)
|
|
|
|
assert "not a passing score" in rendered
|
|
assert "caught every mutation" not in rendered
|
|
|
|
|
|
def test_a_run_that_killed_every_mutant_says_so():
|
|
rendered = report.render(
|
|
_CONFIG, report.MutmutResults(survivors=(), reported=0), {"killed": 48, "survived": 0}
|
|
)
|
|
|
|
assert "caught every mutation" in rendered
|
|
assert "not a passing score" not in rendered
|
|
|
|
|
|
def test_stats_counting_survivors_results_never_listed_is_not_a_clean_sweep():
|
|
rendered = report.render(
|
|
_CONFIG, report.MutmutResults(survivors=(), reported=0), {"killed": 48, "survived": 3}
|
|
)
|
|
|
|
assert "not a passing score" in rendered
|
|
assert "caught every mutation" not in rendered
|
|
assert "3 surviving mutant(s)" in rendered
|
|
|
|
|
|
def test_mutants_that_never_reached_the_tests_are_not_a_clean_sweep():
|
|
rendered = report.render(
|
|
_CONFIG,
|
|
report.MutmutResults(survivors=(), reported=0),
|
|
{"killed": 48, "survived": 0, "no_tests": 4, "timeout": 1},
|
|
)
|
|
|
|
assert "not a passing score" in rendered
|
|
assert "caught every mutation" not in rendered
|
|
assert "4 no tests" in rendered
|
|
assert "1 timeout" in rendered
|
|
|
|
|
|
def test_a_status_the_reporter_has_never_met_still_blocks_a_clean_sweep():
|
|
rendered = report.render(
|
|
_CONFIG,
|
|
report.MutmutResults(survivors=(), reported=0),
|
|
{"killed": 48, "survived": 0, "check_was_interrupted_by_user": 2},
|
|
)
|
|
|
|
assert "not a passing score" in rendered
|
|
assert "caught every mutation" not in rendered
|
|
assert "2 check was interrupted by user" in rendered
|
|
|
|
|
|
def test_no_survivors_without_a_kill_is_not_a_clean_sweep():
|
|
rendered = report.render(
|
|
_CONFIG, report.MutmutResults(survivors=(), reported=48), {"killed": 0, "survived": 0}
|
|
)
|
|
|
|
assert "not a passing score" in rendered
|
|
assert "caught every mutation" not in rendered
|
|
|
|
|
|
def test_no_survivors_and_no_stats_cannot_claim_a_sweep():
|
|
"""`mutmut results` never lists killed mutants, so with the stats file missing an
|
|
empty survivor list is equally consistent with a perfect run and a dead one."""
|
|
rendered = report.render(_CONFIG, report.MutmutResults(survivors=(), reported=48), None)
|
|
|
|
assert "not a passing score" in rendered
|
|
assert "caught every mutation" not in rendered
|
|
|
|
|
|
def test_survivors_are_read_out_of_the_verdicts_they_came_with(monkeypatch):
|
|
class _Proc:
|
|
stdout = (
|
|
"litellm.proxy.management_endpoints.key_management_endpoints.x_1: killed\n"
|
|
"litellm.proxy.management_endpoints.key_management_endpoints.x_2: survived\n"
|
|
"litellm.proxy.management_endpoints.key_management_endpoints.x_3: no tests\n"
|
|
"not a verdict line at all\n"
|
|
)
|
|
|
|
monkeypatch.setattr(report.subprocess, "run", lambda *a, **k: _Proc())
|
|
|
|
results = report.get_survivors()
|
|
|
|
assert results.survivors == (
|
|
"litellm.proxy.management_endpoints.key_management_endpoints.x_2",
|
|
)
|
|
assert results.reported == 3
|
|
|
|
|
|
def test_every_multi_word_verdict_mutmut_can_emit_still_counts(monkeypatch):
|
|
class _Proc:
|
|
stdout = "".join(
|
|
f"litellm.proxy.management_endpoints.key_management_endpoints.x_{i}: {verdict}\n"
|
|
for i, verdict in enumerate(
|
|
(
|
|
"no tests",
|
|
"not checked",
|
|
"caught by type check",
|
|
"check was interrupted by user",
|
|
)
|
|
)
|
|
)
|
|
|
|
monkeypatch.setattr(report.subprocess, "run", lambda *a, **k: _Proc())
|
|
|
|
results = report.get_survivors()
|
|
|
|
assert results.survivors == ()
|
|
assert results.reported == 4
|
|
|
|
|
|
def test_an_empty_mutmut_results_reports_nothing_rather_than_zero_survivors(monkeypatch):
|
|
class _Proc:
|
|
stdout = ""
|
|
|
|
monkeypatch.setattr(report.subprocess, "run", lambda *a, **k: _Proc())
|
|
|
|
assert report.get_survivors() == report.MutmutResults(survivors=(), reported=0)
|