mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
9146667f80
* fix(ci): stop the mutation report publishing a score it never measured Run 32475268575 was the first dispatch of this workflow since May. Every setup step passed and mutmut generated all 48 mutant files, so the suspected zero-mutants bug is not what stops it. It dies in the stats phase, where mutmut times the configured test set once up front. That set included tests/proxy_behavior/management/, a behaviour tier that talks to a real seeded database, so the run ended having mutated nothing. Narrow tests_dir to the unit tier that maps to paths_to_mutate. Run 32476663383 proved a Postgres service is not enough on its own: with a schema but no seed rows the same test fails on a foreign key instead, and a mutation score is only meaningful against the tests that claim to cover the mutated code. The second half is the one that matters. With no results at all, mutation_report.py printed "No surviving mutants, the test suite caught every mutation" and exited 0, so a run that mutated nothing published a perfect score. It now separates no survivors from no results, says which it got, and exits 1. * fix(ci): count mutmut's multi-word verdicts as results The verdict capture was `\w+`, so it matched only single-word statuses. mutmut's status_by_exit_code table has four that are not: `no tests`, `not checked`, `caught by type check` and `check was interrupted by user`. A finished run made entirely of those parsed as zero results, which is exactly the state this script now treats as an unfinished run, so it would have failed a run that had in fact completed. The regression test asserting `reported == 2` on a three-verdict fixture was codifying that, and now asserts 3. A second test walks all four multi-word statuses and checks the report does not call the run unfinished. Caught by Greptile on #37825. * fix(ci): keep the saml tests out of the mutmut stats phase Run 32477695014 got past the database blocker and ran 208 of the configured tests, then ended on one error: test_saml_sso.py builds an x509 certificate in a fixture, and inside mutmut's mutants/ sandbox cryptography's hash classes are imported under a second identity, so .sign() rejects the SHA256 instance with "Algorithm must be a registered hash algorithm". That is a property of the sandbox, not of the tests or the code being mutated, and one erroring test ends the stats phase before a single mutant runs. * fix(ci): only claim a clean sweep when something was shown to be killed `mutmut results` skips killed mutants by design, so its silence means either that everything was killed or that nothing ran. Counting the verdicts it does print cannot tell those apart, which left the report still able to say the suite caught every mutation on a run whose mutants were all `no tests` or `not checked`. The clean-sweep sentence is now gated on mutmut-cicd-stats.json reporting a non-zero killed count, which is the only signal that positively distinguishes the two. Without it the report says so in as many words and main returns 1. A run with zero kills and a stats file says that too. The test asserting a non-killed run was not called unfinished was codifying the same confusion; it is replaced by three that pin each branch. Caught by Greptile on #37825. * fix(ci): treat stats that count survivors the report never listed as untrusted clean_sweep_is_provable passed on any positive kill count, so a stats file reporting 48 killed and 3 survived, next to a `mutmut results` that listed no survivors, still published a clean sweep. The two sources contradict each other there, and neither one is worth believing. It now requires the stats file to agree that nothing survived, and the report says which disagreement it found. * fix(ci): refuse a clean sweep while mutants never reached the tests A run can end with kills, no survivors, and a pile of mutants marked no tests, skipped, suspicious, timeout or segfault. Those never got put in front of the suite, so "caught every mutation" says more than the run measured. The verdict now names which of them it found and withholds the pass, and the status list those five come from is one constant the summary and the verdict share. * fix(ci): read anything that is not a kill or a survivor as unresolved The unresolved statuses were a list of five, so a run ending in a status the reporter had never met, "check was interrupted by user" among them, still counted as a clean sweep. The rule is now the other way round: killed, survived and total are the keys with a meaning here, and every other non-zero count is a mutant that did not reach the tests, whatever mutmut chose to call it. |
||
|---|---|---|
| .. | ||
| adaptive_router_demo | ||
| health_check | ||
| benchmark_anthropic_messages_perf.py | ||
| benchmark_chat_completions_perf.py | ||
| benchmark_mock.py | ||
| benchmark_model_response_creator.py | ||
| benchmark_proxy_vs_provider.py | ||
| benchmark_streaming_chunk_overhead.py | ||
| budget_ratchet_check.py | ||
| check_test_quality.py | ||
| check_type_discipline.py | ||
| create_litellm_branch.ps1 | ||
| create_litellm_branch.sh | ||
| create_team_key_and_submit_guardrail.sh | ||
| eval_compression.py | ||
| gate_slot_lock.py | ||
| install-cli.sh | ||
| install.sh | ||
| install_git_hooks.sh | ||
| mock_bedrock_passthrough_target.py | ||
| mock_grayswan_timeout_server.py | ||
| mutation_report.py | ||
| pre_commit_lint.sh | ||
| prisma_generate_if_needed.py | ||
| ruff_strict_gate.py | ||
| test_agent_mcp_endpoints.sh | ||
| test_guardrails_register_endpoints.sh | ||
| test_quality_gate.py | ||
| test_tool_allowlist_script.py | ||
| tpm_headline_test.sh | ||
| type_check_gate.py | ||
| type_discipline_gate.py | ||
| verify_adaptive_router.py | ||
| with_dashboard_node.sh | ||