scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-26 01:12:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 99

chore(mutmut): include the behavior suite in tests_dir + G5 triage stub

Browse source 
Slice 15 of the management-endpoints behavior-pinning effort. Appends
``tests/proxy_behavior/management/`` to ``[tool.mutmut].tests_dir`` so
the existing mutation-test workflow runs against both the legacy mock
suite AND the new behavior suite — the latter is where the regression
signal will actually surface.

Adds a stub at ``tests/proxy_behavior/management/mutmut_triage/pr1.md``
documenting the G5 triage protocol (zero unreviewed survivors in the 6
Tier-1 handler functions) and a placeholder baseline-metrics table to
fill in after the first manually-triggered mutmut run completes — runs
take hours and run on a manual cadence, so PR1 ships with the wiring +
protocol, not the numbers. The actual baseline is recorded in a
follow-up once ``gh workflow run mutation-test.yml`` finishes.

The kill rate stays telemetry-only, never a gate. G5 (per-survivor
classification) is the binding mutation gate.

Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
This commit is contained in:
Yuneng Jiang 2026-05-19 21:57:43 -07:00
parent cd34004fe6
commit 8753969a73
No known key found for this signature in database
2 changed files with 71 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
pyproject.toml
View file

@ -287,6 +287,11 @@ paths_to_mutate = [
]
tests_dir = [
"tests/test_litellm/proxy/management_endpoints/",
# PR1 (key Tier-1) behavior-pinning suite — see
# tests/proxy_behavior/management/regression_replay/README.md for the
# G4 evidence and tests/proxy_behavior/management/mutmut_triage/pr1.md
# for the G5 survivor-classification protocol.
"tests/proxy_behavior/management/",
]
also_copy = [
"litellm/",

66
tests/proxy_behavior/management/mutmut_triage/pr1.md Normal file
View file

@ -0,0 +1,66 @@
# G5 — Mutmut survivor triage for PR1 (Key Tier-1)
PR1 wires the new behavior suite into `[tool.mutmut].tests_dir` in
`pyproject.toml`. This file records the **first manually-triggered mutmut
run** that exercises `litellm/proxy/management_endpoints/key_management_endpoints.py`
against the new suite, and classifies every surviving mutant inside the 6
Tier-1 handler functions:
* `generate_key_fn` (and helpers `_common_key_generation_helper`,
`key_generation_check`, `_team_key_generation_check`,
`_team_key_operation_team_member_check`, `_get_user_in_team`)
* `update_key_fn` (and `_check_key_admin_access` helper)
* `regenerate_key_fn` (and `_get_and_validate_existing_key` helper)
* `info_key_fn`
* `list_keys`
* `delete_key_fn`
## How to run
```bash
# Manual trigger of the workflow:
gh workflow run mutation-test.yml --ref litellm_/silly-wright-1b8559
gh run watch
```
Or locally (~hours, plan around it):
```bash
uv run --with mutmut==3.5.0 mutmut run
```
Once the run completes, download `mutation-report.md` from the workflow
artifact and:
1. Filter the survivors to the 6 handler functions listed above.
2. For each surviving mutant in those functions, add a row to the table
below classifying it **killed** (write a new test → mutant dies →
rerun confirms) or **accepted** with a one-line reason (equivalent
mutation, defensive-only branch, unreachable under any realistic
world-seed).
G5 is the binding gate for the mutation signal: **zero unreviewed
survivors** in the Tier-1 handler set. The aggregate kill rate is recorded
as telemetry under "Baseline metrics" below — never as a goalpost.
## Baseline metrics (filled after the first mutmut run lands)
| Metric | Value | Notes |
|--------|-------|-------|
| Surviving mutants (all paths_to_mutate) | _TBD_ | |
| Surviving mutants in Tier-1 handler funcs | _TBD_ | G5 gate target: 0 unreviewed |
| Killed mutants | _TBD_ | |
| Kill rate (killed / (killed + survived)) | _TBD_ | Telemetry only |
| Wall-clock minutes | _TBD_ | |
## Survivor triage
| # | Mutant location (file:line, op) | Classification | Reason / new test |
|---|---------------------------------|----------------|-------------------|
| _none yet — first mutmut run pending_ | | | |
## PR2 / PR3 delta
When PR2 (Team Tier-1) ships, append a new "Baseline metrics" block here
keyed on the PR — never overwrite. The headline number to report is the
**delta** vs. PR1's baseline, not the absolute kill rate.