mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
refactor(lint): collapse type/lint budgets to a single per-rule limit (#31883)
* chore(lint): raise basedpyright per-rule slack to 50% of baseline The per-rule ceilings in basedpyright-code-budget.json sat at roughly 10% slack over baseline, which several in-flight PRs are already bumping into. Raise the slack on every rule to at least 50% of its baseline so there is ample headroom for a long while, while never lowering any rule that already had more generous slack (e.g. reportReturnType stays at 100). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(lint): collapse type/lint budgets to a single per-rule limit The three non-frontend budget files (ruff-strict, type-discipline, basedpyright-code) tracked a per-rule baseline and slack whose sum was the ceiling. Nothing consumed the split beyond that sum, so this replaces both keys with a single limit equal to the old baseline + slack; the original baselines live in git history if anyone needs them. The gate scripts and the ratchet guard now read limit directly. lint-budget-update no longer re-captures raw counts; it ratchets each rule's limit down by the number of violations this branch cleared since its branch point (the merge-base), so the granted headroom shrinks by exactly what was fixed and a limit never rises. The ratchet guard reads either schema so it still compares correctly across the migration boundary. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): surface staged-vs-working parity for pre-commit and budget-update make pre-commit selects which checks to run from the staged index but runs the linters over the working tree, so unstaged edits to tracked files and untracked files skew a green/red away from what a commit of only the staged changes would produce. There is no safe in-place way to lint the index, so the script now warns when unstaged or untracked changes are present, and CLAUDE.md documents that you must stage everything first for both make pre-commit and make lint-budget-update to predict CI correctly. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * docs(lint): list type-discipline budget in lint-budget-update instruction --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
This commit is contained in:
Mateo Wang
GitHub
parent
13b590c8ec
commit
e141596204
14 changed files with 515 additions and 589 deletions
|
|
@ -33,9 +33,9 @@ If you ever make public-facing PR descriptions, comments, issues, commit message
|
|||
|
||||
Don't hesitate to use values in .env to get needed API keys and other secrets, as long as you never add them to conversation history, commit them, or include them in GitHub issues / PRs
|
||||
|
||||
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed
|
||||
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed. For `make pre-commit` to work properly you must stage your changes first (git add): it reports CI red or green based on what would happen if you committed your staged changes, but it runs the linters over the working tree, so any unstaged edits to tracked files or untracked files are folded into the result and will skew it away from what CI (which only sees your commit) would report
|
||||
|
||||
When you fix violations gated by `ruff-strict-budget.json` or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered baselines so the ceilings ratchet down instead of leaving stale headroom
|
||||
When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-budget.json`, or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered limits so the ceilings ratchet down instead of leaving stale headroom. It lowers each rule's limit by the number of violations this branch cleared since its branch point and never raises one, measured against the working tree, so stage exactly the fixes you're committing before running it; crediting unstaged fixes you won't commit would over-tighten the limits and turn CI red once the committed subset is checked
|
||||
|
||||
If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing `reportAny` / `reportExplicitAny` closer to their basedpyright ceilings, just validate it in the caller with Pydantic (a model or `TypeAdapter` that returns the typed thing or raises will do) and then pass the now typed variable in
|
||||
|
||||
|
|
|
|||
23
Makefile
23
Makefile
|
|
@ -5,7 +5,7 @@
|
|||
test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
|
||||
test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
|
||||
info lint lint-dev format \
|
||||
lint-basedpyright lint-basedpyright-budget-update \
|
||||
lint-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
|
||||
lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
|
||||
install-dev install-proxy-dev install-test-deps install-hooks \
|
||||
install-helm-unittest check-circular-imports check-import-safety pre-commit \
|
||||
|
|
@ -27,12 +27,12 @@ help:
|
|||
@echo " make lint - Run all linting (Ruff, basedpyright, format check, circular imports, import safety)"
|
||||
@echo " make lint-ruff - Run Ruff linting only"
|
||||
@echo " make lint-basedpyright - Run basedpyright strict, gated by per-rule error counts"
|
||||
@echo " make lint-basedpyright-budget-update - Re-capture the basedpyright per-rule budget (ratchet)"
|
||||
@echo " make lint-basedpyright-budget-update - Ratchet basedpyright limits down by what this branch fixed"
|
||||
@echo " make lint-format - Check ruff format formatting (matches CI)"
|
||||
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its ceiling"
|
||||
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its limit"
|
||||
@echo " make lint-gate - Strict ruff gate in CI-parity mode (fetches staging, simulates the merge)"
|
||||
@echo " make lint-ruff-budget-update - Re-capture per-rule baselines in ruff-strict-budget.json (ratchet)"
|
||||
@echo " make lint-budget-update - Re-capture all ratchet budgets (ruff + basedpyright)"
|
||||
@echo " make lint-ruff-budget-update - Ratchet ruff-strict-budget.json limits down by what this branch fixed"
|
||||
@echo " make lint-budget-update - Ratchet all budgets down (ruff + type-discipline + basedpyright)"
|
||||
@echo " make check-circular-imports - Check for circular imports"
|
||||
@echo " make check-import-safety - Check import safety"
|
||||
@echo " make test - Run all tests"
|
||||
|
|
@ -164,7 +164,9 @@ lint-basedpyright: install-dev lint-fetch-base
|
|||
lint-type-discipline: install-dev lint-fetch-base
|
||||
$(UV_RUN) python scripts/type_discipline_gate.py --base origin/litellm_internal_staging
|
||||
|
||||
lint-basedpyright-budget-update: install-dev
|
||||
# --update lowers each limit by what this branch fixed since its branch point, so
|
||||
# it needs the base ref fetched to resolve the merge-base.
|
||||
lint-basedpyright-budget-update: install-dev lint-fetch-base
|
||||
($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --update
|
||||
|
||||
lint-format: format-check
|
||||
|
|
@ -177,11 +179,14 @@ lint-ruff-budget: install-dev
|
|||
lint-gate: install-dev lint-fetch-base
|
||||
$(UV_RUN) python scripts/ruff_strict_gate.py --base origin/litellm_internal_staging
|
||||
|
||||
lint-ruff-budget-update: install-dev
|
||||
lint-ruff-budget-update: install-dev lint-fetch-base
|
||||
$(UV_RUN) python scripts/ruff_strict_gate.py --update
|
||||
|
||||
# Ratchet all budgets in one shot (ruff strict + basedpyright)
|
||||
lint-budget-update: lint-ruff-budget-update lint-basedpyright-budget-update
|
||||
lint-type-discipline-budget-update: install-dev lint-fetch-base
|
||||
$(UV_RUN) python scripts/type_discipline_gate.py --update
|
||||
|
||||
# Ratchet all budgets in one shot (ruff strict + type-discipline + basedpyright)
|
||||
lint-budget-update: lint-ruff-budget-update lint-type-discipline-budget-update lint-basedpyright-budget-update
|
||||
|
||||
check-circular-imports: install-dev
|
||||
cd litellm && $(UV_RUN) python ../tests/documentation_tests/test_circular_imports.py && cd ..
|
||||
|
|
|
|||
|
|
@ -1,194 +1,146 @@
|
|||
{
|
||||
"reportAny": {
|
||||
"baseline": 24989,
|
||||
"slack": 2500
|
||||
"limit": 37484
|
||||
},
|
||||
"reportArgumentType": {
|
||||
"baseline": 1814,
|
||||
"slack": 180
|
||||
"limit": 2721
|
||||
},
|
||||
"reportAssignmentType": {
|
||||
"baseline": 220,
|
||||
"slack": 22
|
||||
"limit": 330
|
||||
},
|
||||
"reportAttributeAccessIssue": {
|
||||
"baseline": 346,
|
||||
"slack": 35
|
||||
"limit": 519
|
||||
},
|
||||
"reportCallIssue": {
|
||||
"baseline": 87,
|
||||
"slack": 10
|
||||
"limit": 131
|
||||
},
|
||||
"reportConstantRedefinition": {
|
||||
"baseline": 39,
|
||||
"slack": 4
|
||||
"limit": 59
|
||||
},
|
||||
"reportDeprecated": {
|
||||
"baseline": 217,
|
||||
"slack": 22
|
||||
"limit": 326
|
||||
},
|
||||
"reportDuplicateImport": {
|
||||
"baseline": 28,
|
||||
"slack": 3
|
||||
"limit": 42
|
||||
},
|
||||
"reportExplicitAny": {
|
||||
"baseline": 6931,
|
||||
"slack": 700
|
||||
"limit": 10397
|
||||
},
|
||||
"reportFunctionMemberAccess": {
|
||||
"baseline": 7,
|
||||
"slack": 3
|
||||
"limit": 11
|
||||
},
|
||||
"reportGeneralTypeIssues": {
|
||||
"baseline": 151,
|
||||
"slack": 15
|
||||
"limit": 227
|
||||
},
|
||||
"reportIncompatibleMethodOverride": {
|
||||
"baseline": 52,
|
||||
"slack": 5
|
||||
"limit": 78
|
||||
},
|
||||
"reportIncompatibleVariableOverride": {
|
||||
"baseline": 8,
|
||||
"slack": 3
|
||||
"limit": 12
|
||||
},
|
||||
"reportInconsistentOverload": {
|
||||
"baseline": 12,
|
||||
"slack": 3
|
||||
"limit": 18
|
||||
},
|
||||
"reportIndexIssue": {
|
||||
"baseline": 26,
|
||||
"slack": 3
|
||||
"limit": 39
|
||||
},
|
||||
"reportInvalidTypeForm": {
|
||||
"baseline": 23,
|
||||
"slack": 3
|
||||
"limit": 35
|
||||
},
|
||||
"reportInvalidTypeVarUse": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"reportMatchNotExhaustive": {
|
||||
"baseline": 1,
|
||||
"slack": 0
|
||||
"limit": 2
|
||||
},
|
||||
"reportMissingParameterType": {
|
||||
"baseline": 3933,
|
||||
"slack": 390
|
||||
"limit": 5900
|
||||
},
|
||||
"reportMissingTypeArgument": {
|
||||
"baseline": 10612,
|
||||
"slack": 1000
|
||||
"limit": 15918
|
||||
},
|
||||
"reportMissingTypeStubs": {
|
||||
"baseline": 27,
|
||||
"slack": 10
|
||||
"limit": 41
|
||||
},
|
||||
"reportOperatorIssue": {
|
||||
"baseline": 6,
|
||||
"slack": 3
|
||||
"limit": 9
|
||||
},
|
||||
"reportOptionalCall": {
|
||||
"baseline": 4,
|
||||
"slack": 3
|
||||
"limit": 7
|
||||
},
|
||||
"reportOptionalIterable": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"reportOptionalMemberAccess": {
|
||||
"baseline": 724,
|
||||
"slack": 72
|
||||
"limit": 1086
|
||||
},
|
||||
"reportOptionalOperand": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"reportOptionalSubscript": {
|
||||
"baseline": 11,
|
||||
"slack": 3
|
||||
"limit": 17
|
||||
},
|
||||
"reportPossiblyUnboundVariable": {
|
||||
"baseline": 52,
|
||||
"slack": 10
|
||||
"limit": 78
|
||||
},
|
||||
"reportPrivateUsage": {
|
||||
"baseline": 1625,
|
||||
"slack": 160
|
||||
"limit": 2438
|
||||
},
|
||||
"reportRedeclaration": {
|
||||
"baseline": 8,
|
||||
"slack": 3
|
||||
"limit": 12
|
||||
},
|
||||
"reportReturnType": {
|
||||
"baseline": 126,
|
||||
"slack": 100
|
||||
"limit": 226
|
||||
},
|
||||
"reportTypedDictNotRequiredAccess": {
|
||||
"baseline": 20,
|
||||
"slack": 3
|
||||
"limit": 30
|
||||
},
|
||||
"reportUndefinedVariable": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"reportUnknownArgumentType": {
|
||||
"baseline": 30603,
|
||||
"slack": 3000
|
||||
"limit": 45905
|
||||
},
|
||||
"reportUnknownLambdaType": {
|
||||
"baseline": 75,
|
||||
"slack": 10
|
||||
"limit": 113
|
||||
},
|
||||
"reportUnknownMemberType": {
|
||||
"baseline": 27037,
|
||||
"slack": 2500
|
||||
"limit": 40556
|
||||
},
|
||||
"reportUnknownParameterType": {
|
||||
"baseline": 13612,
|
||||
"slack": 1000
|
||||
"limit": 20418
|
||||
},
|
||||
"reportUnknownVariableType": {
|
||||
"baseline": 21445,
|
||||
"slack": 2000
|
||||
"limit": 32168
|
||||
},
|
||||
"reportUnnecessaryCast": {
|
||||
"baseline": 118,
|
||||
"slack": 10
|
||||
"limit": 177
|
||||
},
|
||||
"reportUnnecessaryComparison": {
|
||||
"baseline": 683,
|
||||
"slack": 100
|
||||
"limit": 1025
|
||||
},
|
||||
"reportUnnecessaryContains": {
|
||||
"baseline": 4,
|
||||
"slack": 3
|
||||
"limit": 7
|
||||
},
|
||||
"reportUnnecessaryIsInstance": {
|
||||
"baseline": 808,
|
||||
"slack": 80
|
||||
"limit": 1212
|
||||
},
|
||||
"reportUntypedBaseClass": {
|
||||
"baseline": 110,
|
||||
"slack": 11
|
||||
"limit": 165
|
||||
},
|
||||
"reportUntypedFunctionDecorator": {
|
||||
"baseline": 22,
|
||||
"slack": 3
|
||||
"limit": 33
|
||||
},
|
||||
"reportUnusedClass": {
|
||||
"baseline": 22,
|
||||
"slack": 3
|
||||
"limit": 33
|
||||
},
|
||||
"reportUnusedFunction": {
|
||||
"baseline": 137,
|
||||
"slack": 10
|
||||
"limit": 206
|
||||
},
|
||||
"reportUnusedImport": {
|
||||
"baseline": 670,
|
||||
"slack": 50
|
||||
"limit": 1005
|
||||
},
|
||||
"reportUnusedVariable": {
|
||||
"baseline": 865,
|
||||
"slack": 50
|
||||
"limit": 1298
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,490 +1,368 @@
|
|||
{
|
||||
"ANN001": {
|
||||
"baseline": 2865,
|
||||
"slack": 287
|
||||
"limit": 3152
|
||||
},
|
||||
"ANN002": {
|
||||
"baseline": 64,
|
||||
"slack": 5
|
||||
"limit": 69
|
||||
},
|
||||
"ANN003": {
|
||||
"baseline": 759,
|
||||
"slack": 76
|
||||
"limit": 835
|
||||
},
|
||||
"ANN201": {
|
||||
"baseline": 1944,
|
||||
"slack": 194
|
||||
"limit": 2138
|
||||
},
|
||||
"ANN202": {
|
||||
"baseline": 858,
|
||||
"slack": 86
|
||||
"limit": 944
|
||||
},
|
||||
"ANN204": {
|
||||
"baseline": 658,
|
||||
"slack": 66
|
||||
"limit": 724
|
||||
},
|
||||
"ANN205": {
|
||||
"baseline": 117,
|
||||
"slack": 10
|
||||
"limit": 127
|
||||
},
|
||||
"ANN206": {
|
||||
"baseline": 120,
|
||||
"slack": 10
|
||||
"limit": 130
|
||||
},
|
||||
"ANN401": {
|
||||
"baseline": 1886,
|
||||
"slack": 189
|
||||
"limit": 2075
|
||||
},
|
||||
"ASYNC230": {
|
||||
"baseline": 11,
|
||||
"slack": 3
|
||||
"limit": 14
|
||||
},
|
||||
"B004": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"B006": {
|
||||
"baseline": 180,
|
||||
"slack": 10
|
||||
"limit": 190
|
||||
},
|
||||
"B008": {
|
||||
"baseline": 490,
|
||||
"slack": 15
|
||||
"limit": 505
|
||||
},
|
||||
"B009": {
|
||||
"baseline": 79,
|
||||
"slack": 5
|
||||
"limit": 84
|
||||
},
|
||||
"B010": {
|
||||
"baseline": 187,
|
||||
"slack": 10
|
||||
"limit": 197
|
||||
},
|
||||
"B018": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"B019": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"B021": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"B026": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"B033": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"BLE001": {
|
||||
"baseline": 2854,
|
||||
"slack": 50
|
||||
"limit": 2904
|
||||
},
|
||||
"C401": {
|
||||
"baseline": 8,
|
||||
"slack": 3
|
||||
"limit": 11
|
||||
},
|
||||
"C404": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"C405": {
|
||||
"baseline": 20,
|
||||
"slack": 3
|
||||
"limit": 23
|
||||
},
|
||||
"C408": {
|
||||
"baseline": 11,
|
||||
"slack": 3
|
||||
"limit": 14
|
||||
},
|
||||
"C414": {
|
||||
"baseline": 4,
|
||||
"slack": 3
|
||||
"limit": 7
|
||||
},
|
||||
"C419": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"C901": {
|
||||
"baseline": 301,
|
||||
"slack": 15
|
||||
"limit": 316
|
||||
},
|
||||
"D419": {
|
||||
"baseline": 6,
|
||||
"slack": 3
|
||||
"limit": 9
|
||||
},
|
||||
"DTZ001": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"DTZ003": {
|
||||
"baseline": 30,
|
||||
"slack": 3
|
||||
"limit": 33
|
||||
},
|
||||
"DTZ005": {
|
||||
"baseline": 229,
|
||||
"slack": 15
|
||||
"limit": 244
|
||||
},
|
||||
"DTZ006": {
|
||||
"baseline": 10,
|
||||
"slack": 3
|
||||
"limit": 13
|
||||
},
|
||||
"DTZ007": {
|
||||
"baseline": 20,
|
||||
"slack": 3
|
||||
"limit": 23
|
||||
},
|
||||
"DTZ011": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"EXE001": {
|
||||
"baseline": 4,
|
||||
"slack": 3
|
||||
"limit": 7
|
||||
},
|
||||
"EXE002": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"F401": {
|
||||
"baseline": 20,
|
||||
"slack": 3
|
||||
"limit": 23
|
||||
},
|
||||
"FURB136": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"FURB168": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"FURB188": {
|
||||
"baseline": 49,
|
||||
"slack": 3
|
||||
"limit": 52
|
||||
},
|
||||
"I001": {
|
||||
"baseline": 258,
|
||||
"slack": 15
|
||||
"limit": 273
|
||||
},
|
||||
"LOG015": {
|
||||
"baseline": 5,
|
||||
"slack": 3
|
||||
"limit": 8
|
||||
},
|
||||
"N999": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PERF102": {
|
||||
"baseline": 27,
|
||||
"slack": 3
|
||||
"limit": 30
|
||||
},
|
||||
"PERF401": {
|
||||
"baseline": 136,
|
||||
"slack": 10
|
||||
"limit": 146
|
||||
},
|
||||
"PERF402": {
|
||||
"baseline": 6,
|
||||
"slack": 3
|
||||
"limit": 9
|
||||
},
|
||||
"PERF403": {
|
||||
"baseline": 69,
|
||||
"slack": 5
|
||||
"limit": 74
|
||||
},
|
||||
"PIE790": {
|
||||
"baseline": 263,
|
||||
"slack": 15
|
||||
"limit": 278
|
||||
},
|
||||
"PIE800": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PIE804": {
|
||||
"baseline": 21,
|
||||
"slack": 3
|
||||
"limit": 24
|
||||
},
|
||||
"PIE810": {
|
||||
"baseline": 41,
|
||||
"slack": 3
|
||||
"limit": 44
|
||||
},
|
||||
"PLC0206": {
|
||||
"baseline": 28,
|
||||
"slack": 3
|
||||
"limit": 31
|
||||
},
|
||||
"PLC0208": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PLC0414": {
|
||||
"baseline": 35,
|
||||
"slack": 3
|
||||
"limit": 38
|
||||
},
|
||||
"PLR0124": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PLR0206": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PLR0402": {
|
||||
"baseline": 6,
|
||||
"slack": 3
|
||||
"limit": 9
|
||||
},
|
||||
"PLR1704": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"PLR1711": {
|
||||
"baseline": 31,
|
||||
"slack": 3
|
||||
"limit": 34
|
||||
},
|
||||
"PLR1714": {
|
||||
"baseline": 252,
|
||||
"slack": 15
|
||||
"limit": 267
|
||||
},
|
||||
"PLR1730": {
|
||||
"baseline": 7,
|
||||
"slack": 3
|
||||
"limit": 10
|
||||
},
|
||||
"PLR2044": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PLW0127": {
|
||||
"baseline": 41,
|
||||
"slack": 3
|
||||
"limit": 44
|
||||
},
|
||||
"PLW0133": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"PLW0602": {
|
||||
"baseline": 215,
|
||||
"slack": 15
|
||||
"limit": 230
|
||||
},
|
||||
"PLW0603": {
|
||||
"baseline": 183,
|
||||
"slack": 10
|
||||
"limit": 193
|
||||
},
|
||||
"PLW1508": {
|
||||
"baseline": 188,
|
||||
"slack": 10
|
||||
"limit": 198
|
||||
},
|
||||
"PLW1510": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"PYI030": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"PYI036": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"PYI041": {
|
||||
"baseline": 9,
|
||||
"slack": 3
|
||||
"limit": 12
|
||||
},
|
||||
"PYI064": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"RET501": {
|
||||
"baseline": 35,
|
||||
"slack": 3
|
||||
"limit": 38
|
||||
},
|
||||
"RET504": {
|
||||
"baseline": 702,
|
||||
"slack": 20
|
||||
"limit": 722
|
||||
},
|
||||
"RUF010": {
|
||||
"baseline": 844,
|
||||
"slack": 30
|
||||
"limit": 874
|
||||
},
|
||||
"RUF012": {
|
||||
"baseline": 158,
|
||||
"slack": 10
|
||||
"limit": 168
|
||||
},
|
||||
"RUF015": {
|
||||
"baseline": 8,
|
||||
"slack": 3
|
||||
"limit": 11
|
||||
},
|
||||
"RUF019": {
|
||||
"baseline": 38,
|
||||
"slack": 3
|
||||
"limit": 41
|
||||
},
|
||||
"RUF022": {
|
||||
"baseline": 80,
|
||||
"slack": 5
|
||||
"limit": 85
|
||||
},
|
||||
"RUF023": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"RUF046": {
|
||||
"baseline": 5,
|
||||
"slack": 3
|
||||
"limit": 8
|
||||
},
|
||||
"RUF051": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"RUF059": {
|
||||
"baseline": 69,
|
||||
"slack": 5
|
||||
"limit": 74
|
||||
},
|
||||
"RUF100": {
|
||||
"baseline": 465,
|
||||
"slack": 15
|
||||
"limit": 480
|
||||
},
|
||||
"S110": {
|
||||
"baseline": 222,
|
||||
"slack": 15
|
||||
"limit": 237
|
||||
},
|
||||
"S112": {
|
||||
"baseline": 21,
|
||||
"slack": 3
|
||||
"limit": 24
|
||||
},
|
||||
"SIM101": {
|
||||
"baseline": 58,
|
||||
"slack": 5
|
||||
"limit": 63
|
||||
},
|
||||
"SIM102": {
|
||||
"baseline": 311,
|
||||
"slack": 15
|
||||
"limit": 326
|
||||
},
|
||||
"SIM103": {
|
||||
"baseline": 119,
|
||||
"slack": 10
|
||||
"limit": 129
|
||||
},
|
||||
"SIM113": {
|
||||
"baseline": 3,
|
||||
"slack": 3
|
||||
"limit": 6
|
||||
},
|
||||
"SIM114": {
|
||||
"baseline": 103,
|
||||
"slack": 10
|
||||
"limit": 113
|
||||
},
|
||||
"SIM115": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"SIM117": {
|
||||
"baseline": 7,
|
||||
"slack": 3
|
||||
"limit": 10
|
||||
},
|
||||
"SIM118": {
|
||||
"baseline": 104,
|
||||
"slack": 10
|
||||
"limit": 114
|
||||
},
|
||||
"SIM201": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"SIM210": {
|
||||
"baseline": 9,
|
||||
"slack": 3
|
||||
"limit": 12
|
||||
},
|
||||
"SIM211": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"SIM222": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"SIM401": {
|
||||
"baseline": 9,
|
||||
"slack": 3
|
||||
"limit": 12
|
||||
},
|
||||
"TC004": {
|
||||
"baseline": 5,
|
||||
"slack": 3
|
||||
"limit": 8
|
||||
},
|
||||
"TC005": {
|
||||
"baseline": 6,
|
||||
"slack": 3
|
||||
"limit": 9
|
||||
},
|
||||
"TID251": {
|
||||
"baseline": 2664,
|
||||
"slack": 50
|
||||
"limit": 2714
|
||||
},
|
||||
"TRY002": {
|
||||
"baseline": 528,
|
||||
"slack": 20
|
||||
"limit": 548
|
||||
},
|
||||
"TRY004": {
|
||||
"baseline": 93,
|
||||
"slack": 5
|
||||
"limit": 98
|
||||
},
|
||||
"TRY201": {
|
||||
"baseline": 409,
|
||||
"slack": 15
|
||||
"limit": 424
|
||||
},
|
||||
"TRY203": {
|
||||
"baseline": 113,
|
||||
"slack": 10
|
||||
"limit": 123
|
||||
},
|
||||
"TRY300": {
|
||||
"baseline": 853,
|
||||
"slack": 30
|
||||
"limit": 883
|
||||
},
|
||||
"UP006": {
|
||||
"baseline": 12941,
|
||||
"slack": 100
|
||||
"limit": 13041
|
||||
},
|
||||
"UP007": {
|
||||
"baseline": 2520,
|
||||
"slack": 50
|
||||
"limit": 2570
|
||||
},
|
||||
"UP008": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"UP012": {
|
||||
"baseline": 4,
|
||||
"slack": 3
|
||||
"limit": 7
|
||||
},
|
||||
"UP018": {
|
||||
"baseline": 18,
|
||||
"slack": 3
|
||||
"limit": 21
|
||||
},
|
||||
"UP024": {
|
||||
"baseline": 12,
|
||||
"slack": 3
|
||||
"limit": 15
|
||||
},
|
||||
"UP028": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"UP031": {
|
||||
"baseline": 2,
|
||||
"slack": 3
|
||||
"limit": 5
|
||||
},
|
||||
"UP032": {
|
||||
"baseline": 609,
|
||||
"slack": 20
|
||||
"limit": 629
|
||||
},
|
||||
"UP034": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"UP035": {
|
||||
"baseline": 2250,
|
||||
"slack": 50
|
||||
"limit": 2300
|
||||
},
|
||||
"UP036": {
|
||||
"baseline": 1,
|
||||
"slack": 3
|
||||
"limit": 4
|
||||
},
|
||||
"UP037": {
|
||||
"baseline": 100,
|
||||
"slack": 5
|
||||
"limit": 105
|
||||
},
|
||||
"UP045": {
|
||||
"baseline": 18417,
|
||||
"slack": 100
|
||||
"limit": 18517
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,19 +1,16 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Non-gating ratchet guard: budget baselines and ceilings may only fall, never rise.
|
||||
"""Non-gating ratchet guard: budget limits may only fall, never rise.
|
||||
|
||||
Every `*-budget.json` file (ruff-strict, type-discipline, basedpyright-code) is a
|
||||
one-way ratchet: each rule's ceiling is `baseline + slack`, and both the recorded
|
||||
`baseline` (the live violation count) and that ceiling are meant to be driven DOWN
|
||||
over time. This check compares every budget file against its own content at the
|
||||
merge-base with the target branch and fails (exits 1, red) if:
|
||||
one-way ratchet: each rule's ceiling is its `limit`, and that limit is meant to be
|
||||
driven DOWN over time. This check compares every budget file against its own
|
||||
content at the merge-base with the target branch and fails (exits 1, red) if:
|
||||
|
||||
* a rule's ceiling (`baseline + slack`) went up,
|
||||
* a rule's `baseline` went up, even if `slack` was lowered to keep the ceiling
|
||||
flat (a higher baseline bakes in more accepted debt and must be acknowledged),
|
||||
* a rule's `limit` went up,
|
||||
* a rule was dropped from a budget (its ceiling effectively became infinite), or
|
||||
* an entire budget file was deleted.
|
||||
|
||||
New rules and lowered/equal baselines and ceilings are fine.
|
||||
New rules and lowered/equal limits are fine.
|
||||
|
||||
This is deliberately NOT a gating check. It should turn the run red so that a
|
||||
loosening is impossible to miss in review, but it must stay OUT of the
|
||||
|
|
@ -89,19 +86,21 @@ def _load_base(rel: str, ref: str) -> dict | None:
|
|||
return json.loads(proc.stdout)
|
||||
|
||||
|
||||
def _baselines(budget: dict) -> dict[str, int]:
|
||||
"""Map each rule to its recorded baseline; skip malformed specs."""
|
||||
return {
|
||||
rule: int(spec.get("baseline", 0))
|
||||
for rule, spec in budget.items()
|
||||
if isinstance(spec, dict)
|
||||
}
|
||||
def _ceiling(spec: dict) -> int:
|
||||
"""A rule's ceiling: its `limit`, or legacy `baseline + slack`.
|
||||
|
||||
The base side of the diff can predate the `limit` migration, so a spec is read
|
||||
under either schema and the two are compared on the same footing.
|
||||
"""
|
||||
if "limit" in spec:
|
||||
return int(spec["limit"])
|
||||
return int(spec.get("baseline", 0)) + int(spec.get("slack", 0))
|
||||
|
||||
|
||||
def _caps(budget: dict) -> dict[str, int]:
|
||||
"""Map each rule to its ceiling (baseline + slack); skip malformed specs."""
|
||||
def _limits(budget: dict) -> dict[str, int]:
|
||||
"""Map each rule to its ceiling; skip malformed specs."""
|
||||
return {
|
||||
rule: int(spec.get("baseline", 0)) + int(spec.get("slack", 0))
|
||||
rule: _ceiling(spec)
|
||||
for rule, spec in budget.items()
|
||||
if isinstance(spec, dict)
|
||||
}
|
||||
|
|
@ -109,54 +108,32 @@ def _caps(budget: dict) -> dict[str, int]:
|
|||
|
||||
def _regression_detail(
|
||||
rule: str,
|
||||
base_caps: dict[str, int],
|
||||
head_caps: dict[str, int],
|
||||
base_baselines: dict[str, int],
|
||||
head_baselines: dict[str, int],
|
||||
base_limits: dict[str, int],
|
||||
head_limits: dict[str, int],
|
||||
) -> str | None:
|
||||
"""Why `rule` regressed vs base, or None when it held flat or fell.
|
||||
|
||||
A dropped rule is terminal; otherwise a raised ceiling and a raised baseline are
|
||||
independent loosenings (the latter catches a baseline bump masked by a slack cut),
|
||||
so both reasons are reported when both apply.
|
||||
A dropped rule is terminal; otherwise the only loosening left is a raised limit.
|
||||
"""
|
||||
base_cap = base_caps[rule]
|
||||
if rule not in head_caps:
|
||||
return f"rule dropped (ceiling {base_cap} -> removed)"
|
||||
reasons = tuple(
|
||||
message
|
||||
for raised, message in (
|
||||
(
|
||||
head_caps[rule] > base_cap,
|
||||
f"ceiling raised {base_cap} -> {head_caps[rule]}",
|
||||
),
|
||||
(
|
||||
head_baselines[rule] > base_baselines[rule],
|
||||
f"baseline raised {base_baselines[rule]} -> {head_baselines[rule]}",
|
||||
),
|
||||
)
|
||||
if raised
|
||||
)
|
||||
return "; ".join(reasons) or None
|
||||
base_limit = base_limits[rule]
|
||||
if rule not in head_limits:
|
||||
return f"rule dropped (limit {base_limit} -> removed)"
|
||||
if head_limits[rule] > base_limit:
|
||||
return f"limit raised {base_limit} -> {head_limits[rule]}"
|
||||
return None
|
||||
|
||||
|
||||
def regressions_for(rel: str, base: dict | None, head: dict | None) -> list[Regression]:
|
||||
if base is None:
|
||||
return [] # new budget file: nothing to ratchet against yet
|
||||
if head is None:
|
||||
return [Regression(rel, "*", "budget file was deleted (every ceiling removed)")]
|
||||
return [Regression(rel, "*", "budget file was deleted (every limit removed)")]
|
||||
|
||||
base_caps, head_caps = _caps(base), _caps(head)
|
||||
base_baselines, head_baselines = _baselines(base), _baselines(head)
|
||||
base_limits, head_limits = _limits(base), _limits(head)
|
||||
return [
|
||||
Regression(rel, rule, detail)
|
||||
for rule in sorted(base_caps)
|
||||
if (
|
||||
detail := _regression_detail(
|
||||
rule, base_caps, head_caps, base_baselines, head_baselines
|
||||
)
|
||||
)
|
||||
is not None
|
||||
for rule in sorted(base_limits)
|
||||
if (detail := _regression_detail(rule, base_limits, head_limits)) is not None
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -191,7 +168,7 @@ def main() -> int:
|
|||
|
||||
if regressions:
|
||||
print(
|
||||
f"FAIL: budget baseline(s)/ceiling(s) loosened vs base {args.base} (merge-base {ref[:12]}):"
|
||||
f"FAIL: budget limit(s) loosened vs base {args.base} (merge-base {ref[:12]}):"
|
||||
)
|
||||
for reg in regressions:
|
||||
print(f" {reg.budget} {reg.rule}: {reg.detail}")
|
||||
|
|
@ -203,7 +180,7 @@ def main() -> int:
|
|||
return 1
|
||||
|
||||
suffix = f" ({', '.join(checked)})" if checked else ""
|
||||
print(f"OK: no budget ceiling increased vs base {args.base}{suffix}")
|
||||
print(f"OK: no budget limit increased vs base {args.base}{suffix}")
|
||||
return 0
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -36,6 +36,22 @@ spec_files=$(staged_match '^(litellm/(proxy|types)/.*|ui/litellm-dashboard/(scri
|
|||
ui_prettier_files=$(staged_match '^ui/litellm-dashboard/.*\.(js|jsx|ts|tsx|mjs|cjs|json|css|scss|md|mdx|yml|yaml|html)$')
|
||||
ui_eslint_files=$(staged_match '^ui/litellm-dashboard/.*\.(js|jsx|ts|tsx|mjs|cjs)$')
|
||||
|
||||
# CI lints the committed tree, so this script predicts CI for what you have STAGED
|
||||
# (every trigger above reads `git diff --cached`). The tools it runs, though, read
|
||||
# the working tree, so unstaged edits to tracked files and untracked files fold
|
||||
# into the result and a green/red here won't match a commit of just the staged
|
||||
# changes. There's no safe way to lint the index in place, so surface the gap
|
||||
# instead of hiding it: stage everything you intend to commit before trusting a
|
||||
# pass. This only warns; it never blocks or touches your changes.
|
||||
unstaged=$(git diff --name-only)
|
||||
untracked=$(git ls-files --others --exclude-standard)
|
||||
if [ -n "$unstaged" ] || [ -n "$untracked" ]; then
|
||||
echo "pre-commit: NOTE - unstaged/untracked changes are included in these checks but" >&2
|
||||
echo " won't be in a commit of only your staged changes, so this result may differ from" >&2
|
||||
echo " CI. Stage everything you intend to commit (git add) for an accurate prediction:" >&2
|
||||
printf '%s\n' "$unstaged" "$untracked" | sed '/^$/d' | sed 's/^/ /' >&2
|
||||
fi
|
||||
|
||||
lint_dashboard() {
|
||||
(
|
||||
rc=0
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Total-count gate for the strict ruff rules in ruff-strict.toml.
|
||||
|
||||
Each rule has a hard ceiling (baseline + slack) in ruff-strict-budget.json. The
|
||||
gate counts each rule across the whole tree and fails when a rule is both over
|
||||
its ceiling and higher than the base it merges into, so a change is blamed for
|
||||
the violations it adds, never for drift that already exists in the base.
|
||||
Each rule has a hard ``limit`` in ruff-strict-budget.json. The gate counts each
|
||||
rule across the whole tree and fails when a rule is both over its limit and
|
||||
higher than the base it merges into, so a change is blamed for the violations it
|
||||
adds, never for drift that already exists in the base. ``--update`` ratchets each
|
||||
rule's limit down by the number of violations this branch fixed relative to its
|
||||
branch point (the merge-base).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
|
@ -90,7 +92,7 @@ def base_counts(ref: str) -> dict:
|
|||
def evaluate(head: dict, base: dict, budget: dict) -> list:
|
||||
breaches = []
|
||||
for rule, spec in budget.items():
|
||||
cap = spec["baseline"] + spec["slack"]
|
||||
cap = spec["limit"]
|
||||
total = head.get(rule, 0)
|
||||
if total > cap and total > base.get(rule, 0):
|
||||
breaches.append(Breach(rule, total, cap, total - base.get(rule, 0)))
|
||||
|
|
@ -128,26 +130,49 @@ def cmd_check(base: str) -> None:
|
|||
_run(["git", "diff", base_point, "--unified=0", "--no-color", "--", TARGET])
|
||||
),
|
||||
)
|
||||
print(f"FAIL: strict-rule totals exceed their ceiling (base {base}):")
|
||||
print(f"FAIL: strict-rule totals exceed their limit (base {base}):")
|
||||
for breach in breaches:
|
||||
print(
|
||||
f" {breach.rule}: total {breach.total} over cap {breach.cap} (this change added {breach.added})"
|
||||
f" {breach.rule}: total {breach.total} over limit {breach.cap} (this change added {breach.added})"
|
||||
)
|
||||
for violation in sorted(v for v in new if v.code == breach.rule):
|
||||
print(f" {violation.file}:{violation.line}")
|
||||
print(
|
||||
"Reduce the new violations or remove an equal number elsewhere; the ceiling is baseline + slack in ruff-strict-budget.json."
|
||||
"Reduce the new violations or remove an equal number elsewhere; the ceiling is the limit in ruff-strict-budget.json."
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def cmd_update() -> None:
|
||||
def ratcheted_budget(budget: dict, current: dict, base: dict) -> dict:
|
||||
"""Each rule's limit lowered by the violations `current` fixed vs `base`.
|
||||
|
||||
`base` is the count at the branch point (the commit this branch diverged
|
||||
from). The drop is clamped to what was actually cleared (a rule that grew
|
||||
stays put), so the limit only ever falls.
|
||||
"""
|
||||
return {
|
||||
rule: {
|
||||
"limit": max(0, spec["limit"] - max(0, base.get(rule, 0) - current.get(rule, 0)))
|
||||
}
|
||||
for rule, spec in sorted(budget.items())
|
||||
}
|
||||
|
||||
|
||||
def cmd_update(base_ref: str = DEFAULT_BASE) -> None:
|
||||
"""Ratchet each rule's limit down by the violations this branch fixed.
|
||||
|
||||
The working-tree count is compared against a ruff pass over a detached
|
||||
worktree at the branch point (the merge-base with `base_ref`), so a branch's
|
||||
fixes tighten its own ceilings by exactly what they cleared since it diverged.
|
||||
"""
|
||||
budget = json.loads(BUDGET_PATH.read_text())
|
||||
head = count_by_rule(head_violations())
|
||||
for rule in budget:
|
||||
budget[rule]["baseline"] = head.get(rule, 0)
|
||||
BUDGET_PATH.write_text(json.dumps(budget, indent=2, sort_keys=True) + "\n")
|
||||
print("Re-captured per-rule baselines from the current tree")
|
||||
base_point = _run(["git", "merge-base", base_ref, "HEAD"]).strip() or base_ref
|
||||
updated = ratcheted_budget(
|
||||
budget, count_by_rule(head_violations()), base_counts(base_point)
|
||||
)
|
||||
BUDGET_PATH.write_text(json.dumps(updated, indent=2, sort_keys=True) + "\n")
|
||||
cleared = sum(budget[rule]["limit"] - updated[rule]["limit"] for rule in updated)
|
||||
print(f"Ratcheted strict-rule limits down by {cleared} violations this branch fixed")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
|
|
@ -155,7 +180,7 @@ def main() -> None:
|
|||
parser.add_argument("--base", default=DEFAULT_BASE)
|
||||
parser.add_argument("--update", action="store_true")
|
||||
args = parser.parse_args()
|
||||
cmd_update() if args.update else cmd_check(args.base)
|
||||
cmd_update(args.base) if args.update else cmd_check(args.base)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -3,20 +3,22 @@
|
|||
|
||||
basedpyright's ``--outputjson`` is reduced to a count of errors per *rule*
|
||||
(``reportAny``, ``reportArgumentType``, ...) and checked against a committed
|
||||
budget of the form ``{rule: {baseline, slack}}``, the same shape as
|
||||
budget of the form ``{rule: {limit}}``, the same shape as
|
||||
``ruff-strict-budget.json``. A rule fails only when its codebase-wide total is
|
||||
both over its ceiling (``baseline + slack``) *and* higher than the count on the
|
||||
base it merges into, so a change is blamed for the errors it adds, never for
|
||||
drift that already sits in the base. That ``> base`` guard is what stops an
|
||||
unrelated PR from inheriting a red once two PRs each land near the ceiling and
|
||||
their sum crosses it: the bystander's count equals its base, so it is spared,
|
||||
while any PR that actually grows the rule past the cap still fails.
|
||||
both over its ``limit`` *and* higher than the count on the base it merges into,
|
||||
so a change is blamed for the errors it adds, never for drift that already sits
|
||||
in the base. That ``> base`` guard is what stops an unrelated PR from inheriting
|
||||
a red once two PRs each land near the limit and their sum crosses it: the
|
||||
bystander's count equals its base, so it is spared, while any PR that actually
|
||||
grows the rule past its limit still fails.
|
||||
|
||||
Head counts are read from stdin (the caller runs basedpyright once and pipes
|
||||
``--outputjson`` in); the base count is a second basedpyright pass over a
|
||||
detached worktree at the merge-base, run under the same environment so import
|
||||
resolution matches. ``--update`` re-captures the absolute per-rule baselines for
|
||||
the ratchet, preserving each rule's slack.
|
||||
resolution matches. ``--update`` ratchets each rule's ``limit`` down by the
|
||||
number of errors this branch fixed relative to its branch point (the merge-base),
|
||||
so the headroom you were granted shrinks by exactly what you cleared and never
|
||||
grows.
|
||||
|
||||
``--outputjson`` is used rather than text diagnostics because the latter wrap
|
||||
across lines, leaving the ``(reportRule)`` on a continuation line away from the
|
||||
|
|
@ -44,10 +46,10 @@ DEFAULT_BASE = "origin/litellm_internal_staging"
|
|||
# Bucket for a basedpyright diagnostic with no `rule`. Counted so it's gated.
|
||||
UNCODED = "<uncoded>"
|
||||
|
||||
# Ceiling for a rule that shows up at HEAD but isn't in the budget at all -- a
|
||||
# brand-new error category (new construct, or a tool/version change). baseline
|
||||
# is treated as 0, so the rule fails once it clears this much slack.
|
||||
DEFAULT_SLACK = 10
|
||||
# Limit for a rule that shows up at HEAD but isn't in the budget at all -- a
|
||||
# brand-new error category (new construct, or a tool/version change). The rule
|
||||
# fails once it clears this many errors.
|
||||
DEFAULT_LIMIT = 10
|
||||
|
||||
|
||||
class Breach(NamedTuple):
|
||||
|
|
@ -57,13 +59,6 @@ class Breach(NamedTuple):
|
|||
added: int
|
||||
|
||||
|
||||
def _seed_slack(baseline: int) -> int:
|
||||
"""Slack written for a rule first captured into a budget; busy rules get
|
||||
more headroom, mirroring the tiering in ruff-strict-budget.json. Existing
|
||||
rules keep whatever slack their JSON already declares."""
|
||||
return 10 if baseline >= 50 else 3
|
||||
|
||||
|
||||
def _to_relative(raw: str, root: Path) -> str | None:
|
||||
path = Path(raw)
|
||||
absolute = path if path.is_absolute() else root / path
|
||||
|
|
@ -142,7 +137,7 @@ def evaluate(
|
|||
breaches = []
|
||||
for code, total in head.items():
|
||||
spec = budget.get(code)
|
||||
cap = spec["baseline"] + spec["slack"] if spec else DEFAULT_SLACK
|
||||
cap = spec["limit"] if spec else DEFAULT_LIMIT
|
||||
prior = base.get(code, 0)
|
||||
if total > cap and total > prior:
|
||||
breaches.append(Breach(code, total, cap, total - prior))
|
||||
|
|
@ -155,24 +150,47 @@ def is_vacuous_run(
|
|||
"""True when nothing was parsed but the budget expects errors -- the
|
||||
signature of a type checker that crashed or produced no output. The CI pipe
|
||||
swallows the tool's exit code (`tool || true`), so without this guard an
|
||||
empty run would clear every ceiling and pass silently."""
|
||||
return not counts and any(spec["baseline"] for spec in budget.values())
|
||||
empty run would clear every limit and pass silently."""
|
||||
return not counts and any(spec["limit"] for spec in budget.values())
|
||||
|
||||
|
||||
def cmd_update(counts: Mapping[str, int]) -> None:
|
||||
existing = json.loads(BUDGET_PATH.read_text()) if BUDGET_PATH.exists() else {}
|
||||
budget = {
|
||||
def ratcheted_budget(
|
||||
budget: Mapping[str, Mapping[str, int]],
|
||||
current: Mapping[str, int],
|
||||
base: Mapping[str, int],
|
||||
) -> dict[str, dict[str, int]]:
|
||||
"""Each rule's limit lowered by the errors `current` fixed vs `base`.
|
||||
|
||||
`base` is the count at the branch point (the commit this branch diverged
|
||||
from). The drop is clamped to what was actually cleared (a rule that grew
|
||||
stays put), so the limit only ever falls. Rules absent from the budget are
|
||||
dropped: a genuinely new error category is added to the JSON deliberately,
|
||||
not on update.
|
||||
"""
|
||||
return {
|
||||
code: {
|
||||
"baseline": count,
|
||||
"slack": (
|
||||
existing[code]["slack"] if code in existing else _seed_slack(count)
|
||||
),
|
||||
"limit": max(0, spec["limit"] - max(0, base.get(code, 0) - current.get(code, 0)))
|
||||
}
|
||||
for code, count in sorted(counts.items())
|
||||
for code, spec in sorted(budget.items())
|
||||
}
|
||||
BUDGET_PATH.write_text(json.dumps(budget, indent=2, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def cmd_update(current: Mapping[str, int], base_ref: str = DEFAULT_BASE) -> None:
|
||||
"""Ratchet each rule's limit down by the errors this branch fixed.
|
||||
|
||||
`current` is the working-tree count (piped in); the reference count comes
|
||||
from a second basedpyright pass over a detached worktree at the branch point
|
||||
(the merge-base with `base_ref`), so a branch's fixes tighten its own ceilings
|
||||
by exactly what they cleared since it diverged, and limits never rise.
|
||||
"""
|
||||
budget = json.loads(BUDGET_PATH.read_text()) if BUDGET_PATH.exists() else {}
|
||||
base_point = _run(["git", "merge-base", base_ref, "HEAD"]).strip() or base_ref
|
||||
updated = ratcheted_budget(budget, current, base_counts(base_point))
|
||||
BUDGET_PATH.write_text(json.dumps(updated, indent=2, sort_keys=True) + "\n")
|
||||
cleared = sum(budget[code]["limit"] - updated[code]["limit"] for code in updated)
|
||||
print(
|
||||
f"Re-captured basedpyright per-rule budget: {len(budget)} rules, {sum(counts.values())} errors total"
|
||||
f"Ratcheted basedpyright limits down by {cleared} errors this branch fixed "
|
||||
f"across {len(updated)} rules"
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -180,10 +198,10 @@ def cmd_check(base_ref: str) -> None:
|
|||
budget = json.loads(BUDGET_PATH.read_text())
|
||||
head = count_basedpyright(sys.stdin.read())
|
||||
if is_vacuous_run(head, budget):
|
||||
expected = sum(spec["baseline"] for spec in budget.values())
|
||||
expected = sum(spec["limit"] for spec in budget.values())
|
||||
print(
|
||||
f"FAIL: basedpyright produced no errors, but {BUDGET_PATH.name} expects "
|
||||
f"~{expected}. The type checker almost certainly crashed or emitted "
|
||||
f"FAIL: basedpyright produced no errors, but {BUDGET_PATH.name} allows "
|
||||
f"up to ~{expected}. The type checker almost certainly crashed or emitted "
|
||||
f"nothing; refusing to certify a vacuous run."
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
|
@ -199,17 +217,17 @@ def cmd_check(base_ref: str) -> None:
|
|||
breaches = evaluate(head, base, budget)
|
||||
if not breaches:
|
||||
print(
|
||||
f"OK: every rule is within its basedpyright ceiling or no higher than base ({sum(head.values())} errors total)"
|
||||
f"OK: every rule is within its basedpyright limit or no higher than base ({sum(head.values())} errors total)"
|
||||
)
|
||||
return
|
||||
print("FAIL: basedpyright errors exceed the per-rule ceiling:")
|
||||
print("FAIL: basedpyright errors exceed the per-rule limit:")
|
||||
for breach in breaches:
|
||||
print(
|
||||
f" {breach.code}: total {breach.total} over cap {breach.cap} (this change added {breach.added})"
|
||||
f" {breach.code}: total {breach.total} over limit {breach.cap} (this change added {breach.added})"
|
||||
)
|
||||
print(
|
||||
"Reduce the new errors or remove an equal number elsewhere; the ceiling is "
|
||||
"baseline + slack in basedpyright-code-budget.json."
|
||||
"the limit in basedpyright-code-budget.json."
|
||||
)
|
||||
summary = "; ".join(f"{b.code} {b.total}/{b.cap} (+{b.added})" for b in breaches)
|
||||
print(f"BREACHED RULES: {summary}")
|
||||
|
|
@ -222,7 +240,7 @@ def main() -> None:
|
|||
parser.add_argument("--update", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.update:
|
||||
cmd_update(count_basedpyright(sys.stdin.read()))
|
||||
cmd_update(count_basedpyright(sys.stdin.read()), args.base)
|
||||
else:
|
||||
cmd_check(args.base)
|
||||
|
||||
|
|
|
|||
|
|
@ -2,18 +2,19 @@
|
|||
"""Total-count gate for the LIT* rules in scripts/check_type_discipline.py.
|
||||
|
||||
Sibling of scripts/ruff_strict_gate.py. Each rule listed in
|
||||
type-discipline-budget.json has a hard ceiling (baseline + slack). The gate counts
|
||||
each rule across the whole `litellm` tree and fails when a rule is both over its
|
||||
ceiling and higher than the base it merges into, so a change is blamed for the
|
||||
violations it adds, never for drift that already exists in the base.
|
||||
type-discipline-budget.json has a hard ``limit``. The gate counts each rule
|
||||
across the whole `litellm` tree and fails when a rule is both over its limit and
|
||||
higher than the base it merges into, so a change is blamed for the violations it
|
||||
adds, never for drift that already exists in the base.
|
||||
|
||||
Rules not present in the budget are ignored, but today every rule the checker
|
||||
emits is gated: LIT001 (mutable collection in any annotation), LIT002
|
||||
(mutable-collection construction), LIT003/LIT004 (noqa / ignore without codes or
|
||||
reason), LIT006 (cast), and LIT008 (`**kwargs`) carry slack-buffered ceilings to
|
||||
ratchet down; LIT005 (`*-ok` suppression without a reason) is frozen at slack 0
|
||||
so any net-new reasonless suppression trips the gate; and LIT007 (TypeGuard/TypeIs)
|
||||
is a hard zero. Re-baseline with `--update` to ratchet a ceiling down.
|
||||
reason), LIT006 (cast), and LIT008 (`**kwargs`) carry limits above their current
|
||||
count to ratchet down; LIT005 (`*-ok` suppression without a reason) is frozen at
|
||||
limit 0 so any net-new reasonless suppression trips the gate; and LIT007
|
||||
(TypeGuard/TypeIs) is a hard zero. ``--update`` ratchets a limit down by the
|
||||
violations this branch fixed relative to its branch point (the merge-base).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
|
@ -104,21 +105,21 @@ def base_counts(ref: str) -> dict:
|
|||
|
||||
|
||||
def over_ceiling(head: dict, budget: dict) -> frozenset:
|
||||
"""Rules whose head count already exceeds baseline + slack.
|
||||
"""Rules whose head count already exceeds their limit.
|
||||
|
||||
A rule can only breach when it is over its ceiling, so when none are the base
|
||||
A rule can only breach when it is over its limit, so when none are the base
|
||||
comparison cannot change the verdict and the base worktree scan can be skipped.
|
||||
"""
|
||||
return frozenset(
|
||||
rule for rule, spec in budget.items()
|
||||
if head.get(rule, 0) > spec["baseline"] + spec["slack"]
|
||||
if head.get(rule, 0) > spec["limit"]
|
||||
)
|
||||
|
||||
|
||||
def evaluate(head: dict, base: dict, budget: dict) -> list:
|
||||
breaches = []
|
||||
for rule, spec in budget.items():
|
||||
cap = spec["baseline"] + spec["slack"]
|
||||
cap = spec["limit"]
|
||||
total = head.get(rule, 0)
|
||||
if total > cap and total > base.get(rule, 0):
|
||||
breaches.append(Breach(rule, total, cap, total - base.get(rule, 0)))
|
||||
|
|
@ -160,10 +161,10 @@ def cmd_check(base: str) -> None:
|
|||
_run(["git", "diff", base_point, "--unified=0", "--no-color", "--", TARGET])
|
||||
),
|
||||
)
|
||||
print(f"FAIL: LIT-rule totals exceed their ceiling (base {base}):")
|
||||
print(f"FAIL: LIT-rule totals exceed their limit (base {base}):")
|
||||
for breach in breaches:
|
||||
print(
|
||||
f" {breach.rule}: total {breach.total} over cap {breach.cap} (this change added {breach.added})"
|
||||
f" {breach.rule}: total {breach.total} over limit {breach.cap} (this change added {breach.added})"
|
||||
)
|
||||
for violation in sorted(v for v in new if v.code == breach.rule):
|
||||
print(f" {violation.file}:{violation.line}")
|
||||
|
|
@ -171,19 +172,42 @@ def cmd_check(base: str) -> None:
|
|||
"Remove the new violations, give each a reason (`# noqa: XXX # <reason>`, "
|
||||
"`# pyright: ignore[rule] # <reason>`, `# mutable-ok: <reason>`, "
|
||||
"`# cast-ok: <reason>`, `# guard-ok: <reason>`, `# kwargs-ok: <reason>`), or "
|
||||
"remove an equal number elsewhere; the ceiling is baseline + slack in "
|
||||
"remove an equal number elsewhere; the ceiling is the limit in "
|
||||
"type-discipline-budget.json."
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def cmd_update() -> None:
|
||||
def ratcheted_budget(budget: dict, current: dict, base: dict) -> dict:
|
||||
"""Each rule's limit lowered by the violations `current` fixed vs `base`.
|
||||
|
||||
`base` is the count at the branch point (the commit this branch diverged
|
||||
from). The drop is clamped to what was actually cleared (a rule that grew
|
||||
stays put), so the limit only ever falls.
|
||||
"""
|
||||
return {
|
||||
rule: {
|
||||
"limit": max(0, spec["limit"] - max(0, base.get(rule, 0) - current.get(rule, 0)))
|
||||
}
|
||||
for rule, spec in sorted(budget.items())
|
||||
}
|
||||
|
||||
|
||||
def cmd_update(base_ref: str = DEFAULT_BASE) -> None:
|
||||
"""Ratchet each rule's limit down by the violations this branch fixed.
|
||||
|
||||
The working-tree count is compared against a checker pass over a detached
|
||||
worktree at the branch point (the merge-base with `base_ref`), so a branch's
|
||||
fixes tighten its own ceilings by exactly what they cleared since it diverged.
|
||||
"""
|
||||
budget = json.loads(BUDGET_PATH.read_text())
|
||||
head = count_by_rule(head_violations())
|
||||
for rule in budget:
|
||||
budget[rule]["baseline"] = head.get(rule, 0)
|
||||
BUDGET_PATH.write_text(json.dumps(budget, indent=2, sort_keys=True) + "\n")
|
||||
print("Re-captured per-rule baselines from the current tree")
|
||||
base_point = _run(["git", "merge-base", base_ref, "HEAD"]).strip() or base_ref
|
||||
updated = ratcheted_budget(
|
||||
budget, count_by_rule(head_violations()), base_counts(base_point)
|
||||
)
|
||||
BUDGET_PATH.write_text(json.dumps(updated, indent=2, sort_keys=True) + "\n")
|
||||
cleared = sum(budget[rule]["limit"] - updated[rule]["limit"] for rule in updated)
|
||||
print(f"Ratcheted LIT-rule limits down by {cleared} violations this branch fixed")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
|
|
@ -191,7 +215,7 @@ def main() -> None:
|
|||
parser.add_argument("--base", default=DEFAULT_BASE)
|
||||
parser.add_argument("--update", action="store_true")
|
||||
args = parser.parse_args()
|
||||
cmd_update() if args.update else cmd_check(args.base)
|
||||
cmd_update(args.base) if args.update else cmd_check(args.base)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,9 +1,8 @@
|
|||
"""Tests for scripts/budget_ratchet_check.py.
|
||||
|
||||
The guard's contract is "baselines and ceilings may only fall": a raised ceiling, a
|
||||
raised baseline (even when slack is cut to keep the ceiling flat), a dropped rule, or
|
||||
a deleted file is a regression, while a lowered/equal baseline and ceiling, a brand-new
|
||||
rule, or a brand-new budget file is fine. Each branch is pinned here.
|
||||
The guard's contract is "limits may only fall": a raised limit, a dropped rule, or
|
||||
a deleted file is a regression, while a lowered/equal limit, a brand-new rule, or a
|
||||
brand-new budget file is fine. Each branch is pinned here.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
|
|
@ -19,69 +18,64 @@ ratchet = importlib.util.module_from_spec(_spec)
|
|||
_spec.loader.exec_module(ratchet)
|
||||
|
||||
|
||||
def _spec_of(baseline, slack):
|
||||
return {"baseline": baseline, "slack": slack}
|
||||
def _spec_of(limit):
|
||||
return {"limit": limit}
|
||||
|
||||
|
||||
def test_caps_sum_baseline_and_slack_and_skip_malformed():
|
||||
caps = ratchet._caps({"LIT006": _spec_of(1013, 10), "junk": 5})
|
||||
assert caps == {"LIT006": 1023} # malformed (non-dict) spec ignored
|
||||
def test_limits_read_the_limit_and_skip_malformed():
|
||||
limits = ratchet._limits({"LIT006": _spec_of(1023), "junk": 5})
|
||||
assert limits == {"LIT006": 1023} # malformed (non-dict) spec ignored
|
||||
|
||||
|
||||
def test_raised_ceiling_is_a_regression():
|
||||
base = {"LIT006": _spec_of(1013, 10)}
|
||||
head = {"LIT006": _spec_of(1013, 11)} # cap 1023 -> 1024
|
||||
def test_limits_fall_back_to_legacy_baseline_plus_slack():
|
||||
# The base side of a diff can predate the `limit` migration; its ceiling is
|
||||
# baseline + slack, read on the same footing as a new-schema `limit`.
|
||||
assert ratchet._limits({"LIT006": {"baseline": 1013, "slack": 10}}) == {"LIT006": 1023}
|
||||
|
||||
|
||||
def test_migration_from_legacy_schema_to_equal_limit_is_clean():
|
||||
# baseline+slack (1023) -> limit 1023 is the same ceiling, so no regression.
|
||||
base = {"LIT006": {"baseline": 1013, "slack": 10}}
|
||||
assert ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1023)}) == []
|
||||
# ...and a genuine raise across the migration is still caught.
|
||||
regs = ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1024)})
|
||||
assert [r.rule for r in regs] == ["LIT006"] and "1023 -> 1024" in regs[0].detail
|
||||
|
||||
|
||||
def test_raised_limit_is_a_regression():
|
||||
base = {"LIT006": _spec_of(1023)}
|
||||
head = {"LIT006": _spec_of(1024)}
|
||||
regs = ratchet.regressions_for("b.json", base, head)
|
||||
assert [r.rule for r in regs] == ["LIT006"]
|
||||
assert "1023 -> 1024" in regs[0].detail
|
||||
|
||||
|
||||
def test_lowered_or_equal_ceiling_is_clean():
|
||||
base = {"LIT006": _spec_of(1013, 10)}
|
||||
# baseline drops, slack flat -> ceiling falls
|
||||
assert ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1000, 10)}) == []
|
||||
def test_lowered_or_equal_limit_is_clean():
|
||||
base = {"LIT006": _spec_of(1023)}
|
||||
# limit drops
|
||||
assert ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1000)}) == []
|
||||
# nothing changes
|
||||
assert ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1013, 10)}) == []
|
||||
# slack cut while baseline holds -> ceiling falls, baseline flat
|
||||
assert ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1013, 0)}) == []
|
||||
|
||||
|
||||
def test_raised_baseline_is_a_regression_even_when_ceiling_held_flat():
|
||||
# baseline 1013 -> 1023 with slack cut 10 -> 0 keeps the ceiling at 1023, but a
|
||||
# higher baseline bakes in more accepted debt and must still surface as a regression
|
||||
base = {"LIT006": _spec_of(1013, 10)}
|
||||
regs = ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1023, 0)})
|
||||
assert [r.rule for r in regs] == ["LIT006"]
|
||||
assert "baseline raised 1013 -> 1023" in regs[0].detail
|
||||
assert "ceiling raised" not in regs[0].detail
|
||||
|
||||
|
||||
def test_raised_baseline_and_ceiling_report_both_reasons():
|
||||
base = {"LIT006": _spec_of(1013, 10)}
|
||||
regs = ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1100, 10)})
|
||||
assert [r.rule for r in regs] == ["LIT006"]
|
||||
assert "ceiling raised 1023 -> 1110" in regs[0].detail
|
||||
assert "baseline raised 1013 -> 1100" in regs[0].detail
|
||||
assert ratchet.regressions_for("b.json", base, {"LIT006": _spec_of(1023)}) == []
|
||||
|
||||
|
||||
def test_dropped_rule_is_a_regression():
|
||||
regs = ratchet.regressions_for("b.json", {"LIT007": _spec_of(0, 0)}, {})
|
||||
regs = ratchet.regressions_for("b.json", {"LIT007": _spec_of(0)}, {})
|
||||
assert [r.rule for r in regs] == ["LIT007"]
|
||||
assert "dropped" in regs[0].detail
|
||||
|
||||
|
||||
def test_new_rule_in_head_is_clean():
|
||||
assert ratchet.regressions_for("b.json", {}, {"new-rule": _spec_of(5, 0)}) == []
|
||||
assert ratchet.regressions_for("b.json", {}, {"new-rule": _spec_of(5)}) == []
|
||||
|
||||
|
||||
def test_deleted_budget_file_is_a_regression():
|
||||
regs = ratchet.regressions_for("b.json", {"LIT006": _spec_of(1, 0)}, None)
|
||||
regs = ratchet.regressions_for("b.json", {"LIT006": _spec_of(1)}, None)
|
||||
assert [r.rule for r in regs] == ["*"]
|
||||
assert "deleted" in regs[0].detail
|
||||
|
||||
|
||||
def test_new_budget_file_has_nothing_to_ratchet():
|
||||
assert ratchet.regressions_for("b.json", None, {"LIT006": _spec_of(1, 0)}) == []
|
||||
assert ratchet.regressions_for("b.json", None, {"LIT006": _spec_of(1)}) == []
|
||||
|
||||
|
||||
def test_default_budgets_watch_every_budget_file_in_the_repo():
|
||||
|
|
|
|||
|
|
@ -11,16 +11,16 @@ _spec.loader.exec_module(gate)
|
|||
Violation = gate.Violation
|
||||
|
||||
|
||||
def rule(name, baseline, slack):
|
||||
return {name: {"baseline": baseline, "slack": slack}}
|
||||
def rule(name, limit):
|
||||
return {name: {"limit": limit}}
|
||||
|
||||
|
||||
def test_under_ceiling_passes():
|
||||
assert gate.evaluate({"ANN001": 100}, {"ANN001": 100}, rule("ANN001", 90, 20)) == []
|
||||
assert gate.evaluate({"ANN001": 100}, {"ANN001": 100}, rule("ANN001", 110)) == []
|
||||
|
||||
|
||||
def test_ceiling_is_baseline_plus_slack_boundary():
|
||||
budget = rule("ANN001", 90, 20) # cap 110
|
||||
def test_ceiling_is_the_limit_boundary():
|
||||
budget = rule("ANN001", 110)
|
||||
at = gate.evaluate({"ANN001": 110}, {"ANN001": 90}, budget)
|
||||
over = gate.evaluate({"ANN001": 111}, {"ANN001": 90}, budget)
|
||||
assert at == []
|
||||
|
|
@ -30,23 +30,23 @@ def test_ceiling_is_baseline_plus_slack_boundary():
|
|||
|
||||
|
||||
def test_over_ceiling_and_change_added_fails():
|
||||
breaches = gate.evaluate({"C901": 11}, {"C901": 9}, rule("C901", 10, 0))
|
||||
breaches = gate.evaluate({"C901": 11}, {"C901": 9}, rule("C901", 10))
|
||||
assert [b.rule for b in breaches] == ["C901"]
|
||||
assert breaches[0].added == 2
|
||||
|
||||
|
||||
def test_base_already_over_ceiling_change_added_nothing_is_not_blamed():
|
||||
# drift safety: base is over cap, this change leaves the count where it is
|
||||
assert gate.evaluate({"C901": 15}, {"C901": 15}, rule("C901", 10, 0)) == []
|
||||
# drift safety: base is over limit, this change leaves the count where it is
|
||||
assert gate.evaluate({"C901": 15}, {"C901": 15}, rule("C901", 10)) == []
|
||||
|
||||
|
||||
def test_change_that_reduces_an_over_ceiling_rule_is_not_blamed():
|
||||
# still over cap, but moving the right direction
|
||||
assert gate.evaluate({"C901": 14}, {"C901": 16}, rule("C901", 10, 0)) == []
|
||||
# still over limit, but moving the right direction
|
||||
assert gate.evaluate({"C901": 14}, {"C901": 16}, rule("C901", 10)) == []
|
||||
|
||||
|
||||
def test_rules_are_independent():
|
||||
budget = {**rule("ANN001", 100, 50), **rule("C901", 10, 0)}
|
||||
budget = {**rule("ANN001", 150), **rule("C901", 10)}
|
||||
breaches = gate.evaluate(
|
||||
{"ANN001": 130, "C901": 11}, {"ANN001": 100, "C901": 10}, budget
|
||||
)
|
||||
|
|
@ -54,7 +54,19 @@ def test_rules_are_independent():
|
|||
|
||||
|
||||
def test_missing_rule_counts_as_zero():
|
||||
assert gate.evaluate({}, {}, rule("C901", 0, 0)) == []
|
||||
assert gate.evaluate({}, {}, rule("C901", 0)) == []
|
||||
|
||||
|
||||
def test_update_ratchets_limit_down_by_what_the_branch_fixed_never_up():
|
||||
budget = {**rule("ANN001", 150), **rule("C901", 10)}
|
||||
# ANN001 fixed 20 (100 -> 80) so its limit falls 150 -> 130; C901 grew, so its
|
||||
# limit holds flat at 10 (a fix must never loosen a ceiling).
|
||||
current = {"ANN001": 80, "C901": 12}
|
||||
base = {"ANN001": 100, "C901": 9}
|
||||
assert gate.ratcheted_budget(budget, current, base) == {
|
||||
"ANN001": {"limit": 130},
|
||||
"C901": {"limit": 10},
|
||||
}
|
||||
|
||||
|
||||
def test_parse_changed_lines_maps_added_lines_per_file():
|
||||
|
|
|
|||
|
|
@ -55,77 +55,98 @@ def test_paths_outside_repo_are_skipped():
|
|||
|
||||
|
||||
def test_at_or_under_ceiling_passes():
|
||||
budget = {"no-any-return": {"baseline": 5, "slack": 0}}
|
||||
budget = {"no-any-return": {"limit": 5}}
|
||||
assert gate.evaluate({"no-any-return": 5}, {}, budget) == []
|
||||
|
||||
|
||||
def test_one_more_error_than_ceiling_fails():
|
||||
budget = {"no-any-return": {"baseline": 5, "slack": 0}}
|
||||
budget = {"no-any-return": {"limit": 5}}
|
||||
assert gate.evaluate({"no-any-return": 6}, {}, budget) == [
|
||||
gate.Breach("no-any-return", 6, 5, 6)
|
||||
]
|
||||
|
||||
|
||||
def test_slack_absorbs_small_increase_then_fails_past_it():
|
||||
budget = {"arg-type": {"baseline": 5, "slack": 5}}
|
||||
def test_limit_absorbs_increase_up_to_it_then_fails_past_it():
|
||||
budget = {"arg-type": {"limit": 10}}
|
||||
assert gate.evaluate({"arg-type": 10}, {}, budget) == []
|
||||
assert gate.evaluate({"arg-type": 11}, {}, budget) == [
|
||||
gate.Breach("arg-type", 11, 10, 11)
|
||||
]
|
||||
|
||||
|
||||
def test_unbudgeted_new_code_uses_default_slack():
|
||||
assert gate.evaluate({"brand-new": gate.DEFAULT_SLACK}, {}, {}) == []
|
||||
assert gate.evaluate({"brand-new": gate.DEFAULT_SLACK + 1}, {}, {}) == [
|
||||
def test_unbudgeted_new_code_uses_default_limit():
|
||||
assert gate.evaluate({"brand-new": gate.DEFAULT_LIMIT}, {}, {}) == []
|
||||
assert gate.evaluate({"brand-new": gate.DEFAULT_LIMIT + 1}, {}, {}) == [
|
||||
gate.Breach(
|
||||
"brand-new",
|
||||
gate.DEFAULT_SLACK + 1,
|
||||
gate.DEFAULT_SLACK,
|
||||
gate.DEFAULT_SLACK + 1,
|
||||
gate.DEFAULT_LIMIT + 1,
|
||||
gate.DEFAULT_LIMIT,
|
||||
gate.DEFAULT_LIMIT + 1,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_drift_already_over_cap_in_base_is_not_blamed_on_a_flat_change():
|
||||
# The bystander case: a rule sits over its ceiling because two earlier PRs
|
||||
# The bystander case: a rule sits over its limit because two earlier PRs
|
||||
# summed past it. A PR that branches off that base and adds nothing must pass
|
||||
# -- total > cap but total == base, so the `> base` guard spares it.
|
||||
budget = {"arg-type": {"baseline": 5, "slack": 5}}
|
||||
# -- total > limit but total == base, so the `> base` guard spares it.
|
||||
budget = {"arg-type": {"limit": 10}}
|
||||
assert gate.evaluate({"arg-type": 12}, {"arg-type": 12}, budget) == []
|
||||
|
||||
|
||||
def test_change_that_grows_an_over_cap_rule_is_blamed_for_only_what_it_added():
|
||||
# Over cap AND above base: blamed, and `added` is the delta vs base, not the
|
||||
# Over limit AND above base: blamed, and `added` is the delta vs base, not the
|
||||
# whole overage, so the message points at this change's contribution.
|
||||
budget = {"arg-type": {"baseline": 5, "slack": 5}}
|
||||
budget = {"arg-type": {"limit": 10}}
|
||||
assert gate.evaluate({"arg-type": 14}, {"arg-type": 12}, budget) == [
|
||||
gate.Breach("arg-type", 14, 10, 2)
|
||||
]
|
||||
|
||||
|
||||
def test_reducing_an_over_cap_rule_below_base_passes():
|
||||
budget = {"arg-type": {"baseline": 5, "slack": 5}}
|
||||
budget = {"arg-type": {"limit": 10}}
|
||||
assert gate.evaluate({"arg-type": 11}, {"arg-type": 12}, budget) == []
|
||||
|
||||
|
||||
def test_no_output_against_a_nonempty_budget_is_a_vacuous_run():
|
||||
# A crashed type checker emits nothing; the gate must not certify it as clean.
|
||||
budget = {"no-untyped-def": {"baseline": 4888, "slack": 10}}
|
||||
budget = {"no-untyped-def": {"limit": 4898}}
|
||||
assert gate.is_vacuous_run({}, budget) is True
|
||||
|
||||
|
||||
def test_genuine_zero_and_empty_budget_are_not_vacuous():
|
||||
assert gate.is_vacuous_run({}, {}) is False
|
||||
assert gate.is_vacuous_run({}, {"no-untyped-def": {"limit": 0}}) is False
|
||||
assert (
|
||||
gate.is_vacuous_run({}, {"no-untyped-def": {"baseline": 0, "slack": 3}})
|
||||
is False
|
||||
)
|
||||
assert (
|
||||
gate.is_vacuous_run({"arg-type": 1}, {"arg-type": {"baseline": 9, "slack": 1}})
|
||||
is False
|
||||
gate.is_vacuous_run({"arg-type": 1}, {"arg-type": {"limit": 10}}) is False
|
||||
)
|
||||
|
||||
|
||||
def test_update_ratchets_a_limit_down_by_what_the_branch_fixed():
|
||||
# A rule that dropped from 40 (branch point) to 30 (current) fixed 10, so its
|
||||
# limit of 100 falls to 90 -- the granted headroom (60) is preserved, not the
|
||||
# raw count.
|
||||
budget = {"reportAny": {"limit": 100}}
|
||||
assert gate.ratcheted_budget(budget, {"reportAny": 30}, {"reportAny": 40}) == {
|
||||
"reportAny": {"limit": 90}
|
||||
}
|
||||
|
||||
|
||||
def test_update_never_raises_a_limit_when_a_rule_grows():
|
||||
# Adding violations must not loosen the ceiling; the limit holds flat.
|
||||
budget = {"reportAny": {"limit": 100}}
|
||||
assert gate.ratcheted_budget(budget, {"reportAny": 55}, {"reportAny": 40}) == {
|
||||
"reportAny": {"limit": 100}
|
||||
}
|
||||
|
||||
|
||||
def test_update_clamps_a_limit_at_zero_never_negative():
|
||||
budget = {"reportAny": {"limit": 5}}
|
||||
assert gate.ratcheted_budget(budget, {"reportAny": 0}, {"reportAny": 40}) == {
|
||||
"reportAny": {"limit": 0}
|
||||
}
|
||||
|
||||
|
||||
def test_malformed_basedpyright_json_exits_loudly_not_as_zero_errors():
|
||||
import pytest
|
||||
|
||||
|
|
|
|||
|
|
@ -14,27 +14,39 @@ gate = importlib.util.module_from_spec(_spec)
|
|||
_spec.loader.exec_module(gate)
|
||||
|
||||
|
||||
def _budget(baseline, slack):
|
||||
return {"LIT006": {"baseline": baseline, "slack": slack}}
|
||||
def _budget(limit):
|
||||
return {"LIT006": {"limit": limit}}
|
||||
|
||||
|
||||
def test_over_ceiling_flags_only_counts_above_baseline_plus_slack():
|
||||
budget = _budget(10, 2) # cap 12
|
||||
assert gate.over_ceiling({"LIT006": 12}, budget) == frozenset() # at cap
|
||||
assert gate.over_ceiling({"LIT006": 13}, budget) == frozenset({"LIT006"}) # over cap
|
||||
def test_over_ceiling_flags_only_counts_above_the_limit():
|
||||
budget = _budget(12)
|
||||
assert gate.over_ceiling({"LIT006": 12}, budget) == frozenset() # at limit
|
||||
assert gate.over_ceiling({"LIT006": 13}, budget) == frozenset({"LIT006"}) # over limit
|
||||
assert gate.over_ceiling({}, budget) == frozenset() # missing rule counts as zero
|
||||
|
||||
|
||||
def test_over_ceiling_is_independent_across_rules():
|
||||
budget = {"LIT001": {"baseline": 5, "slack": 0}, "LIT006": {"baseline": 10, "slack": 0}}
|
||||
budget = {"LIT001": {"limit": 5}, "LIT006": {"limit": 10}}
|
||||
assert gate.over_ceiling({"LIT001": 6, "LIT006": 10}, budget) == frozenset({"LIT001"})
|
||||
|
||||
|
||||
def test_evaluate_blames_only_a_rule_over_cap_and_over_base():
|
||||
budget = _budget(10, 0) # cap 10
|
||||
# over cap and grown vs base -> breach
|
||||
def test_evaluate_blames_only_a_rule_over_limit_and_over_base():
|
||||
budget = _budget(10)
|
||||
# over limit and grown vs base -> breach
|
||||
assert [b.rule for b in gate.evaluate({"LIT006": 12}, {"LIT006": 9}, budget)] == ["LIT006"]
|
||||
# over cap but flat vs base (pre-existing drift) -> not blamed
|
||||
# over limit but flat vs base (pre-existing drift) -> not blamed
|
||||
assert gate.evaluate({"LIT006": 12}, {"LIT006": 12}, budget) == []
|
||||
# within cap -> not blamed regardless of base
|
||||
# within limit -> not blamed regardless of base
|
||||
assert gate.evaluate({"LIT006": 10}, {"LIT006": 0}, budget) == []
|
||||
|
||||
|
||||
def test_update_ratchets_limit_down_by_what_the_branch_fixed_never_up():
|
||||
budget = {"LIT001": {"limit": 100}, "LIT006": {"limit": 10}}
|
||||
# LIT001 fixed 15 (60 -> 45) so its limit falls 100 -> 85; LIT006 grew, so its
|
||||
# limit holds flat at 10.
|
||||
current = {"LIT001": 45, "LIT006": 12}
|
||||
base = {"LIT001": 60, "LIT006": 9}
|
||||
assert gate.ratcheted_budget(budget, current, base) == {
|
||||
"LIT001": {"limit": 85},
|
||||
"LIT006": {"limit": 10},
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,34 +1,26 @@
|
|||
{
|
||||
"LIT001": {
|
||||
"baseline": 21452,
|
||||
"slack": 2000
|
||||
"limit": 23452
|
||||
},
|
||||
"LIT002": {
|
||||
"baseline": 25022,
|
||||
"slack": 2500
|
||||
"limit": 27522
|
||||
},
|
||||
"LIT003": {
|
||||
"baseline": 397,
|
||||
"slack": 25
|
||||
"limit": 422
|
||||
},
|
||||
"LIT004": {
|
||||
"baseline": 2515,
|
||||
"slack": 50
|
||||
"limit": 2565
|
||||
},
|
||||
"LIT005": {
|
||||
"baseline": 0,
|
||||
"slack": 0
|
||||
"limit": 0
|
||||
},
|
||||
"LIT006": {
|
||||
"baseline": 1013,
|
||||
"slack": 100
|
||||
"limit": 1113
|
||||
},
|
||||
"LIT007": {
|
||||
"baseline": 0,
|
||||
"slack": 0
|
||||
"limit": 0
|
||||
},
|
||||
"LIT008": {
|
||||
"baseline": 914,
|
||||
"slack": 90
|
||||
"limit": 1004
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue