mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
ffab5a39d0
* feat(ci): ratchet the test suite's zero-assert, mock-echo and global-state debt
The suite's dominant failure mode is tests that cannot fail for the reason anyone
would want them to. The testing-strategy audit measured five shapes of it, and
nothing mechanical stops any of them from reproducing, so they keep reproducing.
`scripts/check_test_quality.py` is an AST checker for those five, emitting the
same `path:line: CODE message` contract as `scripts/check_type_discipline.py`:
TQ001 a collectible test with no assertion of any kind
TQ002 mock-echo, where every assertion only inspects the mock that was patched
TQ003 sys.path.insert inside the test tree
TQ004 raw `os.environ[...] =`, which leaks into whatever runs next
TQ005 `litellm.<attr> =`, the process-wide leak the 491-line conftest undoes
`scripts/test_quality_gate.py` caps each rule against test-quality-budget.json,
seeded at exactly today's count, and fails only when a rule is both over its
limit and higher than the base being merged into, so a change is blamed for what
it adds and never for drift already in the base. `--update` lowers a limit by
what a branch cleared, so the ceilings only ever fall. It runs in the existing
required lint job, which means it enforces without a ruleset change.
TQ001 follows assertions into helpers defined in the same module, transitively.
Without that it flagged 111 tests in tests/e2e, the harness this program holds up
as the reference, because that suite factors its assertions into shared helpers
(`assert_auth_denied(result, ...)`). Following them leaves 25, all of which reach
their assertions across a module boundary; those are grandfathered and documented
rather than papered over.
The seeded counts land within about 10% of the audit's independent numbers for
every rule measured on the same subtree, which is the cross-check that the
definitions here match the ones the audit pinned.
* fix(ci): resolve test helpers per scope, not by bare name
The helper walk keyed every function in a module by its bare name, so two
same-named helpers in different classes collided and the last one parsed won.
A test calling `self._check()` could be cleared by a `_check` belonging to a
different class, or flagged because of one.
Resolution is now scoped: a bare name looks up the module-level functions, and
`self.<name>` looks up the enclosing class's own methods and no other class's.
Recursion is tracked by function identity rather than by name, so the cycle
guard cannot be confused by the same collision.
This surfaced one real zero-assert test that a same-named helper elsewhere had
been clearing, so TQ001 seeds at 750 rather than 749.
The test module has to register itself in sys.modules before exec_module:
`@dataclass(slots=True)` rebuilds its class through `sys.modules[__module__]`,
and Scope fails to construct without it. Recorded at the call site, since it
reads like avoidable global mutation otherwise.
* fix: register test-quality-budget.json with the ratchet alarm
The repo keeps one census over its budget files: every *-budget.json on disk
must appear in DEFAULT_BUDGETS, or its ceilings can be raised with no signal.
tests/test_litellm/test_budget_ratchet_check.py asserts that set equality and
caught the new budget on the way in.
Registering it also turns the alarm on for TQ001-TQ005, so a later PR cannot
quietly raise a test-quality ceiling. The file already uses the {limit: N}
schema the ratchet reads, so no other change was needed.
|
||
|---|---|---|
| .. | ||
| adaptive_router_demo | ||
| health_check | ||
| benchmark_anthropic_messages_perf.py | ||
| benchmark_chat_completions_perf.py | ||
| benchmark_mock.py | ||
| benchmark_model_response_creator.py | ||
| benchmark_proxy_vs_provider.py | ||
| benchmark_streaming_chunk_overhead.py | ||
| budget_ratchet_check.py | ||
| check_test_quality.py | ||
| check_type_discipline.py | ||
| create_litellm_branch.ps1 | ||
| create_litellm_branch.sh | ||
| create_team_key_and_submit_guardrail.sh | ||
| eval_compression.py | ||
| gate_slot_lock.py | ||
| install-cli.sh | ||
| install.sh | ||
| install_git_hooks.sh | ||
| mock_bedrock_passthrough_target.py | ||
| mock_grayswan_timeout_server.py | ||
| mutation_report.py | ||
| pre_commit_lint.sh | ||
| prisma_generate_if_needed.py | ||
| ruff_strict_gate.py | ||
| test_agent_mcp_endpoints.sh | ||
| test_guardrails_register_endpoints.sh | ||
| test_quality_gate.py | ||
| test_tool_allowlist_script.py | ||
| tpm_headline_test.sh | ||
| type_check_gate.py | ||
| type_discipline_gate.py | ||
| verify_adaptive_router.py | ||
| with_dashboard_node.sh | ||