ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.
Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:
- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
on the failure path, so the NameError, not the assertion, is what ran.
test_llm_guard_error_raising is the worst: it passes today with content
safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
supports 3.10. Guarded behind the exceptiongroup backport that anyio already
pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
a helper that catches HTTPException by a name it never imported, so the
challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
test_claude_agent_sdk, and an unused helper in test_end_users calling a
function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
FileNotFoundError behind a NameError.
Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
The existing mutation workflow is manual and mutates a whole folder, which takes
hours, so nothing about a pull request's own test quality gets checked before it
merges.
This adds the diff-scoped half, following Google's approach in "State of Mutation
Testing at Google": mutate what the change touched, not the codebase.
scripts/mutation_diff_scope.py reads the diff against the merge base, rewrites
[tool.mutmut] so paths_to_mutate is the changed production files and tests_dir is
the tests mirroring them, then emits mutant-name globs for the functions holding
the changed lines. mutmut trampolines per function and per method, so that is the
smallest unit it can be asked to run.
The new workflow is advisory: results land in the job summary and an artifact,
and nothing blocks a merge. A --max-functions cap bounds the worst case and
prints what it dropped rather than truncating quietly.
The report script now counts only the mutants a run actually executed, since a
diff-scoped run leaves everything else at "not checked" and the old summary
folded those into the score.
* test: run the 30 test files stranded in the second mirror
tests/litellm sat beside tests/test_litellm, which is the mirror the repo
convention names, and no job collected it. The allowlist called the directory
unresolved and assumed it was a duplicate. It is not: 30 of its 34 files have no
counterpart in the real mirror, so they are tests nobody has run since they were
written, not copies of tests that run elsewhere.
Moving them in is byte-identical, and it is what makes them run. Every one is
now claimed by a shard's test-path rather than by an allowlist entry, and the
216 tests they hold pass. Directories that needed to become packages did, since
several files are named test_transformation.py and pytest cannot import two of
those from non-package directories in one session.
Never running is why three assertions had drifted away from the code:
* nvidia.nemotron-super-3-120b max_output_tokens, 32000 -> 32768
* sambanova/MiniMax-M2.7 max_input_tokens, 204800 -> 196608
* the Vertex text-to-speech handler moved from data= to json=, so the test
reads the decoded body off the json kwarg instead of parsing the data one
The first two follow model_prices_and_context_window.json, which the catalog
sync keeps current; the third follows the handler. In all three the test was the
stale side.
The lint workflow ran test_no_hardcoded_secrets.py by path and now points at the
new one.
Four files stay behind. Each shares a filename with a live test whose contents
are disjoint from it, so landing those means merging test bodies, which is a
content review rather than a move. The allowlist entry now names those four and
records how many tests each would bring, in place of calling the whole
directory unresolved.
* fix(ci): keep the secret scan out of the mirror's conftest
The secret-scan job runs pytest under uv run --no-project, so its environment
holds pytest and nothing else. That worked while the file sat in tests/litellm,
which has no conftest, and broke the moment it moved into tests/test_litellm,
whose conftest imports litellm on collection: ModuleNotFoundError: No module
named 'dotenv', before a single test ran.
The file is a repo-wide static scan that imports only base64, os, re and pytest,
so it belongs with the other repo-wide checks in tests/code_coverage_tests,
which has no conftest, rather than in the package mirror. Installing the full
dependency set into a 15-second job to satisfy a conftest it does not use would
be the wrong trade.
Verified with the job's exact command:
uv run --no-project --with 'pytest==9.0.2' pytest \
tests/code_coverage_tests/test_no_hardcoded_secrets.py -q
1 passed in 0.47s
* feat(ci): catch files a -k expression deselects from every job
The coverage census asks whether some job names a file. It cannot ask what that
job's -k then does with it, and the gap is not hypothetical: tests/local_testing
is globbed by five jobs, two of which carry
-k "... and not router and not assistants and not langfuse and not caching and not cache"
while the other three keep one keyword each. Any file whose path holds an
excluded term is dropped by the first two and matched by none of the rest, so it
runs nowhere while the census counts it as covered. 118 tests across eight
caching files sit in exactly that hole today.
The new mode reads the same CircleCI jobs the census already parses and asks
whether each globbed file survives its job's selector. Two facts about -k make
that decidable without running pytest: it matches an item's own name and its
parents', so a term appearing in the module path deselects the whole file; and
the names it can match are otherwise the classes and functions in the file,
which ast reads. A positive term is therefore satisfied by the path or by a name
inside, which is what keeps a langfuse-named test inside test_logging.py from
being reported.
Where the parser is unsure it stays quiet. An expression with or, parentheses,
or a negated group is left unmodelled and its job is treated as claiming
everything it globs, so an unparsed selector can never raise a false alarm.
Glob translation learned character classes, without which
tests/local_testing/**/test_[a-mA-M]*.py matches nothing and the guard would
report that whole directory. The census and shard counts are unchanged by it,
2423 files and 327 shard children before and after.
Validated against the real thing: collecting tests/local_testing under each
job's own selector leaves 175 of 1577 tests unselected, in exactly the ten files
this check derives statically, no more and no fewer. Two of the ten are named
outright by other jobs, which the check credits, leaving the eight now recorded
in the allowlist as a decision rather than an accident.
Verified red-first: dropping one of those eight from the allowlist reports it,
and adding 'and not embedding' to the two part jobs reports test_embedding.py
and test_get_optional_params_embeddings.py.
* fix(ci): keep the slice guard from pairing one command's -k with another's glob
Two accuracy notes from review, both about the parser's model rather than its
current verdicts.
A job that runs several pytest commands offers no way to tell which glob a -k
belongs to, since both are read out of the same flattened job text. Combining
them could pair one command's exclusion with another command's glob and report a
file that in fact runs. Such a job is now left unmodelled, which means it claims
everything it globs, matching how the parser already treats an expression it
cannot read. Only one job in the config has two globs today and it carries no
-k at all, so no verdict changes.
The second is a deliberate limit, now stated where it lives: an excluded term is
only honoured when it sits in the module path, because that is the case that
takes the whole file with it. A term matching one function inside drops that
test and leaves the file running, and reporting it would be a false alarm.
Answering per-test instead would need a baseline of test ids that churns on
every rename, for a smaller failure than a file going dark.
Both are pinned by tests.
* feat(ci): ratchet tests that skip themselves when a credential is absent
* docs(ci): name the new rule where the gate's rules are listed
* fix(ci): require the condition to test for absence before TQ006 fires
* test: settle three allowlist entries that were open questions
The allowlist is meant to hold decisions, not deferrals, so an entry reading
'needs moving' or 'referenced by no job' is a gap wearing an exemption. These
three each get an answer.
The two prompt-factory tests move into the mirror, which is what their own entry
said they needed. Both were passing the whole time, so the 23 tests they hold
start running and the entry goes away rather than getting reworded.
test_aio_http_image_conversion.py is not a test. It fetches live image URLs,
times aiohttp against httpx, prints the ratio, and asserts nothing, and pytest
cannot collect it because its functions take arguments rather than fixtures.
Running it beside its siblings would buy CI a network dependency and a number
nothing reads, so it stays exempt with that written down.
test_litellm_proxy_extras_utils.py stays exempt with a measured reason. 24 of
its 28 tests pass; the 4 in TestMigrationSQLIdempotency fail because nine
migrations from 2026-04 onward use bare CREATE TABLE, ADD COLUMN and CREATE
INDEX where that file requires guarded forms. The convention eroded quietly
precisely because the test enforcing it has never run. Wiring it up is blocked
on what to do about those migrations, and editing them is not the answer, since
Prisma checksums an applied migration and a changed one breaks migrate deploy
for existing installs.
Allowlist entries 10 -> 9, paths 88 -> 86.
* docs(ci): correct the migration count in the proxy-extras allowlist reason
tests/proxy_unit_tests had a 30-line YAML parser inlined in its workflow that
failed the run when a test file there belonged to no shard. tests/test_litellm
is sharded the same way, with no catch-all bucket, and had no such guard: a new
directory under it (or under its proxy subtree) is collected by nothing and runs
nowhere, and the coverage census cannot see it because a token like
tests/test_litellm/test_*.py already answers 'yes, that tree runs'.
The two questions differ. The census asks whether a file runs at all, so an
ancestor path standing in for everything beneath it is a fine answer. Shard
assignment asks which shard owns a child, and there that same ancestor path is
precisely the bug. _token_covers keeps the first meaning; _token_names adds the
second, and the guard now walks a list of sharded trees rather than one hardcoded
directory. Both read the same test-path keys, so there is one workflow parser.
A directory needs a shard when it holds a test file, not when it is named test_*.
That drops the hardcoded test_configs exception and keeps fixture directories
like expected_fine_tuning_api out on their own merits.
The job keeps its name and its workflow, since assert-shard-coverage is a
required status check on litellm_internal_staging.
Verified red-first: a planted directory under tests/test_litellm, a planted
directory under tests/test_litellm/proxy, and a planted file under
tests/proxy_unit_tests each fail the guard, while a fixture-only directory does
not. 327 children across the three trees are assigned today.
* feat(ci): ratchet the test suite's zero-assert, mock-echo and global-state debt
The suite's dominant failure mode is tests that cannot fail for the reason anyone
would want them to. The testing-strategy audit measured five shapes of it, and
nothing mechanical stops any of them from reproducing, so they keep reproducing.
`scripts/check_test_quality.py` is an AST checker for those five, emitting the
same `path:line: CODE message` contract as `scripts/check_type_discipline.py`:
TQ001 a collectible test with no assertion of any kind
TQ002 mock-echo, where every assertion only inspects the mock that was patched
TQ003 sys.path.insert inside the test tree
TQ004 raw `os.environ[...] =`, which leaks into whatever runs next
TQ005 `litellm.<attr> =`, the process-wide leak the 491-line conftest undoes
`scripts/test_quality_gate.py` caps each rule against test-quality-budget.json,
seeded at exactly today's count, and fails only when a rule is both over its
limit and higher than the base being merged into, so a change is blamed for what
it adds and never for drift already in the base. `--update` lowers a limit by
what a branch cleared, so the ceilings only ever fall. It runs in the existing
required lint job, which means it enforces without a ruleset change.
TQ001 follows assertions into helpers defined in the same module, transitively.
Without that it flagged 111 tests in tests/e2e, the harness this program holds up
as the reference, because that suite factors its assertions into shared helpers
(`assert_auth_denied(result, ...)`). Following them leaves 25, all of which reach
their assertions across a module boundary; those are grandfathered and documented
rather than papered over.
The seeded counts land within about 10% of the audit's independent numbers for
every rule measured on the same subtree, which is the cross-check that the
definitions here match the ones the audit pinned.
* fix(ci): resolve test helpers per scope, not by bare name
The helper walk keyed every function in a module by its bare name, so two
same-named helpers in different classes collided and the last one parsed won.
A test calling `self._check()` could be cleared by a `_check` belonging to a
different class, or flagged because of one.
Resolution is now scoped: a bare name looks up the module-level functions, and
`self.<name>` looks up the enclosing class's own methods and no other class's.
Recursion is tracked by function identity rather than by name, so the cycle
guard cannot be confused by the same collision.
This surfaced one real zero-assert test that a same-named helper elsewhere had
been clearing, so TQ001 seeds at 750 rather than 749.
The test module has to register itself in sys.modules before exec_module:
`@dataclass(slots=True)` rebuilds its class through `sys.modules[__module__]`,
and Scope fails to construct without it. Recorded at the call site, since it
reads like avoidable global mutation otherwise.
* fix: register test-quality-budget.json with the ratchet alarm
The repo keeps one census over its budget files: every *-budget.json on disk
must appear in DEFAULT_BUDGETS, or its ceilings can be raised with no signal.
tests/test_litellm/test_budget_ratchet_check.py asserts that set equality and
caught the new budget on the way in.
Registering it also turns the alarm on for TQ001-TQ005, so a later PR cannot
quietly raise a test-quality ceiling. The file already uses the {limit: N}
schema the ratchet reads, so no other change was needed.
* test: retire tests/old_proxy_tests, which holds no tests
Twenty files named test_*.py, and pytest collects nothing from any of them:
uv run pytest tests/old_proxy_tests --collect-only -q
no tests collected, 16 errors in 114.17s
They are manual snippets against a running proxy, written at module level with
no test function, no assertion and no entry point, so the only thing the name
buys them is a place on the coverage allowlist. Sixteen of the twenty cannot
even be imported in this environment, wanting langchain, llama_index or
google.api_core, and ten still point at 0.0.0.0:8000, which stopped being the
proxy's default port some time ago.
Nothing outside the directory refers to it apart from the allowlist entry, which
goes with it. The other loose contents go too: five load_test_*.py scripts, a
bursty variant, two committed log files, an essay fixture and a stray .js
snippet.
Allowlist paths 88 -> 68, test files 2422 -> 2402, and no job loses anything it
was running. Recoverable from history if a snippet turns out to be someone's
habit.
* test: drop the retired old_proxy_tests paths from the coverage allowlist
The agent job's CircleCI glob collected `tests/agent_tests/**/test_*.py` and then
piped it through `grep -v` to drop `local_only_agent_tests/`. `assert_ci_coverage.py`
reads the glob but not the pipeline, so those two files looked covered and were
invisible to the census. The glob now excludes them structurally and they carry an
allowlist entry instead, which is a decision on the record rather than a hidden
filter. The collected file set is unchanged: `tests/agent_tests/` holds exactly one
CI-runnable test at the top level.
`tests/scim_tests/` held a single JSON fixture and no tests, referenced from nowhere.
`.github/workflows/` is for workflows. Both stray scripts move to `.github/scripts/`
with their callers updated: the price-file updater is invoked by
`auto_update_price_and_context_window.yml`, and the translation-report runner by
`make test-llm-translation`. The audit listed the latter as orphaned, but Makefile
line 317 still runs it, so it moves rather than being deleted.
The rollout heads-up workflow was a deliberate one-shot for the agent-shin rollout.
That rollout is done, the triage and auto-close workflows have been running daily
since June, so the pre-flip warning window is long past. Its script and dedicated
test go with it, and the sibling workflow-invariant test drops its entry.
Nine workflow files existed only to make a single call to _test-unit-base.yml
with a different test-path. Adding a shard meant adding a file; changing
anything shared meant editing nine. One matrix caller replaces them, so a shard
is now one entry.
Check names are unchanged, which is the whole constraint. A reusable-workflow
job reports as "<job name> / <inner job name>", so setting `name` to the shard id
alone reproduces today's context strings exactly: the eleven the matrix produces
are eleven of the twenty-three "/ Run tests" contexts the branch ruleset
requires, matched string for string. No ruleset edit is needed and none should
be made for this.
Every matrix entry states its timeouts even where they equal the base defaults.
An absent matrix key renders as an empty string rather than falling back, and an
empty string is not a number, so a partially-specified entry would fail the call.
tests/proxy_unit_tests keeps test-unit-proxy-db.yml. It is already a matrix and
its shard-coverage guard reads that file by name, so folding it in belongs with
generalising that guard into assert_ci_coverage.py rather than here. Its twelve
shards are the remaining required contexts.
test-unit-documentation.yml stays too: it does not call the base workflow.
The UI unit test job narrows a pull request to `vitest related <changed
files>`. `related` maps a file to the tests that import it, so a file no
test imports maps to nothing, and `--passWithNoTests` turns that empty
selection into a green job. package.json, package-lock.json, the Vitest,
Tailwind and TypeScript configs and tests/setupTests.ts are all in that
category even though each of them can change the behaviour of every test
in the suite, so a dashboard dependency bump merged having run no unit
tests at all and only got real coverage later, from the full run on the
push to litellm_internal_staging.
Keep `related` for the common case where a pull request only touches
files under src/, and fall back to the full suite as soon as one changed
file sits outside it. The decision lives in
.github/scripts/select_ui_test_scope.sh so it can be tested on its own,
next to the existing classify_changes.sh gate.
PR #37550 taught the backend unit-test shards to read the pull request's own
file list, but four required jobs were never wired to that gate and ran in full
on every pull request regardless of what it touched. A UI-only pull request
still paid roughly 17 runner-minutes of Python work it could not have affected,
and a backend-only one still installed and built the dashboard.
Lint and the MCP suite now take the existing backend decision. The dashboard
build and unit tests take a new ui decision, which tracks ui/ rather than
reusing client: client deliberately runs whenever the backend changes, because
it gates CircleCI's end-to-end jobs that drive a real proxy, while the build and
the unit tests cannot see the backend at all. CI config counts as ui-relevant
too, so a pull request that rewrites the dashboard workflows still exercises
them instead of shipping unvalidated.
The gate stays inside the job rather than moving to on.paths or to a job-level
condition on the shard callers. A workflow filtered out by on.paths never starts
and never reports, so a required check waits forever, and a skipped caller job
publishes its own name instead of the nested "<shard> / Run tests" the ruleset
requires. Both were measured before settling on this shape.
Three setup steps in the shard base and in the documentation job also leaked
past the gate, so a skipped shard still spent about twelve seconds installing uv
and restoring its cache. They now carry the same condition, and the documentation
job stops cloning litellm-docs when it has nothing to validate.
detect-backend-changes diffed the event payload's base.sha against the
checked-out ref. Those are two different points in time: actions/checkout
resolves refs/pull/N/merge, and GitHub recomputes that ref whenever the base
branch advances, so the diff picked up whatever landed on staging between the
event firing and the job starting. On a recent UI-only pull request three
backend commits from staging were attributed to the branch, and every backend
shard ran in full
Ask the API which files the pull request touches instead. That is the same set
the Files changed tab shows, and it is immune to either endpoint moving. The
shell body moves into .github/scripts/detect_backend_changes.sh so it can be
exercised directly, and the fail-open paths now also cover an API failure, a
file list past the API's 3000-entry listing ceiling, and a classifier that
prints something unexpected
tests/proxy_unit_tests/ runs twice on every pull request. The nine
alphabetical shards in test-unit-proxy-legacy.yml cover the same
directory as the twelve semantic shards in test-unit-proxy-db.yml,
and all nine are required checks, so each PR pays for the directory
twice before it can merge.
The semantic shards are a strict superset. Expanding both matrices
against the working tree, the legacy globs collect 58 files while the
semantic shards name all 59: test_model_response_typing is a directory
and matches none of the test_[a-z]*.py patterns, so the legacy lane has
silently skipped it. The semantic workflow also carries its own
assert-shard-coverage guard, which fails if a file under that directory
is not assigned to a shard, so a new file cannot drop out of CI once the
alphabetical fallback is gone.
Verified with .github/scripts/assert_ci_coverage.py: 2380 test files
have a runner both before and after the deletion. Removing
test-unit-proxy-db.yml as well takes the same guard red with 58
orphaned files, which confirms the guard is live and that the semantic
shards, not the legacy ones, are what hold the coverage.
The nine bare contexts this workflow published (auth-and-jwt,
key-generation, proxy-config, proxy-server, proxy-server-extras,
proxy-token-counter, proxy-response-and-misc, proxy-user-auth-and-spend,
proxy-utils) still need pruning from the guard-internal-staging ruleset,
which needs admin rights and is not part of this change
Adds vpc_id/public_subnet_ids/private_subnet_ids to deploy into existing
networking, plus create_database/database_url and create_redis/redis_url to
use existing data stores or none at all. Defaults keep today's module-owned
behavior.
A container reads a secret by ARN, which gives Terraform no edge to the
_version that writes the value. The managed-Aurora path hid that behind the
cluster create; the bring-your-own path has nothing slow in between, so the
migration, the bootstrap, and both services now depend on the versions
explicitly.
Supplied private subnets must cover two AZs whenever Aurora or ElastiCache is
module-created, since both subnet groups require it, and a check block warns
when a Redis-less stack can run more than one gateway process, because per-key
rate limits are counted per process without Redis.
The proxy-server-core matrix entry named test_proxy_server_caching.py and
test_proxy_server_langfuse.py by path. This PR deletes both, so pytest exited 5
with "no tests collected" and the whole shard failed without running the four
files that do exist.
assert-shard-coverage did not catch it because it only checks one direction:
every file under tests/proxy_unit_tests/ must appear in some shard. It never
checks that every path a shard names still exists, so a stale entry passes.
After this change no shard names a missing path and no file is left without a
shard. The shard collects 85 tests.
DataTable accepted any mix of its 40-odd props and rejected the incoherent
combinations at runtime, from a validator that threw during the first render.
A caller only found out it had wired server sorting without a `sorting` prop
when the page blew up in front of them.
Split the public prop type into mode-keyed unions instead, so the compiler
rejects those combinations at the call site. `validateDataTableConfig` and
`DataTableConfigError` go away; the component body reads an unchanged flat
`DataTableResolvedProps`, which every union member is assignable to, so there
is no narrowing inside it.
All 44 existing call sites typecheck against the new union unchanged, which
`next build` covers. That build only typechecks the app module graph, so the
prop type itself needed a gate of its own: `npm run test:types` runs vitest's
typecheck mode over `*.test-d.tsx`, and the unit workflow now runs it. The
four guards deleted from `DataTable.test.tsx` come back there as compile-time
assertions, and loosening the union back to the flat shape fails all five.
