mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
8018bc3996
The collector read @pytest.mark.covers off every collected item, and collection does not evaluate skips, so a test carrying both a skip and a covers marker reported its cell as covered while asserting nothing. 17 files under tests/e2e do exactly that, which inflated the headline from 290/434 to 311/434. A cell now counts as covered only when at least one test pytest would actually run declares it; a cell claimed by both a live and a skipped test stays covered. Skip state comes from pytest's own evaluator, so skip, skipif (bool and string conditions), and module-level pytestmark resolve exactly as they do in the e2e run. Cells left uncovered this way are listed under the headline and exported as skipped_markers (JSON) and litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces instead of disappearing; the Loki line contract is unchanged. A marker on a skipped test that points outside the registry is still an orphan, so --strict keeps its reach. Because skipif resolves against the environment the collector runs in, the number now depends on that environment; run it where the e2e suite runs. A pytest.skip() call inside a test body remains invisible to a static pass, which the module docstring and README both state.
92 lines
4.5 KiB
Markdown
92 lines
4.5 KiB
Markdown
# e2e coverage registry
|
|
|
|
This directory is the **denominator** for e2e test coverage: the set of behaviors we
|
|
want covered, one row per behavior, checked into the repo so coverage is a number we
|
|
can track instead of a guess. It implements the plan in the "E2E Coverage Tracking"
|
|
note; the naming grammar lives in `tests/e2e/CLAUDE.md`.
|
|
|
|
## The model
|
|
|
|
A **cell** is one customer-noticeable behavior a single e2e test can assert pass/fail
|
|
on, for example `llm.chat_completions.bedrock_converse.tool_use.stream.works`. Cells are
|
|
grouped `module > feature > test`, with LLM cells split into `Core LLMs` and
|
|
`Non-Core LLMs` for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
|
|
`fail_before_fix` flag.
|
|
|
|
The rows live in per-prefix YAML files (`llm_*.yaml`, `mgmt.yaml`, `mcp.yaml`,
|
|
`reliability.yaml`, `quota_management.yaml`, `logging.yaml`, `guardrail.yaml`,
|
|
`other.yaml`) and validate against
|
|
the discriminated union in `schema.py`, so an LLM row cannot carry a guardrail field and
|
|
vice versa. `llm` rows with `subject_endpoint` of `chat_completions`, `messages`, or
|
|
`responses` roll up to `Core LLMs`; all other LLM endpoints roll up to `Non-Core LLMs`.
|
|
LLM endpoint, route, and capability values are typed in `schema.py`, so new taxonomy
|
|
values require an explicit schema change. `logging` and `guardrail` are two id-prefixes
|
|
that roll up into the single `Logging & Guardrails` dashboard module.
|
|
|
|
A test declares what it covers with a marker:
|
|
|
|
```python
|
|
@pytest.mark.covers("llm.chat_completions.openai.tool_use.stream.works")
|
|
def test_openai_streaming_tool_calls(self) -> None:
|
|
...
|
|
```
|
|
|
|
## The number
|
|
|
|
`collector.py` diffs the registry against those markers and reports coverage per module.
|
|
It is static: a collect-only pass reads the markers, so it runs no test and needs no live
|
|
proxy. Whether a covered cell currently passes or fails is a separate, live concern.
|
|
|
|
A skipped test asserts nothing, so its markers do not count. A cell is covered only when
|
|
at least one test pytest would actually run declares it; a cell claimed by both a live
|
|
test and a skipped one stays covered. Skip state comes from pytest's own evaluator, so
|
|
`skip` and `skipif` resolve exactly as they do in the e2e run, which also means a
|
|
`skipif` on an absent credential makes that cell uncovered in the environments where the
|
|
test cannot run. Cells left uncovered this way are listed under the headline (and counted
|
|
by `litellm_e2e_coverage_skipped_markers`) so an unskipped-pending gap is visible rather
|
|
than inflating the number. The one skip the collector cannot see is `pytest.skip()`
|
|
called from inside a test body, since it does not exist until the test runs.
|
|
|
|
```
|
|
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
|
|
```
|
|
|
|
Use `--format loki` after the e2e pytest run in the same Kubernetes job/pod to print
|
|
structured stdout lines for Loki:
|
|
|
|
```
|
|
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format loki --strict
|
|
```
|
|
|
|
This emits exactly one `COVERAGE_TOTAL` line and one `COVERAGE_MODULE` line per module
|
|
in `MODULE_ORDER`, in that order. Loki uses log-safe `module=` labels from
|
|
`LOKI_MODULE_LABELS` (`core_llms`, `management_ui`, etc.) so existing JSON and
|
|
Prometheus consumers keep their human-readable module names unchanged.
|
|
|
|
The headline is overall coverage. The collector also lists markers that point at ids
|
|
not in the registry, so a typo or an unenumerated behavior surfaces instead of being
|
|
silently dropped.
|
|
|
|
Use strict mode in CI once existing draft markers are reconciled:
|
|
|
|
```
|
|
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
|
|
```
|
|
|
|
Strict mode exits non-zero on `@pytest.mark.covers(...)` ids that are not checked into
|
|
the registry. Add `--fail-on-collection-errors` when the job should also fail on pytest
|
|
collection errors.
|
|
|
|
## Status: this is a draft for review
|
|
|
|
The cells were enumerated from the codebase and the tiers are a first proposal. Known
|
|
things to settle before treating the set as final:
|
|
|
|
- tiers are proposed, not signed off; 125 P0 is a lot to prove fail-before-fix, so P0 may
|
|
want tightening
|
|
- a few cells need a support check or a prune (for example `llm.embeddings.anthropic.*`
|
|
and `reliability.perf.throughput.under_slo`)
|
|
- auth is covered in two places (`other.auth.*` and the mgmt authz assertions); the
|
|
boundary needs a decision, and the auth cluster may deserve promotion to its own module
|
|
- the P2 "niche" cells each stand in for a large tail of integrations/providers by design,
|
|
so the denominator is deliberately P0-weighted rather than a full inventory
|