The collector read @pytest.mark.covers off every collected item, and
collection does not evaluate skips, so a test carrying both a skip and a
covers marker reported its cell as covered while asserting nothing. 17
files under tests/e2e do exactly that, which inflated the headline from
290/434 to 311/434.
A cell now counts as covered only when at least one test pytest would
actually run declares it; a cell claimed by both a live and a skipped
test stays covered. Skip state comes from pytest's own evaluator, so
skip, skipif (bool and string conditions), and module-level pytestmark
resolve exactly as they do in the e2e run. Cells left uncovered this way
are listed under the headline and exported as skipped_markers (JSON) and
litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces
instead of disappearing; the Loki line contract is unchanged. A marker
on a skipped test that points outside the registry is still an orphan,
so --strict keeps its reach.
Because skipif resolves against the environment the collector runs in,
the number now depends on that environment; run it where the e2e suite
runs. A pytest.skip() call inside a test body remains invisible to a
static pass, which the module docstring and README both state.
Introduce the e2e coverage denominator: 282 behavior cells across the six
tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance,
Logging & Guardrails, Other), one validated YAML row each, plus a collector
that diffs the registry against @pytest.mark.covers markers and reports
coverage per module.
The registry rows validate against a pydantic discriminated union so a row
cannot carry a field from another module. The collector is static: a
collect-only pass reads the markers, so it runs no test and needs no live
proxy. Register the covers marker suite-wide so that pass works under
--strict-markers.
This is a draft for review. Tiers are proposed rather than signed off, and a
few cells still need a support check or a prune.
