litellm/tests/e2e/grafana
mubashir1osmani 98765f65af
feat(e2e): emit structured E2E_RESULT lines for package status history (#33578)
* feat(e2e): emit structured E2E_RESULT lines for package status history

Pytest progress logs only expose file basenames and collapse multi-test files
into one status-history row. Emit one logfmt E2E_RESULT per finished node with
package, file, outcome, duration_ms, node_id, and covers so Grafana can roll up
by package (stable cardinality) and drill down by node_id in Explore

* test(e2e): drop unit tests from the live e2e tree

tests/e2e is for live proxy suites only. Remove harness, coverage-registry,
and claude_code unit trees so the e2e run does not collect them

* fix(e2e): type E2E_RESULT hook for basedpyright zero-error gate

Protocol-typed covers extraction and pluggy Result typing on the
makereport hook so tests/e2e stays under the e2e basedpyright ceiling

* fix(e2e): drop unused xfailed/xpassed from E2E_RESULT Outcome

We never emit those states; xfail collapses to skipped/passed via pytest
report flags. Keep the literal honest so the dashboard only sees real outcomes

* fix(e2e): strip tests/e2e prefix when deriving E2E_RESULT package

Repo-root pytest nodeids are tests/e2e/<suite>/...; without stripping,
every line would package=tests and status history would be useless

* fix(e2e): use pytest wrapper=True instead of deprecated hookwrapper

pytest 8.1+ deprecates hookwrapper; yield returns the TestReport directly
so we return it for the outer chain and drop pluggy.Result

* fix(e2e): import e2e_result_reporter at module load

Surface a missing module as a collection-time ImportError instead of a
per-test hook failure mid-run

* test(e2e): restore coverage_registry/test_collector.py

Needed to validate registry coverage math and the checked-in cell
denominator; not a live proxy suite
2026-07-16 15:01:01 -07:00
..
status_history_panels.md feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00