diff --git a/tests/e2e/coverage_registry/GRAFANA_DASHBOARD.md b/tests/e2e/coverage_registry/GRAFANA_DASHBOARD.md deleted file mode 100644 index 13e1cf2d5ee..00000000000 --- a/tests/e2e/coverage_registry/GRAFANA_DASHBOARD.md +++ /dev/null @@ -1,92 +0,0 @@ -# Grafana Dashboard Brief - -This dashboard should answer two questions: - -1. How much e2e coverage do we have by module right now? -2. Is e2e coverage by module improving or regressing over time? - -The dashboard should use plain **Coverage** language. Avoid exposing P0/P1/P2 in the -default view; tiers can be added later as a filter or drilldown. - -## Panels - -### Coverage by Module - -Use a bar chart or table with one row per module: - -- Core LLMs -- Non-Core LLMs -- MCPs -- Management/UI -- Reliability & Performance -- Logging & Guardrails -- Other - -Each row should show: - -- `covered` -- `total` -- `coverage_percent` - -Formula: - -```text -coverage_percent = covered / total * 100 -``` - -### Coverage Trend by Module - -Use a time-series chart with one line per module. - -- X-axis: CI run timestamp, scrape timestamp, or pushed metric timestamp -- Y-axis: `coverage_percent` -- Series label: module name - -This shows whether coverage is improving across dates. - -## Data Contract - -Generate coverage data from the registry collector: - -```bash -cd tests/e2e -PYTHONPATH=. python -m coverage_registry.collector --format prometheus --strict -``` - -For artifact-based jobs, JSON is also available: - -```bash -cd tests/e2e -PYTHONPATH=. python -m coverage_registry.collector --format json --strict -``` - -Prometheus metrics: - -```text -litellm_e2e_coverage_cells{module="",state="covered"} -litellm_e2e_coverage_cells{module="",state="total"} -litellm_e2e_coverage_percent{module=""} -litellm_e2e_coverage_orphan_markers -litellm_e2e_coverage_collection_errors -``` - -Grafana gets the trend by storing these metrics over time. No date needs to be encoded -inside the metric itself. - -## Alerts - -Start with two alerts: - -- Unknown marker count is greater than zero. -- Coverage percent for any module drops compared with the previous successful run. - -After existing collection warnings are fixed, add: - -- Collection error count is greater than zero. - -## Non-Goals - -- Do not use line coverage for this dashboard. This is behavior coverage, not source-line - coverage. -- Do not let tests invent module names. Tests only declare `@pytest.mark.covers(...)`; - the registry decides which module a cell belongs to. diff --git a/tests/e2e/coverage_registry/README.md b/tests/e2e/coverage_registry/README.md index ef94b5639c2..4177cba7766 100644 --- a/tests/e2e/coverage_registry/README.md +++ b/tests/e2e/coverage_registry/README.md @@ -3,8 +3,7 @@ This directory is the **denominator** for e2e test coverage: the set of behaviors we want covered, one row per behavior, checked into the repo so coverage is a number we can track instead of a guess. It implements the plan in the "E2E Coverage Tracking" -note; the naming grammar lives in `tests/e2e/CLAUDE.md`. The Grafana dashboard contract -lives in `GRAFANA_DASHBOARD.md`. +note; the naming grammar lives in `tests/e2e/CLAUDE.md`. ## The model