Commit graph

5 commits

Author SHA1 Message Date
mubashir1osmani
98765f65af
feat(e2e): emit structured E2E_RESULT lines for package status history (#33578)
* feat(e2e): emit structured E2E_RESULT lines for package status history

Pytest progress logs only expose file basenames and collapse multi-test files
into one status-history row. Emit one logfmt E2E_RESULT per finished node with
package, file, outcome, duration_ms, node_id, and covers so Grafana can roll up
by package (stable cardinality) and drill down by node_id in Explore

* test(e2e): drop unit tests from the live e2e tree

tests/e2e is for live proxy suites only. Remove harness, coverage-registry,
and claude_code unit trees so the e2e run does not collect them

* fix(e2e): type E2E_RESULT hook for basedpyright zero-error gate

Protocol-typed covers extraction and pluggy Result typing on the
makereport hook so tests/e2e stays under the e2e basedpyright ceiling

* fix(e2e): drop unused xfailed/xpassed from E2E_RESULT Outcome

We never emit those states; xfail collapses to skipped/passed via pytest
report flags. Keep the literal honest so the dashboard only sees real outcomes

* fix(e2e): strip tests/e2e prefix when deriving E2E_RESULT package

Repo-root pytest nodeids are tests/e2e/<suite>/...; without stripping,
every line would package=tests and status history would be useless

* fix(e2e): use pytest wrapper=True instead of deprecated hookwrapper

pytest 8.1+ deprecates hookwrapper; yield returns the TestReport directly
so we return it for the outer chain and drop pluggy.Result

* fix(e2e): import e2e_result_reporter at module load

Surface a missing module as a collection-time ImportError instead of a
per-test hook failure mid-run

* test(e2e): restore coverage_registry/test_collector.py

Needed to validate registry coverage math and the checked-in cell
denominator; not a live proxy suite
2026-07-16 15:01:01 -07:00
mateo-berri
320a55f01f refactor(e2e): bucket rate limits, budgets, and spend tracking under quota_management 2026-07-11 16:13:48 -07:00
ishaan-berri
ad69d6f3f9
test: emit e2e coverage lines for loki (#32513) 2026-07-08 11:06:56 -07:00
ishaan-berri
3ea27bd64c
test: add e2e coverage module metrics (#32403)
* Split LLM e2e coverage modules

* Add e2e coverage dashboard metrics

* Remove dashboard brief from e2e coverage PR
2026-07-07 19:38:56 -07:00
yuneng-jiang
a43f128a74
test(e2e): add coverage registry and collector (#32304)
Introduce the e2e coverage denominator: 282 behavior cells across the six
tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance,
Logging & Guardrails, Other), one validated YAML row each, plus a collector
that diffs the registry against @pytest.mark.covers markers and reports
coverage per module.

The registry rows validate against a pydantic discriminated union so a row
cannot carry a field from another module. The collector is static: a
collect-only pass reads the markers, so it runs no test and needs no live
proxy. Register the covers marker suite-wide so that pass works under
--strict-markers.

This is a draft for review. Tiers are proposed rather than signed off, and a
few cells still need a support check or a prune.
2026-07-07 15:51:20 -04:00