litellm/tests/e2e
mubashir1osmani 98765f65af
feat(e2e): emit structured E2E_RESULT lines for package status history (#33578)
* feat(e2e): emit structured E2E_RESULT lines for package status history

Pytest progress logs only expose file basenames and collapse multi-test files
into one status-history row. Emit one logfmt E2E_RESULT per finished node with
package, file, outcome, duration_ms, node_id, and covers so Grafana can roll up
by package (stable cardinality) and drill down by node_id in Explore

* test(e2e): drop unit tests from the live e2e tree

tests/e2e is for live proxy suites only. Remove harness, coverage-registry,
and claude_code unit trees so the e2e run does not collect them

* fix(e2e): type E2E_RESULT hook for basedpyright zero-error gate

Protocol-typed covers extraction and pluggy Result typing on the
makereport hook so tests/e2e stays under the e2e basedpyright ceiling

* fix(e2e): drop unused xfailed/xpassed from E2E_RESULT Outcome

We never emit those states; xfail collapses to skipped/passed via pytest
report flags. Keep the literal honest so the dashboard only sees real outcomes

* fix(e2e): strip tests/e2e prefix when deriving E2E_RESULT package

Repo-root pytest nodeids are tests/e2e/<suite>/...; without stripping,
every line would package=tests and status history would be useless

* fix(e2e): use pytest wrapper=True instead of deprecated hookwrapper

pytest 8.1+ deprecates hookwrapper; yield returns the TestReport directly
so we return it for the outer chain and drop pluggy.Result

* fix(e2e): import e2e_result_reporter at module load

Surface a missing module as a collection-time ImportError instead of a
per-test hook failure mid-run

* test(e2e): restore coverage_registry/test_collector.py

Needed to validate registry coverage math and the checked-in cell
denominator; not a live proxy suite
2026-07-16 15:01:01 -07:00
..
access_control test(e2e): migrate access-control and inference-endpoint regression tests (#32016) 2026-07-05 01:39:10 +00:00
batches fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265) 2026-07-14 13:32:10 -07:00
claude_code feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00
coverage_registry feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00
grafana feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00
llm_translation test(e2e): scope virtual keys to the deployment under test 2026-07-14 17:12:15 -07:00
logging chore(e2e): establish litellm_e2e_staging integration line (#33502) 2026-07-16 12:09:24 -07:00
management chore(e2e): establish litellm_e2e_staging integration line (#33502) 2026-07-16 12:09:24 -07:00
quota_management refactor: make the code easier to read 2026-07-14 13:58:23 -07:00
router fix(complexity_router): return empty dict from _classifier_call_metadata when metadata is absent (#33452) 2026-07-15 17:46:00 -07:00
bob_the_builder.py test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
CLAUDE.md feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00
conftest.py feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00
CONTRIBUTING.md ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
docker-compose.yml fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
e2e_config.py test(e2e): datadog log delivery for successful chat, messages, and responses (LIT-4447) (#33415) 2026-07-16 09:54:07 -07:00
e2e_gateway.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
e2e_http.py test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) 2026-07-14 20:23:54 -07:00
e2e_result_reporter.py feat(e2e): emit structured E2E_RESULT lines for package status history (#33578) 2026-07-16 15:01:01 -07:00
lifecycle.py test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
models.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
pytest.ini refactor(e2e): move budgets and spend_tracking suites under quota_management 2026-07-11 16:14:21 -07:00
transport.py test(e2e): probe the full spend read surface including schema-hidden routes (#32267) 2026-07-06 14:02:05 -07:00