2717b5f597
* test(e2e): rename the load suite to perf, one E2E_PERF_TESTS opt-in, and a mixed-provider Locust perf test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): drop the mixed-provider perf test, keep the rename and the E2E_PERF_TESTS consolidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2.4 KiB
e2e harness tests
Tests of the harness under tests/e2e/ (the transport, the clients, the fixture bundle and replay edge, the stack lock, the IdP launcher, the coverage collector, the JUnit properties and the Claude Code driver), not of the product. They live outside tests/e2e/ because the Buildkite e2e run copies that folder into the runner image and runs every test in it, so a harness test in there counts as a product test in the nightly numbers. Nothing here needs a proxy, provider keys or the network
The layout mirrors tests/e2e/: test_e2e_http.py covers tests/e2e/e2e_http.py, logging/test_datadog_reader.py covers tests/e2e/logging/datadog_reader.py, and claude_code/ covers the driver, builder, probe and version resolver. Put a new harness test under the folder that mirrors the suite folder whose module it covers
Run them from the repo root. e2e_config reads LITELLM_MASTER_KEY at import and any value will do, the CI lane sets a dummy:
LITELLM_MASTER_KEY=sk-harness uv run pytest tests/e2e_harness
pytest.ini here puts tests/e2e and the suite folders whose modules are under test on the path, so imports look exactly as they do inside the suite (from e2e_http import ..., from batch_cleanup import ...). claude_code/test_request_determinism.py drives the real claude CLI. Deselect it with -m "not cli_determinism" when the CLI is not installed
Rules: no e2e marker and no @meta, since nothing here drives the proxy. @pytest.mark.covers only where the test proves the collector or the JUnit properties read it. Inputs via arguments or env vars (setting an env var through pytest's monkeypatch fixture is fine, patching a function, class or module is not). And the same typing bar as the suite, make lint-e2e-basedpyright covers this folder and allows zero errors. The raw HTTP client ban (tests/code_coverage_tests/check_e2e_no_raw_requests.py) applies here too
CI: the python job in .github/workflows/test-linting.yml runs this folder whenever anything under tests/e2e/ (except ui/) or tests/e2e_harness/ changes, with the claude CLI installed. It also runs tests/code_coverage_tests/test_provider_replay_harness.py and tests/code_coverage_tests/test_provider_cache.py with --noconftest -o addopts= -o "pythonpath=tests/e2e tests/e2e_harness" -p no:rerunfailures; the replay test imports helpers from test_provider_edge.py