b18188dee2
* test(e2e): retire the quiet_stack lock quiet_stack held an exclusive host-wide lock for six tests, draining every other xdist worker while they ran. On the full e2e lane that left 35-38% of worker time idle. Each of those tests now lives where it can run without pausing the suite - the two memory tests move to perf/ behind the perf marker, so the perf lane runs them on its own fresh stack. Their refusing deployments use a literal key because the perf stack carries no provider keys - the stored-request size check from the memory test becomes an integration test on an owned proxy that stores prompts - the discount and margin spend-row tests are dropped from e2e. Their breakdown assertions join the existing integration pricing test, which already owns the discount and margin repricing flow - the Bedrock guardrail cost test stays in e2e without the pricing reset fixture, since nothing else on the stack writes pricing config now - the MCP semantic tool filter test turns on a proxy-wide setting, so it gets a new proxy_wide_config opt-in marker and runs after the main run * test(e2e): register the claude_code deployments as one batch The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
2.5 KiB
e2e harness tests
Tests of the harness under tests/e2e/ (the transport, the clients, the fixture bundle and replay edge, the IdP launcher, the coverage collector, the JUnit properties and the Claude Code driver), not of the product. They live outside tests/e2e/ because the Buildkite e2e run copies that folder into the runner image and runs every test in it, so a harness test in there counts as a product test in the nightly numbers. Nothing here needs a proxy, provider keys or the network
The layout mirrors tests/e2e/: test_e2e_http.py covers tests/e2e/e2e_http.py, logging/test_datadog_reader.py covers tests/e2e/logging/datadog_reader.py, claude_code/ covers the driver, builder and probe, cli_harness/ covers the version resolver, the CLI env lockdown and the rate limiter, and codex/ covers how the Codex driver reads the CLI's JSONL events and the compactions in a thread's rollout file. Put a new harness test under the folder that mirrors the suite folder whose module it covers
Run them from the repo root. e2e_config reads LITELLM_MASTER_KEY at import and any value will do, the CI lane sets a dummy:
LITELLM_MASTER_KEY=sk-harness uv run pytest tests/e2e_harness
pytest.ini here puts tests/e2e and the suite folders whose modules are under test on the path, so imports look exactly as they do inside the suite (from e2e_http import ..., from batch_cleanup import ...). claude_code/test_request_determinism.py drives the real claude CLI. Deselect it with -m "not cli_determinism" when the CLI is not installed
Rules: no e2e marker and no @meta, since nothing here drives the proxy. @pytest.mark.covers only where the test proves the collector or the JUnit properties read it. Inputs via arguments or env vars (setting an env var through pytest's monkeypatch fixture is fine, patching a function, class or module is not). And the same typing bar as the suite, make lint-e2e-basedpyright covers this folder and allows zero errors. The raw HTTP client ban (tests/code_coverage_tests/check_e2e_no_raw_requests.py) applies here too
CI: the python job in .github/workflows/test-linting.yml runs this folder whenever anything under tests/e2e/ (except ui/) or tests/e2e_harness/ changes, with the claude CLI installed. It also runs tests/code_coverage_tests/test_provider_replay_harness.py and tests/code_coverage_tests/test_provider_cache.py with --noconftest -o addopts= -o "pythonpath=tests/e2e tests/e2e_harness" -p no:rerunfailures; the replay test imports helpers from test_provider_edge.py