diff --git a/tests/e2e/CLAUDE.md b/tests/e2e/CLAUDE.md index f2ca2aea437..40496e5f75c 100644 --- a/tests/e2e/CLAUDE.md +++ b/tests/e2e/CLAUDE.md @@ -51,7 +51,7 @@ The shape is layered so tests stay declarative Each suite provides its own `client` fixture (see `llm_translation/passthrough_client.py`), a frozen dataclass that holds the shared `Gateway` and adds suite-specific routes. Cleanup runs through that same `Gateway`, so whatever keys or customers your test creates get torn down by the `resources` fixture -Request and response bodies are typed pydantic models in `models.py`; only the fields a test reads are modelled, and nothing passes raw dicts. Outcomes come back as a `Result[R]` tagged union (`Success`, `NetworkError`, `UnauthorizedError`, `RateLimitedError`, `ValidationError`, `UnknownApiError`). Handle them with `match`, or call `unwrap(...)` when a non-success should fail the test. The skip-vs-fail split is deliberate: a test marked `e2e` skips when no proxy answers its liveness probe, but once a request reaches the proxy any wrong behavior is a hard failure, never a skip +Request and response bodies are typed pydantic models in `models.py`; only the fields a test reads are modelled, and nothing passes raw dicts. Outcomes come back as a `Result[R]` tagged union (`Success`, `NetworkError`, `UnauthorizedError`, `RateLimitedError`, `ValidationError`, `UnknownApiError`). Handle them with `match`, or call `unwrap(...)` when a non-success should fail the test. The harness hard-fails and never skips: a test marked `e2e` fails when no proxy answers its liveness probe, and once a request reaches the proxy any wrong behavior is likewise a hard failure, so a missing proxy turns the run red instead of being mistaken for a pass Mark live tests with `@pytest.mark.e2e` (on the class or the module). Pure coverage of the harness itself carries no marker and runs regardless. Use `scoped_key` for a fresh all-models key that auto-deletes, `resources` when you need to create and tear down more than a key, and `unique_marker()` from `e2e_config` to keep prompts, tags, and customer ids from colliding across concurrent runs and the shared response cache @@ -173,7 +173,7 @@ other... ``` ## Hard Rules -- no monkeypatching, mock tests or unit tests of any kind. if a contributor asks you to write an end to end test, do NOT stage a unit test with it. if you find a product gap, call it out in the PR description +- no monkeypatching or mock tests, and never substitute a unit test for e2e feature coverage: a product feature is proven end to end against a live proxy, not with a unit test. if a contributor asks you to write an end to end test, do NOT stage a unit test of the feature with it; if you find a product gap, call it out in the PR description. tests that cover the harness itself are the exception and are allowed (for example `coverage_registry/test_collector.py`, which unit-tests the coverage collector): they carry no `e2e` marker, exercise harness plumbing rather than a product feature, and run whether or not a proxy is up - use model management endpoints to create new models for a test. this could be in a conftest / inline for each test. ask the user what they want. diff --git a/tests/e2e/CONTRIBUTING.md b/tests/e2e/CONTRIBUTING.md index 2082f2c9de4..555ac0482e2 100644 --- a/tests/e2e/CONTRIBUTING.md +++ b/tests/e2e/CONTRIBUTING.md @@ -54,7 +54,7 @@ The suites run against a live proxy, so bring one up first. `docker-compose.yml` docker compose down -v ``` -Tests marked `@pytest.mark.e2e` skip when no proxy answers `/health/liveliness`, so a run that reports everything skipped means the stack isn't up, not that anything passed +Tests marked `@pytest.mark.e2e` hard-fail when no proxy answers `/health/liveliness`, so a run that goes red with `No live proxy` at setup means the stack isn't up; they never skip for a missing proxy, so an absent stack can't be mistaken for a pass ## What a complete test looks like @@ -132,7 +132,7 @@ The shape is layered so tests stay declarative Each suite provides its own `client` fixture (see `llm_translation/passthrough_client.py`), a frozen dataclass that holds the shared `Gateway` and adds suite-specific routes. Cleanup runs through that same `Gateway`, so whatever keys or customers your test creates get torn down by the `resources` fixture -Request and response bodies are typed pydantic models in `models.py`; only the fields a test reads are modelled, and nothing passes raw dicts. Outcomes come back as a `Result[R]` tagged union (`Success`, `NetworkError`, `UnauthorizedError`, `RateLimitedError`, `ValidationError`, `UnknownApiError`). Handle them with `match`, or call `unwrap(...)` when a non-success should fail the test. The skip-vs-fail split is deliberate: a test marked `e2e` skips when no proxy answers its liveness probe, but once a request reaches the proxy any wrong behavior is a hard failure, never a skip +Request and response bodies are typed pydantic models in `models.py`; only the fields a test reads are modelled, and nothing passes raw dicts. Outcomes come back as a `Result[R]` tagged union (`Success`, `NetworkError`, `UnauthorizedError`, `RateLimitedError`, `ValidationError`, `UnknownApiError`). Handle them with `match`, or call `unwrap(...)` when a non-success should fail the test. The harness hard-fails and never skips: a test marked `e2e` fails when no proxy answers its liveness probe, and once a request reaches the proxy any wrong behavior is likewise a hard failure, so a missing proxy turns the run red instead of being mistaken for a pass Mark live tests with `@pytest.mark.e2e` (on the class or the module). Pure coverage of the harness itself carries no marker and runs regardless. Use `scoped_key` for a fresh all-models key that auto-deletes, `resources` when you need to create and tear down more than a key, and `unique_marker()` from `e2e_config` to keep prompts, tags, and customer ids from colliding across concurrent runs and the shared response cache diff --git a/tests/e2e/access_control/conftest.py b/tests/e2e/access_control/conftest.py index 9f4a00fe06f..b5681ff76ad 100644 --- a/tests/e2e/access_control/conftest.py +++ b/tests/e2e/access_control/conftest.py @@ -1,4 +1,4 @@ -"""Access-control suite client fixture; lifecycle/skip/marker live in the parent conftest.""" +"""Access-control suite client fixture; lifecycle/liveness gate/marker live in the parent conftest.""" import pytest diff --git a/tests/e2e/batches/conftest.py b/tests/e2e/batches/conftest.py index 2c6070c437a..d3b6d42bc24 100644 --- a/tests/e2e/batches/conftest.py +++ b/tests/e2e/batches/conftest.py @@ -1,6 +1,6 @@ """Batches suite's `client` fixture. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. BatchClient holds the shared Gateway, so the `resources` fixture cleans up keys through it; tests register file deletes and batch cancels via `resources.defer(...)`. diff --git a/tests/e2e/llm_translation/conftest.py b/tests/e2e/llm_translation/conftest.py index 2a87ef7259d..5258b751a8c 100644 --- a/tests/e2e/llm_translation/conftest.py +++ b/tests/e2e/llm_translation/conftest.py @@ -1,6 +1,6 @@ """LLM-translation suite's `client` fixture. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. PassthroughClient holds the shared Gateway, so the `resources` fixture cleans up keys this suite creates. """ diff --git a/tests/e2e/llm_translation/realtime/REALTIME_COVERAGE_MATRIX.md b/tests/e2e/llm_translation/realtime/REALTIME_COVERAGE_MATRIX.md index 4795c3b9f54..bae858d50af 100644 --- a/tests/e2e/llm_translation/realtime/REALTIME_COVERAGE_MATRIX.md +++ b/tests/e2e/llm_translation/realtime/REALTIME_COVERAGE_MATRIX.md @@ -49,10 +49,10 @@ kept commented out in `PROVIDERS` until they pass end-to-end here; re-enable the uncommenting their entry. Every provider is provisioned and asserted; the suite never skips a provider. Per -`tests/e2e/CLAUDE.md` the only sanctioned skip is the whole-suite proxy-liveness -skip, so a provider whose credentials or upstream realtime model are missing on the -gateway is a hard failure, not a skip. Give the gateway each provider's credentials -to turn its tests green. +`tests/e2e/CLAUDE.md` there is no sanctioned skip: the whole-suite proxy-liveness +probe hard-fails when no proxy answers, and a provider whose credentials or upstream +realtime model are missing on the gateway is likewise a hard failure, not a skip. +Give the gateway each provider's credentials to turn its tests green. ## Running @@ -63,5 +63,5 @@ the deployments itself), then uv run pytest tests/e2e/llm_translation/realtime/ -v ``` -The whole suite skips only when no proxy answers `GET /health/liveliness` at +The whole suite hard-fails at setup when no proxy answers `GET /health/liveliness` at `LITELLM_PROXY_URL` (default `http://localhost:4000`). diff --git a/tests/e2e/llm_translation/realtime/conftest.py b/tests/e2e/llm_translation/realtime/conftest.py index 15cd789664e..8e6e596bcd3 100644 --- a/tests/e2e/llm_translation/realtime/conftest.py +++ b/tests/e2e/llm_translation/realtime/conftest.py @@ -1,6 +1,6 @@ """Realtime suite's `client` and `realtime_models` fixtures. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. RealtimeClient holds the shared Gateway, so the `resources` fixture cleans up keys this suite creates. diff --git a/tests/e2e/llm_translation/realtime/test_realtime_e2e.py b/tests/e2e/llm_translation/realtime/test_realtime_e2e.py index 6aaffdd208e..f99fa8d86b3 100644 --- a/tests/e2e/llm_translation/realtime/test_realtime_e2e.py +++ b/tests/e2e/llm_translation/realtime/test_realtime_e2e.py @@ -6,10 +6,10 @@ schema: the session lifecycle, the canonical response event sequence with a reconstructed transcript and usage, and a full tool-call round-trip (call -> tool result -> a follow-up response that uses the result). -One GA-speaking client validates every provider; only the model alias changes. A -provider whose realtime alias is not configured on the proxy skips (skip on -environment); once it is configured, a protocol failure is a hard failure. See -REALTIME_COVERAGE_MATRIX.md. +One GA-speaking client validates every provider; only the model alias changes. +Every provider is provisioned at session start, so a missing realtime alias is a +hard failure, not a skip; once configured, a protocol failure is likewise a hard +failure. See REALTIME_COVERAGE_MATRIX.md. """ import pytest diff --git a/tests/e2e/llm_translation/test_ocr_rust_e2e.py b/tests/e2e/llm_translation/test_ocr_rust_e2e.py index 921010e5eae..e735d9c01b5 100644 --- a/tests/e2e/llm_translation/test_ocr_rust_e2e.py +++ b/tests/e2e/llm_translation/test_ocr_rust_e2e.py @@ -7,9 +7,9 @@ references the proxy resolves at call time, so adding a provider is a new type rather than another inline body. Start the proxy with the Rust OCR path enabled: Each case creates its deployment, drives a real /v1/ocr call, and asserts a -well-formed OCR document comes back. Per the e2e "skip on environment, fail on -behavior" rule, a case skips when no proxy answers but fails (never skips) once a -request reaches it: the proxy fetches each provider's referenced secrets, so a +well-formed OCR document comes back. Per the e2e hard-fail contract, a case +fails when no proxy answers and also fails once a request reaches it: the proxy +fetches each provider's referenced secrets, so a missing credential surfaces as a live provider error rather than silent green. """ diff --git a/tests/e2e/management/conftest.py b/tests/e2e/management/conftest.py index 18da1305c13..4f2dc874a33 100644 --- a/tests/e2e/management/conftest.py +++ b/tests/e2e/management/conftest.py @@ -1,6 +1,6 @@ """Management suite fixtures: the client plus a logged-in dashboard page. -Lifecycle/skip/marker live in the parent conftest. The browser fixtures drive +Lifecycle/liveness gate/marker live in the parent conftest. The browser fixtures drive the dashboard the proxy serves at /ui, so browser tests exercise exactly what an end user sees. playwright is an optional dependency loaded behind importorskip inside the fixture, so the API tests in this suite collect and run without it: diff --git a/tests/e2e/quota_management/budgets/conftest.py b/tests/e2e/quota_management/budgets/conftest.py index 236822f4309..4299d2ffd49 100644 --- a/tests/e2e/quota_management/budgets/conftest.py +++ b/tests/e2e/quota_management/budgets/conftest.py @@ -1,6 +1,6 @@ """Budgets suite's `client` fixture. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. BudgetClient holds the shared Gateway, so the `resources` fixture cleans up keys through it; tests register entity deletes via `resources.defer(...)`. diff --git a/tests/e2e/quota_management/ratelimit/conftest.py b/tests/e2e/quota_management/ratelimit/conftest.py index 4a5a73bb5e4..59dee5e65b3 100644 --- a/tests/e2e/quota_management/ratelimit/conftest.py +++ b/tests/e2e/quota_management/ratelimit/conftest.py @@ -1,6 +1,6 @@ """Quota-management suite's `client` fixture. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. QuotaClient holds the shared Gateway, so the `resources` fixture cleans up keys through it. """ diff --git a/tests/e2e/quota_management/spend_tracking/SPEND_TRACKING_COVERAGE_MATRIX.md b/tests/e2e/quota_management/spend_tracking/SPEND_TRACKING_COVERAGE_MATRIX.md index 062ef8d73da..6baebc4c28c 100644 --- a/tests/e2e/quota_management/spend_tracking/SPEND_TRACKING_COVERAGE_MATRIX.md +++ b/tests/e2e/quota_management/spend_tracking/SPEND_TRACKING_COVERAGE_MATRIX.md @@ -80,5 +80,5 @@ proxy + SpendLogs rows. Status: `covered` / `partial` / `gap`. `proxy_batch_write_at` (~60s) means rows land late; every read polls to a deadline. Fresh scoped key per test (isolation, xdist-safe, cleaned up). Assert invariants (`spend > 0`, `total == prompt + completion`, aggregate == sum), not literal -$/token values, so pricing drift is not a failure. Skip on environment (no proxy / -no provider key), fail on behavior (a real 2xx call with a wrong/missing row). +$/token values, so pricing drift is not a failure. Hard-fail when no proxy +answers, fail on behavior (a real 2xx call with a wrong/missing row). diff --git a/tests/e2e/quota_management/spend_tracking/conftest.py b/tests/e2e/quota_management/spend_tracking/conftest.py index 0e80764236b..434af15b182 100644 --- a/tests/e2e/quota_management/spend_tracking/conftest.py +++ b/tests/e2e/quota_management/spend_tracking/conftest.py @@ -1,6 +1,6 @@ """Spend-tracking suite's `client` fixture and driver-model registration. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. SpendClient exposes the shared Gateway (GatewayProvider), so the `resources` fixture cleans up keys and customers this suite creates. diff --git a/tests/e2e/router/conftest.py b/tests/e2e/router/conftest.py index 046cdd80c2b..344d8ab5c13 100644 --- a/tests/e2e/router/conftest.py +++ b/tests/e2e/router/conftest.py @@ -1,6 +1,6 @@ """Router suite's `client` fixture. -The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker +The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker live in the parent tests/e2e/conftest.py. ComplexityRouterClient holds the shared Gateway, so the `resources` fixture cleans up keys this suite creates.