test(e2e): drop the _e2e suffix from the perf suite files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
devin-ai-integration[bot] 2026-10-12 22:06:12 +00:00
parent 28ad7ff50b
commit eba49f2e68
6 changed files with 3 additions and 3 deletions

View file

@ -96,7 +96,7 @@ jobs:
REDIS_HOST: 127.0.0.1
REDIS_PORT: "6379"
run: |
uv run --no-sync pytest tests/e2e/perf/test_redis_chaos_e2e.py -v --tb=short -rA -s
uv run --no-sync pytest tests/e2e/perf/test_redis_chaos.py -v --tb=short -rA -s
- name: Show proxy log on failure
if: failure()

View file

@ -78,7 +78,7 @@ jobs:
env:
E2E_WEEKLY_ANOMALY: "1"
run: |
uv run --no-sync pytest tests/e2e/perf/test_weekly_session_anomaly_e2e.py -v --tb=short -rA
uv run --no-sync pytest tests/e2e/perf/test_weekly_session_anomaly.py -v --tb=short -rA
- name: Show proxy log on failure
if: failure()

View file

@ -44,7 +44,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family
- `logging/` - logging-integration delivery (datadog and friends)
- `security/` - secret handling and log-leak protection
- `router/` - routing and reliability behavior (fallbacks, cooldowns) plus the memory tests (`test_reliability_memory_e2e.py`: every worker's RSS as read at collection time, before any test traffic, must sit under a fixed idle budget, the release-gate check for a DB-backed boot that idles near the pod limit the way v1.100.x did, and a few hundred failing requests with retries and fallbacks must not grow proxy RSS past a fixed budget nor store a request snapshot past a fixed size, the release-gate check for the v1.100.0 retry-breadcrumb leak)
- `perf/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163). Do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly_e2e.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend, marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos_e2e.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see), needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_PERF_TESTS` is set and excluded from the per-PR selector like the rest of `perf/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in). The perf suite proper is the `perf`-marked tests (`test_perf_*_e2e.py`), each a few lines on the `perf_rig` fixture (`perf_rig.py`): `mocked_group` registers a model group over `mock_response` deployments plus a key pool (zero provider spend, torn down by `ResourceManager`), `measure` runs Locust through a discarded warm-up and a kept measured phase and reads every worker's RSS from `/debug/memory/summary` before and after, and `problems` returns what the run broke in one `PerfBudget` (rps floor, p50/p90/p99 ceilings, worst-worker RSS growth ceiling, zero failures, Locust not saturated), which the test asserts empty, and prints the numbers to the build log. The proxy for those runs boots from one of two configs, `gateway/perf_redis_otel_ci_config.yml` (Redis for shared state, spend written to Postgres by every worker, prometheus and otel) and `gateway/perf_redis_buffer_otel_ci_config.yml` (the same plus `use_redis_transaction_buffer` with `proxy_batch_write_at: 60`, `prometheus_system`, `json_logs` and `store_audit_logs`), which differ only in how spend reaches Postgres so a red on one side points at the spend-write path
- `perf/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163). Do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend, marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see), needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_PERF_TESTS` is set and excluded from the per-PR selector like the rest of `perf/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in). The perf suite proper is the `perf`-marked tests (`test_1k_rps.py` and whatever follows it, no `_e2e` suffix in this folder), each a few lines on the `perf_rig` fixture (`perf_rig.py`): `mocked_group` registers a model group over `mock_response` deployments plus a key pool (zero provider spend, torn down by `ResourceManager`), `measure` runs Locust through a discarded warm-up and a kept measured phase and reads every worker's RSS from `/debug/memory/summary` before and after, and `problems` returns what the run broke in one `PerfBudget` (rps floor, p50/p90/p99 ceilings, worst-worker RSS growth ceiling, zero failures, Locust not saturated), which the test asserts empty, and prints the numbers to the build log. The proxy for those runs boots from one of two configs, `gateway/perf_redis_otel_ci_config.yml` (Redis for shared state, spend written to Postgres by every worker, prometheus and otel) and `gateway/perf_redis_buffer_otel_ci_config.yml` (the same plus `use_redis_transaction_buffer` with `proxy_batch_write_at: 60`, `prometheus_system`, `json_logs` and `store_audit_logs`), which differ only in how spend reaches Postgres so a red on one side points at the spend-write path
- `other/` - the holding-pen suite for the `other.*` registry cluster with no home of its own yet: the master-key auth gate, JWT auth (access tokens issued by a real Keycloak realm, `idp.py` plus `idp_realm.json`, whose JWKS the proxy's `JWT_PUBLIC_KEY_URL` points at, see CONTRIBUTING.md for the start command and config block), and the process-lifecycle health probes (liveness, public readiness, authenticated readiness diagnostics). Promote a cluster out once it is large/stable enough for its own suite
- `secret_manager/` - the gateway's `key_management_system` against a real secret manager: deployment keys resolved from it (`os.environ/<name>` where the name exists only in the manager) and virtual keys written to and deleted from it. The tests are backend-agnostic and each backend is its own lane, because the setting is global to the proxy: `E2E_SECRET_MANAGER=<system>` opts in and picks the backend from `secret_backends.BACKENDS`, the proxy is booted from `gateway/secret_manager_<system>_ci_config.yml` against the live manager, and the tests reach that manager through the backend's `SecretStore` (`secret_store_<system>.py`). A test needing something not every backend does carries `requires_capability(...)` and is deselected on lanes that lack it. `secret_manager/backend.sh up <system>` runs a backend in Docker and writes the proxy's and the tests' env. Marked `secret_manager`, deselected unless `E2E_SECRET_MANAGER` is set, and kept out of the per-PR selector. Backends today: `hashicorp_vault` and `cyberark` (CyberArk Conjur, which cannot delete, so the delete test is Vault-only)
- `gateway/` - proxy configuration only (`litellm-config.yml`). No tests