diff --git a/tests/e2e/AGENTS.md b/tests/e2e/AGENTS.md index d267059cdc7..62ebfd12fe5 100644 --- a/tests/e2e/AGENTS.md +++ b/tests/e2e/AGENTS.md @@ -43,7 +43,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family - `mcp/` - the MCP server surface over api_key auth against the real Datadog remote MCP server (see "MCP suite: real Datadog only" below), plus the gateway-managed OAuth (authorization_code) path exercised through `/chat/completions` in `test_mcp_chat_completion_oauth_e2e.py` and direct MCP protocol operations in `test_mcp_oauth_happy_path_e2e.py`, the one behavior Datadog's static-header auth cannot reach, seeding the per-user upstream token via the interactive authorize dance driven with the mcp SDK's own OAuth client (headless-browser consent from a saved session) and asserting the completion or protocol call lists and executes the server's tools with the stored per-user token - `logging/` - logging-integration delivery (datadog and friends) - `security/` - secret handling and log-leak protection -- `router/` - routing and reliability behavior (fallbacks, cooldowns) plus the memory tests (`test_reliability_memory_e2e.py`: every worker's RSS as read at collection time, before any test traffic, must sit under a fixed idle budget, the release-gate check for a DB-backed boot that idles near the pod limit the way v1.100.x did, and a few hundred failing requests with retries and fallbacks must not grow proxy RSS past a fixed budget nor store a request snapshot past a fixed size, the release-gate check for the v1.100.0 retry-breadcrumb leak) +- `router/` - routing and reliability behavior (fallbacks, cooldowns) plus the memory tests (`test_reliability_memory_e2e.py`: every worker's RSS as read at collection time, before any test traffic, must sit under a fixed idle budget, the release-gate check for a DB-backed boot that idles near the pod limit the way v1.100.x did, and a few hundred failing requests with retries and fallbacks must not grow proxy RSS past a loose ceiling over the measured batches nor store a request snapshot past a fixed size, the release-gate check for the v1.100.0 retry-breadcrumb leak) - `perf/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163). Do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly_e2e.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend, marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos_e2e.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see), needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_PERF_TESTS` is set and excluded from the per-PR selector like the rest of `perf/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in). New perf tests go here too, marked `perf`, behind the same `E2E_PERF_TESTS` opt-in. The perf cases (`test_perf_cases.py`, one `PerformanceTestCase` per module in `perf/cases/`, no `_e2e` suffix) are the request-performance runs. A case names what the gateway was deployed with (model groups and guardrails, written out in the gateway's config: the `proxy_config` block of project-releaser's perf helm values in the lane, `gateway/perf_ci_config.yml` locally, with each deployment's mock knobs in an `x-mock-knobs` extra header so the model ids are real ones like `gpt-6-luna` and `anthropic.claude-sonnet-5`) and lists what the test creates and sends: teams, Keycloak JWT users, virtual keys, `RequestShape`s (a coding-agent session from `perf/setup/corpus/` whose input-token size is drawn per request from the shape's range, with or without tools, and the output cap), a `Traffic` plan (seed, rps, request count, endpoint, shape and stream shares) and `Limit`s (one metric at one percentile under one ceiling). The infra side lives in `perf/setup/`: `provision.py` checks the deployed config matches the case and builds the callers, `plan.py` draws the request list from the seed, `run.py` replays it from a separate Locust process (`locustfile.py`) that builds each body right before sending and records status, header, first-byte, first-token and total time per request, and `observe.py` joins all of that with what the mocks received and the workers' RSS to score every `Limit`. Limits are about what the caller sees and what the gateway costs: every non-blocked request is a 200 (`unexpected_status_count`), the seconds the gateway added per request at p50 and p99 (`gateway_added_seconds`, to the first token on streams), the client's time to the first token of a stream at p50 and p99 (`time_to_first_token_seconds`, mock delay included), worker memory growth (`memory_growth_megabytes`), and the Postgres statements and Redis commands a request waits on before it is forwarded (`postgres_calls_per_request`, `redis_calls_per_request`: auth and routing reads, the rate-limiter script), with the work the gateway does after the response or on a timer (spend writes, daily aggregates, config reloads) reported separately as `postgres_background_calls_per_request` and `redis_background_calls_per_request`; `datastore.py` reads `pg_stat_statements` before and after the run and holds a Redis `MONITOR` connection during it, sorts each statement by verb and table and each command by key, and the run output lists the top statements and commands per bucket so a red run names what grew. Whether a body, a guardrail call or a spend row is right is printed as a diagnostic, never a limit, that is the functional suites' job. The providers and the guardrail are out-of-process servers in `perf/setup/mocks/` (`server.py openai|anthropic|bedrock|guardrail`, each answering with a reply generated for the request in that provider's wire format, sized from the prompt and output cap, a tool call when tools were offered, waiting the case's `mock_latency` first, request logs at `/mock/requests` and `/mock/calls`), reached through `E2E_PERF_MOCK_*_URL` by both the gateway config and the test. Adding a perf test is adding a module with a `CASE` to `perf/cases/`; a new deployment or guardrail is a config change; a new metric is a name in `Metric` plus a branch in `observe.observe` - `other/` - the holding-pen suite for the `other.*` registry cluster with no home of its own yet: the master-key auth gate, JWT auth (access tokens issued by a real Keycloak realm, `idp.py` plus `idp_realm.json`, whose JWKS the proxy's `JWT_PUBLIC_KEY_URL` points at, see CONTRIBUTING.md for the start command and config block), and the process-lifecycle health probes (liveness, public readiness, authenticated readiness diagnostics). Promote a cluster out once it is large/stable enough for its own suite - `secret_manager/` - the gateway's `key_management_system` against a real secret manager: deployment keys resolved from it (`os.environ/` where the name exists only in the manager) and virtual keys written to and deleted from it. The tests are backend-agnostic and each backend is its own lane, because the setting is global to the proxy: `E2E_SECRET_MANAGER=` opts in and picks the backend from `secret_backends.BACKENDS`, the proxy is booted from `gateway/secret_manager__ci_config.yml` against the live manager, and the tests reach that manager through the backend's `SecretStore` (`secret_store_.py`). A test needing something not every backend does carries `requires_capability(...)` and is deselected on lanes that lack it. `secret_manager/backend.sh up ` runs a backend in Docker and writes the proxy's and the tests' env. Marked `secret_manager`, deselected unless `E2E_SECRET_MANAGER` is set, and kept out of the per-PR selector. Backends today: `hashicorp_vault` and `cyberark` (CyberArk Conjur, which cannot delete, so the delete test is Vault-only) diff --git a/tests/e2e/e2e_config.py b/tests/e2e/e2e_config.py index 8fc7e50d95c..efbc95966c4 100644 --- a/tests/e2e/e2e_config.py +++ b/tests/e2e/e2e_config.py @@ -237,7 +237,7 @@ MEMORY_TRANSCRIPT_TURNS = int(os.environ.get("E2E_MEMORY_TRANSCRIPT_TURNS", "40" MEMORY_CONCURRENCY = int(os.environ.get("E2E_MEMORY_CONCURRENCY", "4")) MEMORY_RSS_SETTLE_SAMPLES = int(os.environ.get("E2E_MEMORY_RSS_SETTLE_SAMPLES", "15")) MEMORY_RSS_SAMPLE_INTERVAL_SECONDS = float(os.environ.get("E2E_MEMORY_RSS_SAMPLE_INTERVAL_SECONDS", "1")) -MEMORY_RSS_BUDGET_MB = float(os.environ.get("E2E_MEMORY_RSS_BUDGET_MB", "48")) +MEMORY_RSS_CEILING_MB = float(os.environ.get("E2E_MEMORY_RSS_CEILING_MB", "200")) MEMORY_IDLE_RSS_BUDGET_MB = float(os.environ.get("E2E_MEMORY_IDLE_RSS_BUDGET_MB", "768")) MEMORY_STORED_REQUEST_BUDGET_KB = float(os.environ.get("E2E_MEMORY_STORED_REQUEST_BUDGET_KB", "64")) diff --git a/tests/e2e/router/test_reliability_memory_e2e.py b/tests/e2e/router/test_reliability_memory_e2e.py index 196784b5345..fb114092303 100644 --- a/tests/e2e/router/test_reliability_memory_e2e.py +++ b/tests/e2e/router/test_reliability_memory_e2e.py @@ -1,5 +1,5 @@ """Live e2e: a few hundred requests that fail before any provider answers must not -grow the proxy's resident memory past a fixed budget once the proxy is warm. +grow the proxy's resident memory past a loose ceiling once the proxy is warm. The regression this guards shipped in v1.100.0: every retry breadcrumb copied the whole request and the copies nested into one router-global list, so a proxy under @@ -13,26 +13,27 @@ deployment that fails a handful of times in a row is benched (a bad-credential 4 included), the router answers "No deployments available" without retrying, and the retry loop that leaks stops running; cooldown_time 0 keeps it running. -Three identical batches run back to back. The first two are the warmup that grows -the proxy's caches and allocator arenas to their steady state, the third is the one -the budget applies to, so a healthy proxy shows the third batch adding roughly -nothing while a leaking one adds a fixed amount per request. One warmup batch is not -enough: each batch's in-flight retries peak well above the live heap, the allocator -keeps that high-water mark, and on a cold worker the second batch still set a new one -(about 50 MB of RSS with the live Python heap flat to within 1 MB), where the later -batches each settled under 25 MB. RSS is read through /debug/memory/summary on every +Four identical batches run back to back. The first two are the warmup that grows +the proxy's caches and allocator arenas, the last two are the measured window, and +each worker's RSS may grow at most a loose ceiling over that window. RSS cannot be +held to a tight per-batch budget: on the x86 release gate (rc/1.107.0 and v1.105.1 +on isolated stacks, 28 batches each) the allocator kept setting new high-water marks +for the first several batches, about 60 MB on the third batch of a fresh stack and +more after the rest of the suite, with the gc-tracked object count flat to 0.1%, and +only settled to a couple of MB per batch after twenty. So the ceiling catches a gross +leak and nothing slower. RSS is read through /debug/memory/summary on every configured replica; a burst of failing calls leaves a transient bulge of garbage that gc reclaims within seconds, so each checkpoint samples until no new worker has answered for a settle window and keeps the lowest reading per worker. A checkpoint runs before the warmup and after every batch, and each batch's settled growth per worker rides out as a repeated `rss_batch_growth` JUnit property as soon as that -batch settles, so a run that fails keeps the curve up to the batch it failed in. The budget is judged per worker -(by replica address, hostname and pid, since pods in their own pid namespaces report -the same pids) so each worker is compared with itself, and the checkpoints around the -measured batch must see the same workers: a single load-balanced address reaches the -workers behind it one answer at a time, and a worker that answered only one -checkpoint would otherwise drop out of the comparison, which is where a leaking -worker could hide. +batch settles, so a run that fails keeps the curve up to the batch it failed in. The +ceiling is judged per worker (by replica address, hostname and pid, since pods in +their own pid namespaces report the same pids) so each worker is compared with +itself, and the checkpoints around the measured window must see the same workers: a +single load-balanced address reaches the workers behind it one answer at a time, and +a worker that answered only one checkpoint would otherwise drop out of the +comparison, which is where a leaking worker could hide. RSS alone is a coarse gauge: on the release stack (spend logs storing prompts, json logs, prometheus and otel callbacks) the same v1.100.0 breadcrumbs grew RSS @@ -79,7 +80,7 @@ from e2e_config import ( MEMORY_IDLE_RSS_BUDGET_MB, MEMORY_REQUESTS_PER_PHASE, MEMORY_RETRIES_PER_REQUEST, - MEMORY_RSS_BUDGET_MB, + MEMORY_RSS_CEILING_MB, MEMORY_RSS_SAMPLE_INTERVAL_SECONDS, MEMORY_RSS_SETTLE_SAMPLES, MEMORY_STORED_REQUEST_BUDGET_KB, @@ -99,7 +100,9 @@ pytestmark = [pytest.mark.e2e, pytest.mark.quiet_stack] DEPLOYMENTS_PER_GROUP: Final = 2 RSS_SAMPLE_CAP: Final = 4 * MEMORY_RSS_SETTLE_SAMPLES WARMUP_BATCHES: Final = 2 -MEASURED_BATCH: Final = WARMUP_BATCHES + 1 +MEASURED_BATCHES: Final = 2 +FIRST_MEASURED_BATCH: Final = WARMUP_BATCHES + 1 +LAST_BATCH: Final = WARMUP_BATCHES + MEASURED_BATCHES @dataclass(frozen=True, slots=True) @@ -207,7 +210,7 @@ def _record_batch_growth( ) -def _settled_rss_around_measured_batch( +def _settled_rss_after_batches( proxy: ProxyClient, key: str, model: str, @@ -216,21 +219,25 @@ def _settled_rss_around_measured_batch( record_property: Callable[[str, object], None], before: Mapping[WorkerKey, RssReading], batch: int, -) -> tuple[Mapping[WorkerKey, RssReading], Mapping[WorkerKey, RssReading]]: + last_batch: int, +) -> Mapping[WorkerKey, RssReading]: calls: Final = _fail_many(proxy, key, model, override) _assert_every_call_failed_through_fallback(calls, fallback) after: Final = _settled_rss_per_worker(proxy) _record_batch_growth(record_property, batch, before, after) - if batch == MEASURED_BATCH: - return before, after - return _settled_rss_around_measured_batch(proxy, key, model, override, fallback, record_property, after, batch + 1) + if batch == last_batch: + return after + return _settled_rss_after_batches( + proxy, key, model, override, fallback, record_property, after, batch + 1, last_batch + ) def _heaviest_worker_growth( warm: Mapping[WorkerKey, RssReading], after: Mapping[WorkerKey, RssReading] ) -> WorkerGrowth: assert warm.keys() == after.keys(), ( - f"the workers answering /debug/memory/summary changed around measured batch {MEASURED_BATCH}, so not every " + f"the workers answering /debug/memory/summary changed around measured batches {FIRST_MEASURED_BATCH}-" + f"{LAST_BATCH}, so not every " f"worker can be compared with itself: gone after it {sorted(warm.keys() - after.keys())} (a worker " f"that died or was restarted under failing traffic, which is what an OOM kill looks like), first seen " f"after it {sorted(after.keys() - warm.keys())} (the warm window never reached them, so they have no " @@ -332,7 +339,7 @@ class TestReliabilityMemory: f"request into the stored snapshot the way the v1.100.0 ones did" ) - warm, after = _settled_rss_around_measured_batch( + warm: Final = _settled_rss_after_batches( client.proxy, scoped_key, primary, @@ -341,16 +348,26 @@ class TestReliabilityMemory: record_property, _settled_rss_per_worker(client.proxy), 1, + WARMUP_BATCHES, + ) + after: Final = _settled_rss_after_batches( + client.proxy, + scoped_key, + primary, + override, + fallback, + record_property, + warm, + FIRST_MEASURED_BATCH, + LAST_BATCH, ) heaviest: Final = _heaviest_worker_growth(warm, after) - assert heaviest.growth_mb <= MEMORY_RSS_BUDGET_MB, ( - f"proxy RSS grew {heaviest.growth_mb:.1f} MB over measured batch {MEASURED_BATCH} of " - f"{MEMORY_REQUESTS_PER_PHASE} failing requests ({MEMORY_RETRIES_PER_REQUEST} retries each plus a fallback) " - f"after {WARMUP_BATCHES} identical warmup batches, past the {MEMORY_RSS_BUDGET_MB:.0f} MB budget: " - f"worker pid {heaviest.warm.worker_pid} on " + assert heaviest.growth_mb <= MEMORY_RSS_CEILING_MB, ( + f"proxy RSS grew {heaviest.growth_mb:.1f} MB over measured batches {FIRST_MEASURED_BATCH}-{LAST_BATCH} " + f"({MEASURED_BATCHES * MEMORY_REQUESTS_PER_PHASE} failing requests) after {WARMUP_BATCHES} identical " + f"warmup batches, past the {MEMORY_RSS_CEILING_MB:.0f} MB ceiling: worker pid {heaviest.warm.worker_pid} on " f"{heaviest.warm.hostname or 'an unnamed host'} behind {heaviest.warm.replica} settled at " - f"{heaviest.warm.ram_usage_mb:.1f} MB warm and " - f"{heaviest.after.ram_usage_mb:.1f} MB after; failing requests are leaking memory the way the " - f"v1.100.0 retry breadcrumbs did" + f"{heaviest.warm.ram_usage_mb:.1f} MB warm and {heaviest.after.ram_usage_mb:.1f} MB after; failing " + f"requests are leaking memory the way the v1.100.0 retry breadcrumbs did" )