scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-24 05:45:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 71

test(e2e): hold failing-request RSS to a loose ceiling over two measured batches (#47170)

Browse source 
The 48 MB budget on the third batch kept failing on the x86 release gate with no
leak behind it. Isolated rc/1.107.0 and v1.105.1 stacks both grew about 60 MB on
the third batch with gc-tracked objects flat, and both levelled off to a couple of
MB per batch after twenty batches. Measure batches 3-4 against a 200 MB per-worker
ceiling instead, which a native leak of 400 KB per failed request still trips. The
stored-request size check stays the deterministic catch for the v1.100.0 leak
This commit is contained in:
yuneng-jiang 2026-10-17 14:05:42 -07:00 • committed by GitHub
parent 159f185ff4
commit e0514d92b8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 52 additions and 35 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
tests/e2e/AGENTS.md
View file

File diff suppressed because one or more lines are too long

2
tests/e2e/e2e_config.py
View file

@ -237,7 +237,7 @@ MEMORY_TRANSCRIPT_TURNS = int(os.environ.get("E2E_MEMORY_TRANSCRIPT_TURNS", "40"
MEMORY_CONCURRENCY = int(os.environ.get("E2E_MEMORY_CONCURRENCY", "4"))
MEMORY_RSS_SETTLE_SAMPLES = int(os.environ.get("E2E_MEMORY_RSS_SETTLE_SAMPLES", "15"))
MEMORY_RSS_SAMPLE_INTERVAL_SECONDS = float(os.environ.get("E2E_MEMORY_RSS_SAMPLE_INTERVAL_SECONDS", "1"))
MEMORY_RSS_BUDGET_MB = float(os.environ.get("E2E_MEMORY_RSS_BUDGET_MB", "48"))
MEMORY_RSS_CEILING_MB = float(os.environ.get("E2E_MEMORY_RSS_CEILING_MB", "200"))
MEMORY_IDLE_RSS_BUDGET_MB = float(os.environ.get("E2E_MEMORY_IDLE_RSS_BUDGET_MB", "768"))
MEMORY_STORED_REQUEST_BUDGET_KB = float(os.environ.get("E2E_MEMORY_STORED_REQUEST_BUDGET_KB", "64"))

83
tests/e2e/router/test_reliability_memory_e2e.py
View file

@ -1,5 +1,5 @@
"""Live e2e: a few hundred requests that fail before any provider answers must not
grow the proxy's resident memory past a fixed budget once the proxy is warm.
grow the proxy's resident memory past a loose ceiling once the proxy is warm.
The regression this guards shipped in v1.100.0: every retry breadcrumb copied the
whole request and the copies nested into one router-global list, so a proxy under
@ -13,26 +13,27 @@ deployment that fails a handful of times in a row is benched (a bad-credential 4
included), the router answers "No deployments available" without retrying, and the
retry loop that leaks stops running; cooldown_time 0 keeps it running.
Three identical batches run back to back. The first two are the warmup that grows
the proxy's caches and allocator arenas to their steady state, the third is the one
the budget applies to, so a healthy proxy shows the third batch adding roughly
nothing while a leaking one adds a fixed amount per request. One warmup batch is not
enough: each batch's in-flight retries peak well above the live heap, the allocator
keeps that high-water mark, and on a cold worker the second batch still set a new one
(about 50 MB of RSS with the live Python heap flat to within 1 MB), where the later
batches each settled under 25 MB. RSS is read through /debug/memory/summary on every
Four identical batches run back to back. The first two are the warmup that grows
the proxy's caches and allocator arenas, the last two are the measured window, and
each worker's RSS may grow at most a loose ceiling over that window. RSS cannot be
held to a tight per-batch budget: on the x86 release gate (rc/1.107.0 and v1.105.1
on isolated stacks, 28 batches each) the allocator kept setting new high-water marks
for the first several batches, about 60 MB on the third batch of a fresh stack and
more after the rest of the suite, with the gc-tracked object count flat to 0.1%, and
only settled to a couple of MB per batch after twenty. So the ceiling catches a gross
leak and nothing slower. RSS is read through /debug/memory/summary on every
configured replica; a burst of failing calls leaves a transient bulge of garbage that
gc reclaims within seconds, so each checkpoint samples until no new worker has
answered for a settle window and keeps the lowest reading per worker. A checkpoint
runs before the warmup and after every batch, and each batch's settled growth per
worker rides out as a repeated `rss_batch_growth` JUnit property as soon as that
batch settles, so a run that fails keeps the curve up to the batch it failed in. The budget is judged per worker
(by replica address, hostname and pid, since pods in their own pid namespaces report
the same pids) so each worker is compared with itself, and the checkpoints around the
measured batch must see the same workers: a single load-balanced address reaches the
workers behind it one answer at a time, and a worker that answered only one
checkpoint would otherwise drop out of the comparison, which is where a leaking
worker could hide.
batch settles, so a run that fails keeps the curve up to the batch it failed in. The
ceiling is judged per worker (by replica address, hostname and pid, since pods in
their own pid namespaces report the same pids) so each worker is compared with
itself, and the checkpoints around the measured window must see the same workers: a
single load-balanced address reaches the workers behind it one answer at a time, and
a worker that answered only one checkpoint would otherwise drop out of the
comparison, which is where a leaking worker could hide.
RSS alone is a coarse gauge: on the release stack (spend logs storing prompts,
json logs, prometheus and otel callbacks) the same v1.100.0 breadcrumbs grew RSS
@ -79,7 +80,7 @@ from e2e_config import (
MEMORY_IDLE_RSS_BUDGET_MB,
MEMORY_REQUESTS_PER_PHASE,
MEMORY_RETRIES_PER_REQUEST,
MEMORY_RSS_BUDGET_MB,
MEMORY_RSS_CEILING_MB,
MEMORY_RSS_SAMPLE_INTERVAL_SECONDS,
MEMORY_RSS_SETTLE_SAMPLES,
MEMORY_STORED_REQUEST_BUDGET_KB,
@ -99,7 +100,9 @@ pytestmark = [pytest.mark.e2e, pytest.mark.quiet_stack]
DEPLOYMENTS_PER_GROUP: Final = 2
RSS_SAMPLE_CAP: Final = 4 * MEMORY_RSS_SETTLE_SAMPLES
WARMUP_BATCHES: Final = 2
MEASURED_BATCH: Final = WARMUP_BATCHES + 1
MEASURED_BATCHES: Final = 2
FIRST_MEASURED_BATCH: Final = WARMUP_BATCHES + 1
LAST_BATCH: Final = WARMUP_BATCHES + MEASURED_BATCHES
@dataclass(frozen=True, slots=True)
@ -207,7 +210,7 @@ def _record_batch_growth(
)
def _settled_rss_around_measured_batch(
def _settled_rss_after_batches(
proxy: ProxyClient,
key: str,
model: str,
@ -216,21 +219,25 @@ def _settled_rss_around_measured_batch(
record_property: Callable[[str, object], None],
before: Mapping[WorkerKey, RssReading],
batch: int,
) -> tuple[Mapping[WorkerKey, RssReading], Mapping[WorkerKey, RssReading]]:
last_batch: int,
) -> Mapping[WorkerKey, RssReading]:
calls: Final = _fail_many(proxy, key, model, override)
_assert_every_call_failed_through_fallback(calls, fallback)
after: Final = _settled_rss_per_worker(proxy)
_record_batch_growth(record_property, batch, before, after)
if batch == MEASURED_BATCH:
return before, after
return _settled_rss_around_measured_batch(proxy, key, model, override, fallback, record_property, after, batch + 1)
if batch == last_batch:
return after
return _settled_rss_after_batches(
proxy, key, model, override, fallback, record_property, after, batch + 1, last_batch
)
def _heaviest_worker_growth(
warm: Mapping[WorkerKey, RssReading], after: Mapping[WorkerKey, RssReading]
) -> WorkerGrowth:
assert warm.keys() == after.keys(), (
f"the workers answering /debug/memory/summary changed around measured batch {MEASURED_BATCH}, so not every "
f"the workers answering /debug/memory/summary changed around measured batches {FIRST_MEASURED_BATCH}-"
f"{LAST_BATCH}, so not every "
f"worker can be compared with itself: gone after it {sorted(warm.keys() - after.keys())} (a worker "
f"that died or was restarted under failing traffic, which is what an OOM kill looks like), first seen "
f"after it {sorted(after.keys() - warm.keys())} (the warm window never reached them, so they have no "
@ -332,7 +339,7 @@ class TestReliabilityMemory:
f"request into the stored snapshot the way the v1.100.0 ones did"
)
warm, after = _settled_rss_around_measured_batch(
warm: Final = _settled_rss_after_batches(
client.proxy,
scoped_key,
primary,
@ -341,16 +348,26 @@ class TestReliabilityMemory:
record_property,
_settled_rss_per_worker(client.proxy),
1,
WARMUP_BATCHES,
)
after: Final = _settled_rss_after_batches(
client.proxy,
scoped_key,
primary,
override,
fallback,
record_property,
warm,
FIRST_MEASURED_BATCH,
LAST_BATCH,
)
heaviest: Final = _heaviest_worker_growth(warm, after)
assert heaviest.growth_mb <= MEMORY_RSS_BUDGET_MB, (
f"proxy RSS grew {heaviest.growth_mb:.1f} MB over measured batch {MEASURED_BATCH} of "
f"{MEMORY_REQUESTS_PER_PHASE} failing requests ({MEMORY_RETRIES_PER_REQUEST} retries each plus a fallback) "
f"after {WARMUP_BATCHES} identical warmup batches, past the {MEMORY_RSS_BUDGET_MB:.0f} MB budget: "
f"worker pid {heaviest.warm.worker_pid} on "
assert heaviest.growth_mb <= MEMORY_RSS_CEILING_MB, (
f"proxy RSS grew {heaviest.growth_mb:.1f} MB over measured batches {FIRST_MEASURED_BATCH}-{LAST_BATCH} "
f"({MEASURED_BATCHES * MEMORY_REQUESTS_PER_PHASE} failing requests) after {WARMUP_BATCHES} identical "
f"warmup batches, past the {MEMORY_RSS_CEILING_MB:.0f} MB ceiling: worker pid {heaviest.warm.worker_pid} on "
f"{heaviest.warm.hostname or 'an unnamed host'} behind {heaviest.warm.replica} settled at "
f"{heaviest.warm.ram_usage_mb:.1f} MB warm and "
f"{heaviest.after.ram_usage_mb:.1f} MB after; failing requests are leaking memory the way the "
f"v1.100.0 retry breadcrumbs did"
f"{heaviest.warm.ram_usage_mb:.1f} MB warm and {heaviest.after.ram_usage_mb:.1f} MB after; failing "
f"requests are leaking memory the way the v1.100.0 retry breadcrumbs did"
)