Commit graph

7 commits

Author SHA1 Message Date
Kerry Lu
b5037f28d1
test(perf): qualify database upgrades with shared 5 RPS traffic (#46802)
* test(perf): qualify database upgrades with shared 5 RPS traffic

* test(perf): share regular latency and memory limits for upgrades

* fix(perf): escape index format placeholder in upgrade worker

* fix(perf): handle uninitialized multiprocess failure counters

* test(perf): match persisted upgrade requests by call ID

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
2026-10-17 11:39:00 -07:00
Kerry Lu
22e17b69fa
test(e2e): event_loop_stalls charges a hold to the longest gc pass that fits it, not to the sum of every pass since the last hold (#47089)
* test(e2e): event_loop_stalls charges a hold to the longest gc pass that fits it, not to the sum of every pass since the last hold

* test(e2e): event_loop_stalls lists the pass-through post-stream logging step seen once at 0.105 s

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-17 09:48:07 -07:00
Kerry Lu
0a051efba7
test(e2e): event_loop_stalls perf case fails on a gateway event-loop hold over 100 ms that is not on its allowlist (#46974)
* test(e2e): event_loop_stalls perf case reads the gateway's asyncio debug log and fails on a loop hold over 100 ms that is not on its allowlist

The gateway runs the stdlib loop in debug mode (new LITELLM_EVENT_LOOP=asyncio knob next to
PYTHONASYNCIODEBUG=1), the case replays the mixed body mix at 1 rps and groups asyncio's
'Executing <Task ...> took X seconds' lines by coroutine and location. Known holds are listed with
the count they show today; anything else, or a known one past its count, fails the case. Opt-in
through E2E_PERF_EVENT_LOOP=1 so the nightly build skips it

* test(e2e): event_loop_stalls reads the pod list and pod logs through e2e_http instead of a raw httpx client

* test(e2e): event_loop_stalls charges a hold to garbage collection when the gateway's debug-mode gc log accounts for it

The proxy registers a gc.callbacks hook only when the event loop runs in debug mode and logs every
collection over the loop's slow-callback threshold (generation, collected count, seconds). The case
parser pairs such a pass with the hold line that follows it and reports the hold under a bounded
garbage collection bucket instead of the task asyncio happened to blame, which made 7 of 10 local
runs fail on a different task name each time. Holds the collector does not account for keep the
strict allowlist, which gains the prisma first-query warm-up and the second pub/sub subscriber

* test(e2e): event_loop_stalls allowlist caps set at twice the ten-run maximum

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-17 06:55:53 +00:00
Kerry Lu
8227f9bf20
test(perf): cap every background Postgres statement at 0.9 calls per request (#46957)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:58:32 +00:00
Kerry Lu
1dfceccf10
test(perf): cap every awaited Postgres statement at 0.5 calls per request (#46944)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 18:50:27 +00:00
Kerry Lu
cb7e6a7f4c
test(e2e): mixed_size_requests_2rps perf case (coding-agent bodies 5k to 1m tokens with tools, JWT and key callers, two guardrails) (#46717)
Some checks failed
unit / assert-shard-coverage (push) Waiting to run
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
* test(e2e): mixed_size_requests_2rps perf case (coding-agent bodies, JWT and key callers, two guardrails)

* test(e2e_harness): test_observe builds DbCounters with statements and expects the background Postgres line

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 03:35:35 +00:00
Nate Armstrong
f13d0430e8
test(e2e): attach perf case metrics and limits as JUnit properties (#46774)
* test(e2e): attach perf case metrics and limits as JUnit properties

The perf case only printed its metrics table, so a green run left no numbers
behind. Each observed metric and its limit now land as <property> entries on
the test's JUnit testcase, named perf_<metric>[_p<percentile>] and
perf_<metric>[_p<percentile>]_limit, so project-releaser can collect them

* test(e2e): compare perf properties exactly instead of through pytest.approx

basedpyright flags pytest.approx as partially unknown in the e2e lint step,
so the harness test now uses metric values that are exact in binary floating
point
2026-10-15 19:12:07 -07:00