mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-18 04:44:37 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kerry Lu
|
34d724c609
|
test(e2e): perf datastore limits count the Postgres statements and Redis commands a request waits on, background work reported on its own (#46798)
* test(e2e): perf datastore limits count what a request waits on, background Postgres and Redis work reported on its own postgres_calls_per_request and redis_calls_per_request were whole-server totals (pg_stat_database commits, Redis INFO commands), so spend flushes and config reloads moved them and a new per-request lookup could hide in the noise. They now count the statements and commands a request waits on before it is forwarded (auth and routing reads, the rate-limiter script), from pg_stat_statements read before and after the run and a Redis MONITOR connection held during it. Spend writes, daily aggregates, config reloads and cache invalidations are postgres_background_calls_per_request and redis_background_calls_per_request. The run output lists the top statements and commands per bucket, and the counters are read right after the run so the test's own spend sample is not counted * test(e2e): perf datastore reads only the runtime user's pg_stat_statements rows, other users' rows have a NULL queryid * test(e2e): perf Redis classifier strips the litellm.caching: namespace the deployed gateway puts on every key * test(e2e): small_bodies_5rps limits from ten split-metric runs (postgres awaited 1.8, redis awaited 18, postgres background 12.5), spend buffers and cron locks classified as background * test(e2e): small_bodies_5rps asserts only the awaited Postgres and Redis counts, background lines stay printed * test(e2e): perf Redis classifier puts the cron lock script's GET and DEL in background, not awaited --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
f13d0430e8
|
test(e2e): attach perf case metrics and limits as JUnit properties (#46774)
* test(e2e): attach perf case metrics and limits as JUnit properties The perf case only printed its metrics table, so a green run left no numbers behind. Each observed metric and its limit now land as <property> entries on the test's JUnit testcase, named perf_<metric>[_p<percentile>] and perf_<metric>[_p<percentile>]_limit, so project-releaser can collect them * test(e2e): compare perf properties exactly instead of through pytest.approx basedpyright flags pytest.approx as partially unknown in the e2e lint step, so the harness test now uses metric values that are exact in binary floating point |
||
|
Kerry Lu
|
5fd6379a78
|
test(e2e): perf users start staggered with jitter, tool-call streams get a first token, small_bodies_5rps sends from the mixed case's callers (#46776)
* test(e2e): perf streams take a reasoning or tool-call delta as the first token, tool-call replies had no time_to_first_token * test(e2e): perf users start at a random offset and jitter each interval by up to 25%, no more lockstep bursts at the planned rate * test(e2e): small_bodies_5rps sends from the mixed case's callers (two teams, JWT users, keys in and out of teams) through the pre and post guardrail --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
fbd078bcbe
|
test(e2e): perf rig (out-of-process mocks, PerformanceTestCase runner, per-request limits) and the small_bodies_5rps case (#46713)
* test(e2e): out-of-process mock OpenAI, Anthropic, Bedrock and guardrail servers for the perf cases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): coding-agent request corpus and real provider captures for the perf mocks Request bodies come from a deterministic coding-agent session (system prompt, Read/Edit/Write/Bash/Grep/Glob tools, tool_use and tool_result turns carrying real source files picked by the request marker) in OpenAI chat, Responses and Anthropic formats. captures/ holds one real response per provider, route, shape and stream mode (gpt-6-luna, claude-sonnet-5, us.anthropic.claude-sonnet-5), cold prompt cache, body only, raw stream chunks with time offsets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): replay module for the perf mocks, captured provider responses picked by route, mode and request size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mocks replay the captured provider responses, knobs shrink to flake and speed The OpenAI, Anthropic and Bedrock mocks answer with the captured body or stream chunks for their route, mode and nearest request size, ids swapped per request, on the captured clock scaled by the p knob. The synthetic reply builders, the per-token stream pacing and the hand-rolled Bedrock event-stream encoder are gone Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mock request log names its provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf guardrail mock remembers the request marker by call id for the response hook Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mock captures retaken cold, one marker and system-prompt prefix per cell Run 2 shared one marker across cells, so every stream capture was served from the provider's prompt cache (cache_read_input_tokens up to 523946) and the json cells had small prefix hits. Every cell now carries its own marker, also prepended to the system prompt, and all 30 captures report zero cached tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf corpus reads tests/e2e files so the lane's runner image has them * test(e2e): perf long_context json captures for Claude retaken with a 16k output cap, no longer cut mid-thinking * test(e2e): perf mocks keep only what the replay path and the test read * test(e2e): perf mocks interpolate the provider's time to first byte from the captures around the request size * test(e2e): perf captures at 100k (short) and 500k (long) tokens with tools, the fixed-size set removed * test(e2e): PerformanceTestCase schema, deterministic request plan and wire records for the perf cases RequestShape names a corpus session (input_tokens, tools, output_tokens) and wire.request_body builds it with setup/corpus/session.py, so a 300k request is 85 file reads of real source in the endpoint's own format. The plan no longer predicts reply text, the mocks replay captured provider responses and the runner joins what the client saw against the mock's log by marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf Metric gains time_to_first_token_seconds Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf User loses the unused role field, every JWT user is an internal_user * test(e2e): perf RequestShape sizes requests from an input-token range, short and long stock shapes * test(e2e): perf case runner (provision, Locust plan replay, observe) and the 50 rps mixed-provider case provision.py checks the deployed config against the case and builds the callers, run.py replays the plan from a separate Locust process (locustfile.py) recording status, header, first-byte, first-token and total time per request, observe.py joins the records with what the mocks received and the workers' RSS to score every Limit, and compares each served body with the capture the mock replayed for it as a diagnostic. The case sends 15,000 requests at 50 rps over three model groups, two guardrails and the three request shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf case asserts time_to_first_token_seconds at percentile 50 and 99 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf provision creates every JWT user as internal_user * test(e2e): perf observe reads only the mock log fields that remain * test(e2e): 50 rps case sends short and long coding-agent sessions sized per request * test(e2e): perf run keeps the per-request records as perf-records.jsonl under E2E_ARTIFACT_DIR * test(e2e): the mixed small-gateway case runs at 25 rps (50 saturated one 4-worker pod on main and on v1.105.1) * test(e2e): small_bodies_high_rate perf case, the github issue 46276 shape (11 to 17 KB chat bodies, keys only, 200 rps) * test(e2e): the mixed small-gateway case runs at 10 rps for 6,000 requests (25 rps still saturated one pod on v1.105.1) * ci: python lint job gets 20 minutes, the e2e harness phase (625 tests plus the replay harness suite) pushes a run past 15 when tests/e2e changes * test(e2e): small-bodies perf case runs at 5 rps so gateway_added_seconds is per-request overhead, not queue wait * test(e2e): mixed-provider perf case runs at 5 rps x 3,000 requests from 100 users * test(e2e): mixed-provider perf case sends happy-path traffic only (no flaky deployment, no guardrail interventions or blocks) * test(e2e): perf gateway model_list is three happy-path deployments, one per provider * test(e2e): Postgres and Redis work per request as perf case limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mocks wait a case-level latency with jitter instead of replaying captured timings, cpu_seconds_per_request limit from metrics-server * test(e2e): perf suite PR carries the small_bodies_5rps case only, the mixed-provider case moves to its own PR * test(e2e): small_bodies_5rps limits from the ten v1.105.1 baselines (1.5x on datastore counts, 2x on latency and cpu, absolute on errors and memory) * test(e2e): perf mocks keep only the short captures the small case reads, captured timings and the Bedrock invoke route removed * test(e2e): perf mocks generate each reply from the request (provider schema, usage, ids, event order, chunk sizes) instead of replaying captures --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |