scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-21 00:21:49 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 74

Latency issue measurements

Browse source 
Wether callbacks are on or not the issue is the same, the first request
of a user is expensive, and it gets worse if many users are making their
first request at the same time
This commit is contained in:
Alexsander Hamir 2026-02-09 09:54:08 -08:00
parent 3f663aea8c
commit 3feee539c5
2 changed files with 130 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

65
callbacks_off_baseline.txt Normal file
View file

@ -0,0 +1,65 @@
=== 42 requests, 42 users, fire-as-fast-as-possible ===
Latency = time from request start until last byte of response received
[OK] User 19 request 1: 4.144s
[OK] User 24 request 1: 3.493s
[OK] User 38 request 1: 1.644s
[OK] User 4 request 1: 6.345s
[OK] User 3 request 1: 6.493s
[OK] User 6 request 1: 6.075s
[OK] User 12 request 1: 5.247s
[OK] User 39 request 1: 1.615s
[OK] User 41 request 1: 1.364s
[OK] User 20 request 1: 4.183s
[OK] User 31 request 1: 2.732s
[OK] User 42 request 1: 1.254s
[OK] User 1 request 1: 6.857s
[OK] User 36 request 1: 2.082s
[OK] User 9 request 1: 5.730s
[OK] User 23 request 1: 3.834s
[OK] User 34 request 1: 2.360s
[OK] User 33 request 1: 2.494s
[OK] User 35 request 1: 2.227s
[OK] User 11 request 1: 5.470s
[OK] User 25 request 1: 3.580s
[OK] User 21 request 1: 4.117s
[OK] User 14 request 1: 5.067s
[OK] User 37 request 1: 1.975s
[OK] User 18 request 1: 4.522s
[OK] User 40 request 1: 1.576s
[OK] User 22 request 1: 3.984s
[OK] User 13 request 1: 5.208s
[OK] User 15 request 1: 4.937s
[OK] User 26 request 1: 3.451s
[OK] User 8 request 1: 5.906s
[OK] User 30 request 1: 2.923s
[OK] User 16 request 1: 4.803s
[OK] User 17 request 1: 4.672s
[OK] User 5 request 1: 6.323s
[OK] User 2 request 1: 6.761s
[OK] User 10 request 1: 5.635s
[OK] User 27 request 1: 3.329s
[OK] User 32 request 1: 2.664s
[OK] User 29 request 1: 3.065s
[OK] User 7 request 1: 6.054s
[OK] User 28 request 1: 3.213s
[OK] 42 succeeded, [FAIL] 0 failed
Latency (successful):
min: 1.254s
avg: 4.034s
max: 6.857s
p95: 6.345s
p99: 6.761s
Requests above threshold (successful only):
Above 1s: 42/42 (100.0%)
Above 2s: 36/42 (85.7%)
Above 3s: 29/42 (69.0%)
Above 4s: 21/42 (50.0%)
Above 5s: 14/42 (33.3%)
Above 6s: 7/42 (16.7%)
Above 7s: 0/42 (0.0%)
Above 8s: 0/42 (0.0%)
Above 9s: 0/42 (0.0%)
Above 10s: 0/42 (0.0%)

65
callbacks_on_baseline.txt Normal file
View file

@ -0,0 +1,65 @@
=== 42 requests, 42 users, fire-as-fast-as-possible ===
Latency = time from request start until last byte of response received
[OK] User 29 request 1: 2.908s
[OK] User 9 request 1: 5.687s
[OK] User 1 request 1: 6.917s
[OK] User 6 request 1: 6.203s
[OK] User 37 request 1: 1.901s
[OK] User 30 request 1: 2.907s
[OK] User 24 request 1: 3.728s
[OK] User 27 request 1: 3.315s
[OK] User 13 request 1: 5.271s
[OK] User 20 request 1: 4.310s
[OK] User 33 request 1: 2.524s
[OK] User 16 request 1: 4.879s
[OK] User 21 request 1: 4.187s
[OK] User 15 request 1: 5.039s
[OK] User 34 request 1: 2.427s
[OK] User 4 request 1: 6.620s
[OK] User 17 request 1: 4.806s
[OK] User 12 request 1: 5.502s
[OK] User 19 request 1: 4.533s
[OK] User 10 request 1: 5.799s
[OK] User 18 request 1: 4.686s
[OK] User 2 request 1: 6.928s
[OK] User 28 request 1: 3.304s
[OK] User 31 request 1: 2.909s
[OK] User 8 request 1: 6.099s
[OK] User 25 request 1: 3.730s
[OK] User 42 request 1: 1.367s
[OK] User 39 request 1: 1.795s
[OK] User 5 request 1: 6.533s
[OK] User 32 request 1: 2.786s
[OK] User 26 request 1: 3.628s
[OK] User 3 request 1: 6.839s
[OK] User 7 request 1: 6.277s
[OK] User 11 request 1: 5.723s
[OK] User 35 request 1: 2.396s
[OK] User 41 request 1: 1.551s
[OK] User 22 request 1: 4.204s
[OK] User 14 request 1: 5.325s
[OK] User 36 request 1: 2.279s
[OK] User 40 request 1: 1.715s
[OK] User 38 request 1: 2.001s
[OK] User 23 request 1: 4.138s
[OK] 42 succeeded, [FAIL] 0 failed
Latency (successful):
min: 1.367s
avg: 4.183s
max: 6.928s
p95: 6.620s
p99: 6.917s
Requests above threshold (successful only):
Above 1s: 42/42 (100.0%)
Above 2s: 37/42 (88.1%)
Above 3s: 28/42 (66.7%)
Above 4s: 23/42 (54.8%)
Above 5s: 15/42 (35.7%)
Above 6s: 8/42 (19.0%)
Above 7s: 0/42 (0.0%)
Above 8s: 0/42 (0.0%)
Above 9s: 0/42 (0.0%)
Above 10s: 0/42 (0.0%)