From 7615085e673a2c86c6248cacfd7521adc5885199 Mon Sep 17 00:00:00 2001 From: Alexsander Hamir Date: Mon, 9 Feb 2026 11:24:35 -0800 Subject: [PATCH] Add created mode to latency test and capture happy-path results - measure_latency.py: add --user-mode created, --key-mode per_user Pre-creates end users via POST /customer/new and keys via /key/generate - problem_tracker: add created-mode scenarios, checkboxes, and analysis Created mode (pre-existing users) shows better latency than random (non-existent users) at 10 req/user: 1.2s avg vs 1.5s, 10% vs 33% above 1s --- callbacks_off_10_per_user_created.txt | 1028 +++++++++++++++++++++++++ callbacks_off_1_per_user_created.txt | 128 +++ measure_latency.py | 4 +- problem_tracker.md | 25 +- 4 files changed, 1181 insertions(+), 4 deletions(-) create mode 100644 callbacks_off_10_per_user_created.txt create mode 100644 callbacks_off_1_per_user_created.txt diff --git a/callbacks_off_10_per_user_created.txt b/callbacks_off_10_per_user_created.txt new file mode 100644 index 00000000000..a87fa18c1d1 --- /dev/null +++ b/callbacks_off_10_per_user_created.txt @@ -0,0 +1,1028 @@ +Creating 100 keys via /key/generate... +Created 100 keys. +Creating 100 end users via /customer/new... +Created 100 end users. +=== 1000 requests, 100 users, fire-as-fast-as-possible === +User mode: created, Key mode: per_user +Latency = time from request start until last byte of response received + [OK] User 4 request 1: 14.670s + [OK] User 4 request 2: 0.005s + [OK] User 4 request 3: 0.008s + [OK] User 4 request 4: 0.008s + [OK] User 4 request 5: 0.026s + [OK] User 24 request 1: 11.950s + [OK] User 4 request 6: 0.017s + [OK] User 24 request 2: 0.017s + [OK] User 59 request 1: 7.169s + [OK] User 4 request 7: 0.014s + [OK] User 24 request 3: 0.028s + [OK] User 59 request 2: 0.098s + [OK] User 4 request 8: 0.098s + [OK] User 3 request 1: 15.010s + [OK] User 2 request 1: 15.165s + [OK] User 24 request 4: 0.188s + [OK] User 29 request 1: 11.496s + [OK] User 98 request 1: 2.029s + [OK] User 18 request 1: 13.126s + [OK] User 20 request 1: 12.849s + [OK] User 96 request 1: 2.395s + [OK] User 1 request 1: 15.483s + [OK] User 17 request 1: 13.296s + [OK] User 27 request 1: 11.912s + [OK] User 36 request 1: 10.679s + [OK] User 6 request 1: 14.819s + [OK] User 30 request 1: 11.503s + [OK] User 13 request 1: 13.850s + [OK] User 28 request 1: 11.776s + [OK] User 22 request 1: 12.638s + [OK] User 10 request 1: 14.299s + [OK] User 15 request 1: 13.609s + [OK] User 12 request 1: 14.024s + [OK] User 21 request 1: 12.777s + [OK] User 59 request 3: 0.315s + [OK] User 16 request 1: 13.499s + [OK] User 31 request 1: 11.430s + [OK] User 26 request 1: 12.114s + [OK] User 34 request 1: 11.019s + [OK] User 25 request 1: 12.253s + [OK] User 87 request 1: 3.738s + [OK] User 67 request 1: 6.484s + [OK] User 47 request 1: 9.264s + [OK] User 32 request 1: 11.327s + [OK] User 43 request 1: 9.816s + [OK] User 33 request 1: 11.191s + [OK] User 49 request 1: 8.992s + [OK] User 48 request 1: 9.129s + [OK] User 7 request 1: 14.781s + [OK] User 41 request 1: 10.093s + [OK] User 38 request 1: 10.503s + [OK] User 3 request 2: 0.351s + [OK] User 35 request 1: 10.943s + [OK] User 9 request 1: 14.530s + [OK] User 44 request 1: 9.707s + [OK] User 5 request 1: 15.086s + [OK] User 2 request 2: 0.364s + [OK] User 74 request 1: 5.613s + [OK] User 72 request 1: 5.888s + [OK] User 42 request 1: 10.011s + [OK] User 64 request 1: 6.988s + [OK] User 11 request 1: 14.283s + [OK] User 85 request 1: 4.105s + [OK] User 90 request 1: 3.416s + [OK] User 95 request 1: 2.722s + [OK] User 79 request 1: 4.931s + [OK] User 77 request 1: 5.205s + [OK] User 82 request 1: 4.541s + [OK] User 56 request 1: 8.115s + [OK] User 60 request 1: 7.568s + [OK] User 76 request 1: 5.369s + [OK] User 65 request 1: 6.881s + [OK] User 66 request 1: 6.741s + [OK] User 80 request 1: 4.823s + [OK] User 92 request 1: 3.167s + [OK] User 97 request 1: 2.496s + [OK] User 23 request 1: 12.677s + [OK] User 58 request 1: 7.877s + [OK] User 8 request 1: 14.760s + [OK] User 62 request 1: 7.325s + [OK] User 54 request 1: 8.427s + [OK] User 93 request 1: 3.061s + [OK] User 83 request 1: 4.444s + [OK] User 91 request 1: 3.342s + [OK] User 55 request 1: 8.291s + [OK] User 99 request 1: 2.228s + [OK] User 73 request 1: 5.815s + [OK] User 51 request 1: 8.839s + [OK] User 61 request 1: 7.465s + [OK] User 71 request 1: 6.089s + [OK] User 24 request 5: 0.375s + [OK] User 29 request 2: 0.375s + [OK] User 50 request 1: 8.998s + [OK] User 53 request 1: 8.589s + [OK] User 14 request 1: 13.957s + [OK] User 69 request 1: 6.388s + [OK] User 52 request 1: 8.727s + [OK] User 19 request 1: 13.264s + [OK] User 84 request 1: 4.333s + [OK] User 94 request 1: 2.949s + [OK] User 86 request 1: 4.057s + [OK] User 98 request 2: 0.381s + [OK] User 88 request 1: 3.807s + [OK] User 40 request 1: 10.407s + [OK] User 45 request 1: 9.721s + [OK] User 100 request 1: 2.148s + [OK] User 37 request 1: 10.820s + [OK] User 39 request 1: 10.544s + [OK] User 46 request 1: 9.583s + [OK] User 63 request 1: 7.273s + [OK] User 70 request 1: 6.312s + [OK] User 4 request 9: 0.570s + [OK] User 81 request 1: 4.827s + [OK] User 18 request 2: 0.369s + [OK] User 20 request 2: 0.368s + [OK] User 96 request 2: 0.368s + [OK] User 68 request 1: 6.638s + [OK] User 17 request 2: 0.364s + [OK] User 27 request 2: 0.364s + [OK] User 36 request 2: 0.364s + [OK] User 6 request 2: 0.364s + [OK] User 30 request 2: 0.364s + [OK] User 13 request 2: 0.364s + [OK] User 28 request 2: 0.364s + [OK] User 57 request 1: 8.176s + [OK] User 22 request 2: 0.358s + [OK] User 10 request 2: 0.358s + [OK] User 15 request 2: 0.358s + [OK] User 12 request 2: 0.358s + [OK] User 21 request 2: 0.358s + [OK] User 59 request 4: 0.363s + [OK] User 16 request 2: 0.361s + [OK] User 31 request 2: 0.361s + [OK] User 26 request 2: 0.361s + [OK] User 34 request 2: 0.361s + [OK] User 25 request 2: 0.361s + [OK] User 87 request 2: 0.361s + [OK] User 67 request 2: 0.361s + [OK] User 47 request 2: 0.365s + [OK] User 32 request 2: 0.364s + [OK] User 43 request 2: 0.365s + [OK] User 33 request 2: 0.364s + [OK] User 49 request 2: 0.364s + [OK] User 48 request 2: 0.364s + [OK] User 7 request 2: 0.364s + [OK] User 41 request 2: 0.365s + [OK] User 38 request 2: 0.365s + [OK] User 3 request 3: 0.370s + [OK] User 35 request 2: 0.368s + [OK] User 9 request 2: 0.368s + [OK] User 44 request 2: 0.368s + [OK] User 5 request 2: 0.368s + [OK] User 74 request 2: 0.365s + [OK] User 72 request 2: 0.365s + [OK] User 42 request 2: 0.365s + [OK] User 64 request 2: 0.365s + [OK] User 11 request 2: 0.365s + [OK] User 85 request 2: 0.365s + [OK] User 90 request 2: 0.365s + [OK] User 95 request 2: 0.365s + [OK] User 79 request 2: 0.365s + [OK] User 77 request 2: 0.365s + [OK] User 75 request 1: 5.850s + [OK] User 82 request 2: 0.358s + [OK] User 56 request 2: 0.357s + [OK] User 60 request 2: 0.355s + [OK] User 76 request 2: 0.354s + [OK] User 65 request 2: 0.354s + [OK] User 66 request 2: 0.355s + [OK] User 80 request 2: 0.355s + [OK] User 92 request 2: 0.355s + [OK] User 97 request 2: 0.348s + [OK] User 23 request 2: 0.346s + [OK] User 58 request 2: 0.341s + [OK] User 8 request 2: 0.341s + [OK] User 62 request 2: 0.341s + [OK] User 54 request 2: 0.341s + [OK] User 93 request 2: 0.341s + [OK] User 83 request 2: 0.341s + [OK] User 91 request 2: 0.341s + [OK] User 55 request 2: 0.341s + [OK] User 99 request 2: 0.341s + [OK] User 73 request 2: 0.341s + [OK] User 51 request 2: 0.341s + [OK] User 61 request 2: 0.341s + [OK] User 71 request 2: 0.341s + [OK] User 24 request 6: 0.348s + [OK] User 29 request 3: 0.348s + [OK] User 50 request 2: 0.337s + [OK] User 53 request 2: 0.336s + [OK] User 14 request 2: 0.336s + [OK] User 69 request 2: 0.336s + [OK] User 52 request 2: 0.336s + [OK] User 19 request 2: 0.336s + [OK] User 84 request 2: 0.336s + [OK] User 94 request 2: 0.336s + [OK] User 86 request 2: 0.336s + [OK] User 1 request 2: 0.615s + [OK] User 98 request 3: 0.343s + [OK] User 88 request 2: 0.335s + [OK] User 40 request 2: 0.330s + [OK] User 45 request 2: 0.329s + [OK] User 100 request 2: 0.329s + [OK] User 37 request 2: 0.329s + [OK] User 39 request 2: 0.329s + [OK] User 46 request 2: 0.330s + [OK] User 63 request 2: 0.334s + [OK] User 70 request 2: 0.333s + [OK] User 4 request 10: 0.604s + [OK] User 81 request 2: 0.592s + [OK] User 78 request 1: 5.845s + [OK] User 18 request 3: 0.604s + [OK] User 20 request 3: 0.604s + [OK] User 96 request 3: 0.604s + [OK] User 68 request 2: 0.585s + [OK] User 17 request 3: 0.599s + [OK] User 27 request 3: 0.599s + [OK] User 36 request 3: 0.599s + [OK] User 6 request 3: 0.599s + [OK] User 30 request 3: 0.599s + [OK] User 13 request 3: 0.599s + [OK] User 28 request 3: 0.599s + [OK] User 57 request 2: 0.583s + [OK] User 22 request 3: 0.593s + [OK] User 10 request 3: 0.593s + [OK] User 15 request 3: 0.593s + [OK] User 12 request 3: 0.593s + [OK] User 21 request 3: 0.593s + [OK] User 2 request 3: 0.835s + [OK] User 59 request 5: 0.588s + [OK] User 16 request 3: 0.588s + [OK] User 31 request 3: 0.588s + [OK] User 26 request 3: 0.588s + [OK] User 34 request 3: 0.588s + [OK] User 25 request 3: 0.588s + [OK] User 87 request 3: 0.588s + [OK] User 67 request 3: 0.588s + [OK] User 47 request 3: 0.583s + [OK] User 32 request 3: 0.583s + [OK] User 43 request 3: 0.583s + [OK] User 33 request 3: 0.583s + [OK] User 49 request 3: 0.583s + [OK] User 48 request 3: 0.583s + [OK] User 7 request 3: 0.583s + [OK] User 41 request 3: 0.583s + [OK] User 38 request 3: 0.583s + [OK] User 35 request 3: 0.580s + [OK] User 9 request 3: 0.580s + [OK] User 44 request 3: 0.580s + [OK] User 5 request 3: 0.580s + [OK] User 74 request 3: 0.575s + [OK] User 72 request 3: 0.575s + [OK] User 42 request 3: 0.575s + [OK] User 64 request 3: 0.575s + [OK] User 11 request 3: 0.575s + [OK] User 85 request 3: 0.575s + [OK] User 90 request 3: 0.575s + [OK] User 95 request 3: 0.575s + [OK] User 79 request 3: 0.575s + [OK] User 77 request 3: 0.575s + [OK] User 75 request 2: 0.568s + [OK] User 82 request 3: 0.572s + [OK] User 56 request 3: 0.572s + [OK] User 60 request 3: 0.572s + [OK] User 76 request 3: 0.572s + [OK] User 65 request 3: 0.572s + [OK] User 66 request 3: 0.571s + [OK] User 80 request 3: 0.572s + [OK] User 92 request 3: 0.572s + [OK] User 97 request 3: 0.569s + [OK] User 23 request 3: 0.569s + [OK] User 58 request 3: 0.569s + [OK] User 8 request 3: 0.568s + [OK] User 62 request 3: 0.568s + [OK] User 54 request 3: 0.569s + [OK] User 93 request 3: 0.569s + [OK] User 83 request 3: 0.569s + [OK] User 91 request 3: 0.568s + [OK] User 55 request 3: 0.568s + [OK] User 99 request 3: 0.568s + [OK] User 73 request 3: 0.568s + [OK] User 51 request 3: 0.568s + [OK] User 61 request 3: 0.568s + [OK] User 71 request 3: 0.568s + [OK] User 24 request 7: 0.571s + [OK] User 29 request 4: 0.571s + [OK] User 50 request 3: 0.571s + [OK] User 53 request 3: 0.571s + [OK] User 14 request 3: 0.571s + [OK] User 69 request 3: 0.571s + [OK] User 52 request 3: 0.571s + [OK] User 19 request 3: 0.570s + [OK] User 84 request 3: 0.570s + [OK] User 94 request 3: 0.570s + [OK] User 86 request 3: 0.570s + [OK] User 1 request 3: 0.568s + [OK] User 98 request 4: 0.568s + [OK] User 88 request 3: 0.568s + [OK] User 40 request 3: 0.568s + [OK] User 45 request 3: 0.568s + [OK] User 100 request 3: 0.568s + [OK] User 37 request 3: 0.568s + [OK] User 39 request 3: 0.568s + [OK] User 46 request 3: 0.568s + [OK] User 63 request 3: 0.556s + [OK] User 70 request 3: 0.556s + [OK] User 81 request 3: 0.305s + [OK] User 78 request 2: 0.291s + [OK] User 18 request 4: 0.306s + [OK] User 20 request 4: 0.306s + [OK] User 96 request 4: 0.306s + [OK] User 68 request 3: 0.306s + [OK] User 17 request 4: 0.308s + [OK] User 27 request 4: 0.308s + [OK] User 36 request 4: 0.308s + [OK] User 30 request 4: 0.307s + [OK] User 13 request 4: 0.307s + [OK] User 28 request 4: 0.307s + [OK] User 57 request 3: 0.307s + [OK] User 3 request 4: 0.779s + [OK] User 22 request 4: 0.310s + [OK] User 10 request 4: 0.310s + [OK] User 15 request 4: 0.310s + [OK] User 12 request 4: 0.310s + [OK] User 2 request 4: 0.307s + [OK] User 59 request 6: 0.312s + [OK] User 16 request 4: 0.312s + [OK] User 31 request 4: 0.312s + [OK] User 26 request 4: 0.312s + [OK] User 34 request 4: 0.312s + [OK] User 25 request 4: 0.312s + [OK] User 87 request 4: 0.312s + [OK] User 67 request 4: 0.312s + [OK] User 47 request 4: 0.313s + [OK] User 32 request 4: 0.313s + [OK] User 43 request 4: 0.313s + [OK] User 33 request 4: 0.313s + [OK] User 49 request 4: 0.313s + [OK] User 48 request 4: 0.313s + [OK] User 7 request 4: 0.313s + [OK] User 41 request 4: 0.313s + [OK] User 38 request 4: 0.313s + [OK] User 35 request 4: 0.315s + [OK] User 9 request 4: 0.315s + [OK] User 44 request 4: 0.315s + [OK] User 5 request 4: 0.315s + [OK] User 74 request 4: 0.318s + [OK] User 72 request 4: 0.318s + [OK] User 42 request 4: 0.318s + [OK] User 64 request 4: 0.318s + [OK] User 11 request 4: 0.318s + [OK] User 85 request 4: 0.318s + [OK] User 90 request 4: 0.318s + [OK] User 95 request 4: 0.318s + [OK] User 79 request 4: 0.318s + [OK] User 77 request 4: 0.318s + [OK] User 75 request 3: 0.318s + [OK] User 82 request 4: 0.320s + [OK] User 56 request 4: 0.321s + [OK] User 60 request 4: 0.320s + [OK] User 76 request 4: 0.320s + [OK] User 65 request 4: 0.321s + [OK] User 66 request 4: 0.321s + [OK] User 80 request 4: 0.320s + [OK] User 92 request 4: 0.320s + [OK] User 89 request 1: 4.839s + [OK] User 97 request 4: 0.320s + [OK] User 23 request 4: 0.320s + [OK] User 58 request 4: 0.320s + [OK] User 8 request 4: 0.320s + [OK] User 62 request 4: 0.320s + [OK] User 54 request 4: 0.320s + [OK] User 93 request 4: 0.320s + [OK] User 83 request 4: 0.320s + [OK] User 91 request 4: 0.320s + [OK] User 55 request 4: 0.320s + [OK] User 99 request 4: 0.320s + [OK] User 73 request 4: 0.320s + [OK] User 51 request 4: 0.320s + [OK] User 61 request 4: 0.320s + [OK] User 71 request 4: 0.321s + [OK] User 24 request 8: 0.321s + [OK] User 29 request 5: 0.321s + [OK] User 50 request 4: 0.321s + [OK] User 53 request 4: 0.321s + [OK] User 14 request 4: 0.321s + [OK] User 69 request 4: 0.321s + [OK] User 52 request 4: 0.321s + [OK] User 19 request 4: 0.321s + [OK] User 84 request 4: 0.321s + [OK] User 94 request 4: 0.321s + [OK] User 86 request 4: 0.321s + [OK] User 1 request 4: 0.321s + [OK] User 98 request 5: 0.317s + [OK] User 88 request 4: 0.317s + [OK] User 40 request 4: 0.317s + [OK] User 45 request 4: 0.317s + [OK] User 100 request 4: 0.317s + [OK] User 37 request 4: 0.317s + [OK] User 39 request 4: 0.317s + [OK] User 46 request 4: 0.317s + [OK] User 6 request 4: 0.533s + [OK] User 63 request 4: 0.321s + [OK] User 70 request 4: 0.321s + [OK] User 21 request 4: 0.532s + [OK] User 81 request 4: 0.315s + [OK] User 78 request 3: 0.315s + [OK] User 18 request 5: 0.322s + [OK] User 20 request 5: 0.322s + [OK] User 96 request 5: 0.322s + [OK] User 68 request 4: 0.322s + [OK] User 17 request 5: 0.321s + [OK] User 27 request 5: 0.321s + [OK] User 36 request 5: 0.321s + [OK] User 30 request 5: 0.321s + [OK] User 13 request 5: 0.321s + [OK] User 28 request 5: 0.321s + [OK] User 57 request 4: 0.321s + [OK] User 3 request 5: 0.318s + [OK] User 22 request 5: 0.325s + [OK] User 10 request 5: 0.325s + [OK] User 15 request 5: 0.325s + [OK] User 12 request 5: 0.325s + [OK] User 2 request 5: 0.325s + [OK] User 59 request 7: 0.326s + [OK] User 31 request 5: 0.325s + [OK] User 26 request 5: 0.325s + [OK] User 34 request 5: 0.325s + [OK] User 25 request 5: 0.325s + [OK] User 87 request 5: 0.325s + [OK] User 67 request 5: 0.325s + [OK] User 47 request 5: 0.326s + [OK] User 32 request 5: 0.326s + [OK] User 43 request 5: 0.326s + [OK] User 33 request 5: 0.326s + [OK] User 49 request 5: 0.326s + [OK] User 48 request 5: 0.326s + [OK] User 7 request 5: 0.326s + [OK] User 41 request 5: 0.326s + [OK] User 38 request 5: 0.326s + [OK] User 35 request 5: 0.322s + [OK] User 9 request 5: 0.322s + [OK] User 44 request 5: 0.322s + [OK] User 5 request 5: 0.322s + [OK] User 74 request 5: 0.320s + [OK] User 72 request 5: 0.320s + [OK] User 42 request 5: 0.320s + [OK] User 64 request 5: 0.320s + [OK] User 11 request 5: 0.320s + [OK] User 85 request 5: 0.320s + [OK] User 90 request 5: 0.320s + [OK] User 95 request 5: 0.320s + [OK] User 79 request 5: 0.320s + [OK] User 77 request 5: 0.320s + [OK] User 75 request 4: 0.320s + [OK] User 82 request 5: 0.325s + [OK] User 56 request 5: 0.324s + [OK] User 60 request 5: 0.324s + [OK] User 76 request 5: 0.325s + [OK] User 65 request 5: 0.324s + [OK] User 66 request 5: 0.324s + [OK] User 80 request 5: 0.324s + [OK] User 92 request 5: 0.325s + [OK] User 89 request 2: 0.315s + [OK] User 97 request 5: 0.327s + [OK] User 23 request 5: 0.327s + [OK] User 58 request 5: 0.327s + [OK] User 8 request 5: 0.326s + [OK] User 62 request 5: 0.326s + [OK] User 54 request 5: 0.326s + [OK] User 93 request 5: 0.327s + [OK] User 83 request 5: 0.327s + [OK] User 91 request 5: 0.327s + [OK] User 55 request 5: 0.327s + [OK] User 99 request 5: 0.326s + [OK] User 73 request 5: 0.326s + [OK] User 51 request 5: 0.326s + [OK] User 61 request 5: 0.326s + [OK] User 71 request 5: 0.325s + [OK] User 24 request 9: 0.326s + [OK] User 50 request 5: 0.325s + [OK] User 53 request 5: 0.326s + [OK] User 14 request 5: 0.326s + [OK] User 69 request 5: 0.326s + [OK] User 52 request 5: 0.325s + [OK] User 19 request 5: 0.326s + [OK] User 84 request 5: 0.326s + [OK] User 94 request 5: 0.326s + [OK] User 86 request 5: 0.326s + [OK] User 1 request 5: 0.326s + [OK] User 98 request 6: 0.328s + [OK] User 88 request 5: 0.328s + [OK] User 40 request 5: 0.328s + [OK] User 45 request 5: 0.328s + [OK] User 100 request 5: 0.328s + [OK] User 37 request 5: 0.328s + [OK] User 39 request 5: 0.328s + [OK] User 46 request 5: 0.328s + [OK] User 6 request 5: 0.327s + [OK] User 63 request 5: 0.322s + [OK] User 70 request 5: 0.322s + [OK] User 21 request 5: 0.320s + [OK] User 81 request 5: 0.324s + [OK] User 78 request 4: 0.324s + [OK] User 16 request 5: 0.543s + [OK] User 18 request 6: 0.316s + [OK] User 20 request 6: 0.315s + [OK] User 96 request 6: 0.315s + [OK] User 68 request 5: 0.315s + [OK] User 17 request 6: 0.327s + [OK] User 27 request 6: 0.327s + [OK] User 36 request 6: 0.327s + [OK] User 13 request 6: 0.327s + [OK] User 28 request 6: 0.327s + [OK] User 57 request 5: 0.327s + [OK] User 3 request 6: 0.326s + [OK] User 22 request 6: 0.325s + [OK] User 10 request 6: 0.325s + [OK] User 15 request 6: 0.325s + [OK] User 12 request 6: 0.325s + [OK] User 2 request 6: 0.325s + [OK] User 59 request 8: 0.325s + [OK] User 31 request 6: 0.325s + [OK] User 26 request 6: 0.325s + [OK] User 34 request 6: 0.325s + [OK] User 25 request 6: 0.325s + [OK] User 87 request 6: 0.325s + [OK] User 67 request 6: 0.325s + [OK] User 47 request 6: 0.323s + [OK] User 32 request 6: 0.324s + [OK] User 43 request 6: 0.323s + [OK] User 33 request 6: 0.324s + [OK] User 49 request 6: 0.324s + [OK] User 48 request 6: 0.324s + [OK] User 7 request 6: 0.323s + [OK] User 41 request 6: 0.324s + [OK] User 38 request 6: 0.324s + [OK] User 35 request 6: 0.327s + [OK] User 9 request 6: 0.327s + [OK] User 44 request 6: 0.327s + [OK] User 5 request 6: 0.327s + [OK] User 74 request 6: 0.329s + [OK] User 72 request 6: 0.329s + [OK] User 42 request 6: 0.329s + [OK] User 64 request 6: 0.329s + [OK] User 11 request 6: 0.329s + [OK] User 85 request 6: 0.329s + [OK] User 90 request 6: 0.329s + [OK] User 95 request 6: 0.328s + [OK] User 79 request 6: 0.329s + [OK] User 77 request 6: 0.329s + [OK] User 75 request 5: 0.329s + [OK] User 29 request 6: 0.597s + [OK] User 82 request 6: 0.326s + [OK] User 56 request 6: 0.326s + [OK] User 60 request 6: 0.326s + [OK] User 76 request 6: 0.326s + [OK] User 65 request 6: 0.326s + [OK] User 66 request 6: 0.326s + [OK] User 80 request 6: 0.327s + [OK] User 92 request 6: 0.326s + [OK] User 89 request 3: 0.326s + [OK] User 97 request 6: 0.326s + [OK] User 23 request 6: 0.326s + [OK] User 58 request 6: 0.326s + [OK] User 8 request 6: 0.326s + [OK] User 62 request 6: 0.326s + [OK] User 54 request 6: 0.327s + [OK] User 93 request 6: 0.326s + [OK] User 83 request 6: 0.326s + [OK] User 91 request 6: 0.327s + [OK] User 55 request 6: 0.327s + [OK] User 99 request 6: 0.327s + [OK] User 73 request 6: 0.327s + [OK] User 51 request 6: 0.327s + [OK] User 61 request 6: 0.327s + [OK] User 71 request 6: 0.327s + [OK] User 24 request 10: 0.325s + [OK] User 50 request 6: 0.325s + [OK] User 53 request 6: 0.325s + [OK] User 14 request 6: 0.325s + [OK] User 69 request 6: 0.325s + [OK] User 52 request 6: 0.325s + [OK] User 19 request 6: 0.325s + [OK] User 84 request 6: 0.325s + [OK] User 94 request 6: 0.325s + [OK] User 86 request 6: 0.325s + [OK] User 1 request 6: 0.325s + [OK] User 98 request 7: 0.326s + [OK] User 88 request 6: 0.326s + [OK] User 40 request 6: 0.326s + [OK] User 45 request 6: 0.326s + [OK] User 100 request 6: 0.326s + [OK] User 37 request 6: 0.326s + [OK] User 39 request 6: 0.326s + [OK] User 46 request 6: 0.326s + [OK] User 6 request 6: 0.326s + [OK] User 30 request 6: 0.558s + [OK] User 63 request 6: 0.326s + [OK] User 70 request 6: 0.326s + [OK] User 21 request 6: 0.326s + [OK] User 81 request 6: 0.326s + [OK] User 78 request 5: 0.326s + [OK] User 16 request 6: 0.324s + [OK] User 18 request 7: 0.328s + [OK] User 96 request 7: 0.328s + [OK] User 68 request 6: 0.328s + [OK] User 17 request 7: 0.319s + [OK] User 27 request 7: 0.319s + [OK] User 36 request 7: 0.320s + [OK] User 13 request 7: 0.320s + [OK] User 28 request 7: 0.320s + [OK] User 57 request 6: 0.320s + [OK] User 3 request 7: 0.320s + [OK] User 22 request 7: 0.323s + [OK] User 15 request 7: 0.323s + [OK] User 12 request 7: 0.323s + [OK] User 2 request 7: 0.323s + [OK] User 59 request 9: 0.320s + [OK] User 31 request 7: 0.320s + [OK] User 26 request 7: 0.320s + [OK] User 34 request 7: 0.320s + [OK] User 25 request 7: 0.320s + [OK] User 87 request 7: 0.320s + [OK] User 67 request 7: 0.320s + [OK] User 47 request 7: 0.320s + [OK] User 32 request 7: 0.320s + [OK] User 43 request 7: 0.320s + [OK] User 33 request 7: 0.320s + [OK] User 49 request 7: 0.320s + [OK] User 48 request 7: 0.320s + [OK] User 7 request 7: 0.320s + [OK] User 41 request 7: 0.320s + [OK] User 38 request 7: 0.320s + [OK] User 35 request 7: 0.321s + [OK] User 9 request 7: 0.321s + [OK] User 44 request 7: 0.321s + [OK] User 5 request 7: 0.321s + [OK] User 74 request 7: 0.322s + [OK] User 72 request 7: 0.322s + [OK] User 42 request 7: 0.322s + [OK] User 64 request 7: 0.322s + [OK] User 11 request 7: 0.322s + [OK] User 85 request 7: 0.322s + [OK] User 90 request 7: 0.323s + [OK] User 95 request 7: 0.323s + [OK] User 79 request 7: 0.323s + [OK] User 77 request 7: 0.323s + [OK] User 75 request 6: 0.323s + [OK] User 29 request 7: 0.320s + [OK] User 82 request 7: 0.323s + [OK] User 56 request 7: 0.323s + [OK] User 60 request 7: 0.323s + [OK] User 76 request 7: 0.323s + [OK] User 65 request 7: 0.323s + [OK] User 66 request 7: 0.323s + [OK] User 80 request 7: 0.323s + [OK] User 92 request 7: 0.323s + [OK] User 89 request 4: 0.323s + [OK] User 97 request 7: 0.324s + [OK] User 23 request 7: 0.324s + [OK] User 58 request 7: 0.325s + [OK] User 8 request 7: 0.325s + [OK] User 62 request 7: 0.325s + [OK] User 54 request 7: 0.324s + [OK] User 93 request 7: 0.324s + [OK] User 83 request 7: 0.324s + [OK] User 91 request 7: 0.324s + [OK] User 55 request 7: 0.324s + [OK] User 99 request 7: 0.324s + [OK] User 73 request 7: 0.324s + [OK] User 51 request 7: 0.324s + [OK] User 61 request 7: 0.324s + [OK] User 71 request 7: 0.324s + [OK] User 50 request 7: 0.323s + [OK] User 53 request 7: 0.322s + [OK] User 14 request 7: 0.323s + [OK] User 69 request 7: 0.323s + [OK] User 52 request 7: 0.323s + [OK] User 19 request 7: 0.322s + [OK] User 84 request 7: 0.323s + [OK] User 94 request 7: 0.323s + [OK] User 86 request 7: 0.323s + [OK] User 1 request 7: 0.323s + [OK] User 98 request 8: 0.318s + [OK] User 88 request 7: 0.318s + [OK] User 40 request 7: 0.318s + [OK] User 45 request 7: 0.318s + [OK] User 100 request 7: 0.318s + [OK] User 37 request 7: 0.319s + [OK] User 39 request 7: 0.319s + [OK] User 46 request 7: 0.319s + [OK] User 6 request 7: 0.319s + [OK] User 30 request 7: 0.317s + [OK] User 20 request 7: 0.583s + [OK] User 10 request 7: 0.522s + [OK] User 63 request 7: 0.319s + [OK] User 70 request 7: 0.319s + [OK] User 21 request 7: 0.319s + [OK] User 81 request 7: 0.317s + [OK] User 78 request 6: 0.317s + [OK] User 16 request 7: 0.317s + [OK] User 18 request 8: 0.317s + [OK] User 96 request 8: 0.317s + [OK] User 68 request 7: 0.317s + [OK] User 17 request 8: 0.317s + [OK] User 27 request 8: 0.316s + [OK] User 36 request 8: 0.316s + [OK] User 57 request 7: 0.316s + [OK] User 3 request 8: 0.316s + [OK] User 22 request 8: 0.313s + [OK] User 15 request 8: 0.313s + [OK] User 12 request 8: 0.313s + [OK] User 2 request 8: 0.313s + [OK] User 59 request 10: 0.324s + [OK] User 31 request 8: 0.324s + [OK] User 26 request 8: 0.324s + [OK] User 34 request 8: 0.324s + [OK] User 25 request 8: 0.324s + [OK] User 87 request 8: 0.324s + [OK] User 67 request 8: 0.324s + [OK] User 47 request 8: 0.322s + [OK] User 32 request 8: 0.322s + [OK] User 43 request 8: 0.322s + [OK] User 33 request 8: 0.322s + [OK] User 49 request 8: 0.322s + [OK] User 48 request 8: 0.322s + [OK] User 7 request 8: 0.323s + [OK] User 41 request 8: 0.322s + [OK] User 38 request 8: 0.322s + [OK] User 35 request 8: 0.321s + [OK] User 9 request 8: 0.321s + [OK] User 44 request 8: 0.321s + [OK] User 5 request 8: 0.321s + [OK] User 74 request 8: 0.326s + [OK] User 72 request 8: 0.326s + [OK] User 42 request 8: 0.326s + [OK] User 64 request 8: 0.326s + [OK] User 11 request 8: 0.325s + [OK] User 85 request 8: 0.326s + [OK] User 90 request 8: 0.325s + [OK] User 95 request 8: 0.325s + [OK] User 79 request 8: 0.325s + [OK] User 77 request 8: 0.325s + [OK] User 75 request 7: 0.325s + [OK] User 29 request 8: 0.325s + [OK] User 82 request 8: 0.325s + [OK] User 56 request 8: 0.325s + [OK] User 60 request 8: 0.325s + [OK] User 76 request 8: 0.325s + [OK] User 65 request 8: 0.325s + [OK] User 66 request 8: 0.325s + [OK] User 80 request 8: 0.326s + [OK] User 92 request 8: 0.325s + [OK] User 89 request 5: 0.325s + [OK] User 97 request 8: 0.328s + [OK] User 23 request 8: 0.329s + [OK] User 58 request 8: 0.328s + [OK] User 8 request 8: 0.328s + [OK] User 62 request 8: 0.328s + [OK] User 54 request 8: 0.329s + [OK] User 93 request 8: 0.329s + [OK] User 83 request 8: 0.329s + [OK] User 91 request 8: 0.329s + [OK] User 55 request 8: 0.329s + [OK] User 99 request 8: 0.329s + [OK] User 73 request 8: 0.329s + [OK] User 51 request 8: 0.329s + [OK] User 61 request 8: 0.329s + [OK] User 71 request 8: 0.329s + [OK] User 50 request 8: 0.326s + [OK] User 53 request 8: 0.326s + [OK] User 14 request 8: 0.326s + [OK] User 69 request 8: 0.326s + [OK] User 52 request 8: 0.326s + [OK] User 19 request 8: 0.326s + [OK] User 84 request 8: 0.326s + [OK] User 94 request 8: 0.326s + [OK] User 86 request 8: 0.326s + [OK] User 1 request 8: 0.326s + [OK] User 98 request 9: 0.329s + [OK] User 88 request 8: 0.329s + [OK] User 40 request 8: 0.329s + [OK] User 45 request 8: 0.329s + [OK] User 100 request 8: 0.329s + [OK] User 37 request 8: 0.329s + [OK] User 39 request 8: 0.329s + [OK] User 46 request 8: 0.329s + [OK] User 6 request 8: 0.329s + [OK] User 30 request 8: 0.329s + [OK] User 20 request 8: 0.327s + [OK] User 10 request 8: 0.327s + [OK] User 13 request 8: 0.560s + [OK] User 63 request 8: 0.328s + [OK] User 70 request 8: 0.328s + [OK] User 21 request 8: 0.328s + [OK] User 81 request 8: 0.326s + [OK] User 78 request 7: 0.326s + [OK] User 16 request 8: 0.326s + [OK] User 28 request 8: 0.599s + [OK] User 96 request 9: 0.323s + [OK] User 68 request 8: 0.323s + [OK] User 17 request 9: 0.321s + [OK] User 27 request 9: 0.322s + [OK] User 36 request 9: 0.322s + [OK] User 57 request 8: 0.322s + [OK] User 3 request 9: 0.322s + [OK] User 22 request 9: 0.319s + [OK] User 15 request 9: 0.319s + [OK] User 12 request 9: 0.319s + [OK] User 2 request 9: 0.318s + [OK] User 31 request 9: 0.312s + [OK] User 34 request 9: 0.311s + [OK] User 25 request 9: 0.312s + [OK] User 87 request 9: 0.312s + [OK] User 67 request 9: 0.312s + [OK] User 47 request 9: 0.311s + [OK] User 32 request 9: 0.311s + [OK] User 43 request 9: 0.311s + [OK] User 33 request 9: 0.311s + [OK] User 49 request 9: 0.311s + [OK] User 48 request 9: 0.311s + [OK] User 7 request 9: 0.311s + [OK] User 41 request 9: 0.311s + [OK] User 38 request 9: 0.311s + [OK] User 35 request 9: 0.318s + [OK] User 9 request 9: 0.318s + [OK] User 44 request 9: 0.318s + [OK] User 5 request 9: 0.318s + [OK] User 74 request 9: 0.315s + [OK] User 72 request 9: 0.315s + [OK] User 42 request 9: 0.315s + [OK] User 64 request 9: 0.316s + [OK] User 11 request 9: 0.316s + [OK] User 85 request 9: 0.315s + [OK] User 90 request 9: 0.315s + [OK] User 95 request 9: 0.315s + [OK] User 79 request 9: 0.315s + [OK] User 77 request 9: 0.316s + [OK] User 75 request 8: 0.316s + [OK] User 29 request 9: 0.316s + [OK] User 82 request 9: 0.320s + [OK] User 56 request 9: 0.320s + [OK] User 60 request 9: 0.320s + [OK] User 76 request 9: 0.320s + [OK] User 65 request 9: 0.320s + [OK] User 66 request 9: 0.320s + [OK] User 80 request 9: 0.320s + [OK] User 92 request 9: 0.320s + [OK] User 89 request 6: 0.320s + [OK] User 97 request 9: 0.318s + [OK] User 23 request 9: 0.318s + [OK] User 58 request 9: 0.318s + [OK] User 8 request 9: 0.318s + [OK] User 62 request 9: 0.318s + [OK] User 54 request 9: 0.318s + [OK] User 93 request 9: 0.317s + [OK] User 83 request 9: 0.317s + [OK] User 91 request 9: 0.318s + [OK] User 55 request 9: 0.318s + [OK] User 99 request 9: 0.318s + [OK] User 73 request 9: 0.318s + [OK] User 51 request 9: 0.318s + [OK] User 61 request 9: 0.318s + [OK] User 71 request 9: 0.318s + [OK] User 50 request 9: 0.319s + [OK] User 53 request 9: 0.319s + [OK] User 14 request 9: 0.319s + [OK] User 69 request 9: 0.319s + [OK] User 52 request 9: 0.319s + [OK] User 19 request 9: 0.319s + [OK] User 84 request 9: 0.319s + [OK] User 94 request 9: 0.319s + [OK] User 86 request 9: 0.319s + [OK] User 1 request 9: 0.319s + [OK] User 98 request 10: 0.317s + [OK] User 88 request 9: 0.317s + [OK] User 40 request 9: 0.317s + [OK] User 45 request 9: 0.317s + [OK] User 100 request 9: 0.317s + [OK] User 37 request 9: 0.317s + [OK] User 39 request 9: 0.317s + [OK] User 46 request 9: 0.317s + [OK] User 6 request 9: 0.317s + [OK] User 30 request 9: 0.317s + [OK] User 20 request 9: 0.317s + [OK] User 10 request 9: 0.317s + [OK] User 13 request 9: 0.315s + [OK] User 18 request 9: 0.584s + [OK] User 63 request 9: 0.320s + [OK] User 70 request 9: 0.320s + [OK] User 21 request 9: 0.320s + [OK] User 81 request 9: 0.318s + [OK] User 78 request 8: 0.318s + [OK] User 16 request 9: 0.318s + [OK] User 28 request 9: 0.316s + [OK] User 26 request 9: 0.535s + [OK] User 96 request 10: 0.318s + [OK] User 68 request 9: 0.318s + [OK] User 17 request 10: 0.319s + [OK] User 27 request 10: 0.318s + [OK] User 36 request 10: 0.318s + [OK] User 57 request 9: 0.318s + [OK] User 3 request 10: 0.318s + [OK] User 22 request 10: 0.317s + [OK] User 15 request 10: 0.316s + [OK] User 12 request 10: 0.316s + [OK] User 2 request 10: 0.316s + [OK] User 31 request 10: 0.313s + [OK] User 34 request 10: 0.313s + [OK] User 87 request 10: 0.312s + [OK] User 67 request 10: 0.312s + [OK] User 47 request 10: 0.313s + [OK] User 32 request 10: 0.313s + [OK] User 43 request 10: 0.313s + [OK] User 33 request 10: 0.312s + [OK] User 49 request 10: 0.312s + [OK] User 48 request 10: 0.312s + [OK] User 7 request 10: 0.312s + [OK] User 41 request 10: 0.312s + [OK] User 38 request 10: 0.311s + [OK] User 35 request 10: 0.302s + [OK] User 44 request 10: 0.302s + [OK] User 5 request 10: 0.302s + [OK] User 74 request 10: 0.303s + [OK] User 72 request 10: 0.303s + [OK] User 42 request 10: 0.303s + [OK] User 64 request 10: 0.303s + [OK] User 11 request 10: 0.302s + [OK] User 85 request 10: 0.302s + [OK] User 90 request 10: 0.302s + [OK] User 95 request 10: 0.302s + [OK] User 79 request 10: 0.302s + [OK] User 77 request 10: 0.301s + [OK] User 75 request 9: 0.301s + [OK] User 29 request 10: 0.302s + [OK] User 82 request 10: 0.299s + [OK] User 56 request 10: 0.299s + [OK] User 60 request 10: 0.299s + [OK] User 76 request 10: 0.299s + [OK] User 65 request 10: 0.299s + [OK] User 66 request 10: 0.299s + [OK] User 80 request 10: 0.298s + [OK] User 92 request 10: 0.298s + [OK] User 89 request 7: 0.298s + [OK] User 97 request 10: 0.296s + [OK] User 23 request 10: 0.296s + [OK] User 58 request 10: 0.295s + [OK] User 8 request 10: 0.295s + [OK] User 62 request 10: 0.295s + [OK] User 54 request 10: 0.295s + [OK] User 93 request 10: 0.295s + [OK] User 83 request 10: 0.295s + [OK] User 91 request 10: 0.295s + [OK] User 55 request 10: 0.295s + [OK] User 99 request 10: 0.295s + [OK] User 73 request 10: 0.295s + [OK] User 51 request 10: 0.295s + [OK] User 61 request 10: 0.295s + [OK] User 71 request 10: 0.295s + [OK] User 50 request 10: 0.291s + [OK] User 53 request 10: 0.291s + [OK] User 14 request 10: 0.290s + [OK] User 69 request 10: 0.290s + [OK] User 52 request 10: 0.290s + [OK] User 19 request 10: 0.290s + [OK] User 84 request 10: 0.290s + [OK] User 94 request 10: 0.290s + [OK] User 86 request 10: 0.290s + [OK] User 1 request 10: 0.290s + [OK] User 88 request 10: 0.283s + [OK] User 40 request 10: 0.283s + [OK] User 45 request 10: 0.283s + [OK] User 100 request 10: 0.283s + [OK] User 37 request 10: 0.283s + [OK] User 39 request 10: 0.283s + [OK] User 46 request 10: 0.283s + [OK] User 6 request 10: 0.283s + [OK] User 30 request 10: 0.283s + [OK] User 20 request 10: 0.282s + [OK] User 10 request 10: 0.282s + [OK] User 13 request 10: 0.282s + [OK] User 18 request 10: 0.281s + [OK] User 63 request 10: 0.266s + [OK] User 70 request 10: 0.266s + [OK] User 21 request 10: 0.266s + [OK] User 81 request 10: 0.252s + [OK] User 78 request 9: 0.252s + [OK] User 16 request 10: 0.252s + [OK] User 28 request 10: 0.252s + [OK] User 26 request 10: 0.251s + [OK] User 25 request 10: 0.474s + [OK] User 68 request 10: 0.239s + [OK] User 57 request 10: 0.221s + [OK] User 9 request 10: 0.416s + [OK] User 75 request 10: 0.096s + [OK] User 89 request 8: 0.077s + [OK] User 78 request 10: 0.027s + [OK] User 89 request 9: 0.008s + [OK] User 89 request 10: 0.004s + +[OK] 1000 succeeded, [FAIL] 0 failed + +Latency (successful): + min: 0.004s + avg: 1.198s + max: 15.483s + p95: 8.839s + p99: 14.283s + +Requests above threshold (successful only): + Above 1s: 100/1000 (10.0%) + Above 2s: 100/1000 (10.0%) + Above 3s: 93/1000 (9.3%) + Above 4s: 87/1000 (8.7%) + Above 5s: 78/1000 (7.8%) + Above 6s: 71/1000 (7.1%) + Above 7s: 63/1000 (6.3%) + Above 8s: 57/1000 (5.7%) + Above 9s: 48/1000 (4.8%) + Above 10s: 42/1000 (4.2%) diff --git a/callbacks_off_1_per_user_created.txt b/callbacks_off_1_per_user_created.txt new file mode 100644 index 00000000000..6c6aad33722 --- /dev/null +++ b/callbacks_off_1_per_user_created.txt @@ -0,0 +1,128 @@ +Creating 100 keys via /key/generate... +Created 100 keys. +Creating 100 end users via /customer/new... +Created 100 end users. +=== 100 requests, 100 users, fire-as-fast-as-possible === +User mode: created, Key mode: per_user +Latency = time from request start until last byte of response received + [OK] User 33 request 1: 11.071s + [OK] User 9 request 1: 14.408s + [OK] User 55 request 1: 8.065s + [OK] User 18 request 1: 13.189s + [OK] User 31 request 1: 11.393s + [OK] User 6 request 1: 14.864s + [OK] User 23 request 1: 12.506s + [OK] User 10 request 1: 14.330s + [OK] User 17 request 1: 13.356s + [OK] User 67 request 1: 6.459s + [OK] User 24 request 1: 12.404s + [OK] User 8 request 1: 14.644s + [OK] User 13 request 1: 13.961s + [OK] User 2 request 1: 15.497s + [OK] User 3 request 1: 15.367s + [OK] User 12 request 1: 14.116s + [OK] User 15 request 1: 13.713s + [OK] User 71 request 1: 5.974s + [OK] User 22 request 1: 12.739s + [OK] User 30 request 1: 11.649s + [OK] User 35 request 1: 10.953s + [OK] User 28 request 1: 11.925s + [OK] User 70 request 1: 6.128s + [OK] User 29 request 1: 11.788s + [OK] User 57 request 1: 7.917s + [OK] User 52 request 1: 8.609s + [OK] User 79 request 1: 4.902s + [OK] User 11 request 1: 14.304s + [OK] User 78 request 1: 5.043s + [OK] User 32 request 1: 11.391s + [OK] User 77 request 1: 5.183s + [OK] User 73 request 1: 5.735s + [OK] User 86 request 1: 3.936s + [OK] User 84 request 1: 4.232s + [OK] User 90 request 1: 3.399s + [OK] User 82 request 1: 4.529s + [OK] User 4 request 1: 15.337s + [OK] User 26 request 1: 12.282s + [OK] User 72 request 1: 5.931s + [OK] User 34 request 1: 11.172s + [OK] User 20 request 1: 13.116s + [OK] User 1 request 1: 15.774s + [OK] User 25 request 1: 12.439s + [OK] User 27 request 1: 12.163s + [OK] User 54 request 1: 8.428s + [OK] User 14 request 1: 13.966s + [OK] User 99 request 1: 2.212s + [OK] User 97 request 1: 2.489s + [OK] User 21 request 1: 12.992s + [OK] User 96 request 1: 2.631s + [OK] User 91 request 1: 3.324s + [OK] User 7 request 1: 14.962s + [OK] User 89 request 1: 3.617s + [OK] User 68 request 1: 6.524s + [OK] User 60 request 1: 7.626s + [OK] User 38 request 1: 10.657s + [OK] User 59 request 1: 7.764s + [OK] User 69 request 1: 6.386s + [OK] User 44 request 1: 9.827s + [OK] User 63 request 1: 7.215s + [OK] User 45 request 1: 9.702s + [OK] User 88 request 1: 3.803s + [OK] User 19 request 1: 13.338s + [OK] User 40 request 1: 10.433s + [OK] User 49 request 1: 9.191s + [OK] User 75 request 1: 5.613s + [OK] User 56 request 1: 8.228s + [OK] User 39 request 1: 10.584s + [OK] User 53 request 1: 8.653s + [OK] User 58 request 1: 7.967s + [OK] User 48 request 1: 9.342s + [OK] User 41 request 1: 10.308s + [OK] User 37 request 1: 10.863s + [OK] User 74 request 1: 5.764s + [OK] User 61 request 1: 7.556s + [OK] User 16 request 1: 13.780s + [OK] User 85 request 1: 4.243s + [OK] User 43 request 1: 10.041s + [OK] User 36 request 1: 11.013s + [OK] User 46 request 1: 9.628s + [OK] User 51 request 1: 8.939s + [OK] User 47 request 1: 9.491s + [OK] User 42 request 1: 10.180s + [OK] User 81 request 1: 4.806s + [OK] User 62 request 1: 7.432s + [OK] User 94 request 1: 3.010s + [OK] User 66 request 1: 6.885s + [OK] User 92 request 1: 3.287s + [OK] User 95 request 1: 2.871s + [OK] User 98 request 1: 2.456s + [OK] User 80 request 1: 4.949s + [OK] User 87 request 1: 3.980s + [OK] User 83 request 1: 4.552s + [OK] User 64 request 1: 7.229s + [OK] User 5 request 1: 15.457s + [OK] User 100 request 1: 2.317s + [OK] User 76 request 1: 5.767s + [OK] User 50 request 1: 9.364s + [OK] User 65 request 1: 7.319s + [OK] User 93 request 1: 3.485s + +[OK] 100 succeeded, [FAIL] 0 failed + +Latency (successful): + min: 2.212s + avg: 8.923s + max: 15.774s + p95: 14.962s + p99: 15.497s + +Requests above threshold (successful only): + Above 1s: 100/100 (100.0%) + Above 2s: 100/100 (100.0%) + Above 3s: 94/100 (94.0%) + Above 4s: 85/100 (85.0%) + Above 5s: 78/100 (78.0%) + Above 6s: 70/100 (70.0%) + Above 7s: 65/100 (65.0%) + Above 8s: 56/100 (56.0%) + Above 9s: 50/100 (50.0%) + Above 10s: 43/100 (43.0%) diff --git a/measure_latency.py b/measure_latency.py index 032eea1b813..d983230b908 100644 --- a/measure_latency.py +++ b/measure_latency.py @@ -33,8 +33,8 @@ KEY_MODES (add-on; can be combined with any user mode): BASE_URL = "http://localhost:4000" API_KEY = "sk-1234" MODEL = "gpt-o1" # must match a model_name in your proxy config (e.g. test_config_123.yaml) -NUM_REQUESTS = 10 -NUM_CONCURRENT = 1 # Concurrent users; each has one connection, reuses it for their requests +NUM_REQUESTS = 1000 +NUM_CONCURRENT = 100 # Concurrent users; each has one connection, reuses it for their requests MESSAGES = [{"role": "user", "content": "Say hello in one word."}] TIMEOUT = 30000.0 # ----------------------------------------------------------------------------- diff --git a/problem_tracker.md b/problem_tracker.md index ad4be460d5d..8434d3b0951 100644 --- a/problem_tracker.md +++ b/problem_tracker.md @@ -1,4 +1,4 @@ -### Context +### Context - Problem 1 **Issue:** Prometheus configured as a callback causes orders-of-magnitude worse performance (reported by Zurich). @@ -261,7 +261,7 @@ The issue was caused by hitting the database on every request when Prometheus wa ### Next Steps -**Context:** Callbacks on vs off no longer affects latency (Prometheus fix). The remaining baseline spikes are likely from end_user lookups, provider latency, or other bottlenecks. Comparing user modes will isolate whether passing `user` (and cache behavior) contributes. +**Context - Problem 2:** Callbacks on vs off no longer affects latency (Prometheus fix). The remaining baseline spikes are likely from end_user lookups, provider latency, or other bottlenecks. Comparing user modes will isolate whether passing `user` (and cache behavior) contributes. **Measure baseline latency with each user mode** (run `measure_latency.py`): @@ -326,6 +326,16 @@ Passing `user` in the request payload spikes latency up to ~7×. Using a shared - [x] Run with `--user-mode random`, 100 requests, 100 users (1 per user) → `callbacks_off_1_per_user.txt` - [x] Run with `--user-mode random`, 1000 requests, 100 users (10 per user) → `callbacks_off_10_per_user.txt` +**Same scenarios with `--user-mode created --key-mode per_user`** (happy path: pre-created end users): + +| Scenario | NUM_REQUESTS | NUM_CONCURRENT | Req/user | Output file | +|-------------------|--------------|----------------|----------|---------------------------------------------| +| One per user | 100 | 100 | 1 | `callbacks_off_1_per_user_created.txt` | +| Multiple per user | 1000 | 100 | 10 | `callbacks_off_10_per_user_created.txt` | + +- [x] Run with `--user-mode created --key-mode per_user`, 100 requests, 100 users (1 per user) → `callbacks_off_1_per_user_created.txt` +- [x] Run with `--user-mode created --key-mode per_user`, 1000 requests, 100 users (10 per user) → `callbacks_off_10_per_user_created.txt` + #### Analysis (1 vs 10 requests per user, callbacks off, `--user-mode random`) | Scenario | Requests | Users | Req/user | avg | p95 | Above 1s | @@ -337,6 +347,17 @@ Passing `user` in the request payload spikes latency up to ~7×. Using a shared With **1 request per user**, every request triggers an end_user cache miss → DB lookup. Latency is ~5.6× higher (avg 8.6s vs 1.5s) and 100% of requests exceed 1s vs 33% with 10 per user. With **10 requests per user**, the first request per user misses cache; the next 9 hit cache. Cache hits on subsequent requests dramatically reduce latency. This confirms that end_user lookups (especially cache misses) are a major driver of baseline latency spikes. +#### Analysis (1 vs 10 requests per user, callbacks off, `--user-mode created --key-mode per_user`) + +| Scenario | Requests | Users | Req/user | avg | p95 | Above 1s | +|-------------------|----------|-------|----------|--------|---------|----------| +| One per user | 100 | 100 | 1 | 8.923s | 14.962s | 100.0% | +| Multiple per user | 1000 | 100 | 10 | 1.198s | 8.839s | 10.0% | + +**Findings** + +Same pattern as random: 1 per user → all cache misses, ~7.5× higher avg latency and 100% above 1s; 10 per user → first request misses, next 9 hit, avg 1.2s and only 10% above 1s. **Created mode performs better than random** at 10 per user: avg 1.2s vs 1.5s, and 10% vs 33% above 1s. With pre-created end users, lookups succeed and cache; with random (non-existent) IDs, the negative path (no negative caching) keeps hitting DB. The 100 requests above 1s in created 10-per-user are the first request per user; the remaining 900 are cache hits with sub-second latency. + ### Next Steps - [x] Run proxy + `measure_latency.py --user-mode random --key-mode shared`, capture proxy stdout → `end_user_profile_proxy_logs.txt`