From 29a31e17dd7e8234ffac9dd7f547aca852836584 Mon Sep 17 00:00:00 2001 From: Alexsander Hamir Date: Sat, 4 Oct 2025 10:43:13 -0700 Subject: [PATCH] [Doc] Perf: Last week improvement (#15193) * doc: perf update * fix: mixed up changes * fix: add gist --- .../release_notes/v1.75.5-stable/index.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/my-website/release_notes/v1.75.5-stable/index.md b/docs/my-website/release_notes/v1.75.5-stable/index.md index 270be64190e..da13906cd24 100644 --- a/docs/my-website/release_notes/v1.75.5-stable/index.md +++ b/docs/my-website/release_notes/v1.75.5-stable/index.md @@ -51,6 +51,31 @@ pip install litellm==1.75.5.post2 - **Digital Ocean's Gradient AI** - New LLM provider for calling models on Digital Ocean's Gradient AI platform. +### 54% RPS Improvement + +Throughput increased by 54% (1,040 → 1,602 RPS, aggregated) per instance while maintaining a 40 ms median overhead. The improvement comes from fixing major O(n²) inefficiencies in the router, primarily caused by repeated use of in statements inside loops over large arrays. Tests were run with a database-only setup (no cache hits). As a result, p95 latency improved by 30% (2,700 → 1,900 ms), enhancing overall stability and scalability under heavy load. + +--- + +### Test Setup + +All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable. + +**System Specs** + +- **CPU:** 8 vCPUs +- **Memory:** 32 GB RAM + +**Configuration (config.yaml)** + +View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4) + +**Load Script (no_cache_hits.py)** + +View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42) + +--- + ### Risk of Upgrade If you build the proxy from the pip package, you should hold off on upgrading. This version makes `prisma migrate deploy` our default for managing the DB. This is safer, as it doesn't reset the DB, but it requires a manual `prisma generate` step.