mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
[Doc] Perf: Last week improvement (#15193)
* doc: perf update * fix: mixed up changes * fix: add gist
This commit is contained in:
parent
ee36c30217
commit
29a31e17dd
1 changed files with 25 additions and 0 deletions
|
|
@ -51,6 +51,31 @@ pip install litellm==1.75.5.post2
|
|||
- **Digital Ocean's Gradient AI** - New LLM provider for calling models on Digital Ocean's Gradient AI platform.
|
||||
|
||||
|
||||
### 54% RPS Improvement
|
||||
|
||||
Throughput increased by 54% (1,040 → 1,602 RPS, aggregated) per instance while maintaining a 40 ms median overhead. The improvement comes from fixing major O(n²) inefficiencies in the router, primarily caused by repeated use of in statements inside loops over large arrays. Tests were run with a database-only setup (no cache hits). As a result, p95 latency improved by 30% (2,700 → 1,900 ms), enhancing overall stability and scalability under heavy load.
|
||||
|
||||
---
|
||||
|
||||
### Test Setup
|
||||
|
||||
All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.
|
||||
|
||||
**System Specs**
|
||||
|
||||
- **CPU:** 8 vCPUs
|
||||
- **Memory:** 32 GB RAM
|
||||
|
||||
**Configuration (config.yaml)**
|
||||
|
||||
View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
|
||||
|
||||
**Load Script (no_cache_hits.py)**
|
||||
|
||||
View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
|
||||
|
||||
---
|
||||
|
||||
### Risk of Upgrade
|
||||
|
||||
If you build the proxy from the pip package, you should hold off on upgrading. This version makes `prisma migrate deploy` our default for managing the DB. This is safer, as it doesn't reset the DB, but it requires a manual `prisma generate` step.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue