mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
fix 1.77.7 stable
This commit is contained in:
parent
4e88e21266
commit
525b86687c
3 changed files with 5 additions and 1 deletions
BIN
docs/my-website/img/release_notes/perf_77_5.png
Normal file
BIN
docs/my-website/img/release_notes/perf_77_5.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
|
|
@ -59,6 +59,8 @@ pip install litellm==1.77.5
|
|||
|
||||
### Performance Improvements - 54% RPS Improvement
|
||||
|
||||
<Image img={require('../../img/release_notes/perf_77_5.png')} style={{ width: '800px', height: 'auto' }} />
|
||||
|
||||
Throughput increased by 54% (1,040 → 1,602 RPS, aggregated) per instance while maintaining a 40 ms median overhead. The improvement comes from fixing major O(n²) inefficiencies in the router, primarily caused by repeated use of in statements inside loops over large arrays. Tests were run with a database-only setup (no cache hits). As a result, p95 latency improved by 30% (2,700 → 1,900 ms), enhancing overall stability and scalability under heavy load.
|
||||
|
||||
#### Test Setup
|
||||
|
|
|
|||
|
|
@ -67,7 +67,9 @@ pip install litellm==1.77.7.rc.1
|
|||
|
||||
### 2.9x Lower Median Latency
|
||||
|
||||
<Image img={require('../../img/perf_77_7.png')} style={{ width: '800px', height: 'auto' }} />
|
||||
<Image img={require('../../img/release_notes/perf_77_7.png')} style={{ width: '800px', height: 'auto' }} />
|
||||
|
||||
<br/>
|
||||
|
||||
This update removes LiteLLM router inefficiencies, reducing complexity from O(M×N) to O(1). Previously, it built a new array and ran repeated checks like data["model"] in llm_router.get_model_ids(). Now, a direct ID-to-deployment map eliminates redundant allocations and scans.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue