scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102

docs: add performance improvements section to v1.80.14 release notes

Browse source 
Add new performance section highlighting 50% overhead reduction and 2.5x throughput improvement with before/after benchmark statistics.
This commit is contained in:
Alexsander Hamir 2026-01-10 16:00:33 -08:00
parent 58bc308d47
commit c305e12cd1
1 changed files with 50 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

50
docs/my-website/release_notes/v1.80.14/index.md
View file

@ -57,6 +57,56 @@ pip install litellm==1.80.14
- **UI Usage - Endpoint Activity** - Users can now see Endpoint Activity Metrics in the UI.
---
## Performance - 50% Overhead Reduction
LiteLLM now sends 2.5× more requests to LLM providers by reducing wrapper function overhead across all endpoints.
### Before
```
============================================================
Fake LLM Provier Stats (When called by LiteLLM)
============================================================
Requests/Second: 10746.68
Latency Statistics (seconds):
Mean: 0.2039s
Median (p50): 0.2310s
Min: 0.0323s
Max: 0.3928s
Std Dev: 0.1166s
p95: 0.3574s
p99: 0.3748s
Status Codes:
200: 6000
```
### After
```
============================================================
Fake LLM Provier Stats (When called by LiteLLM)
============================================================
Requests/Second: 4224.49
Latency Statistics (seconds):
Mean: 0.5300s
Median (p50): 0.5871s
Min: 0.0885s
Max: 1.0482s
Std Dev: 0.3065s
p95: 0.9750s
p99: 1.0444s
Status Codes:
200: 6000
```
> The benchmarks run LiteLLM locally with a lightweight LLM provider to eliminate network latency, isolating internal overhead and bottlenecks so we can focus on reducing pure LiteLLM overhead on a single instance.
---
### UI Usage - Endpoint Activity