From 300a14292649cc61f524baac7eeadcdfa294cd45 Mon Sep 17 00:00:00 2001 From: Alexsander Hamir Date: Sat, 11 Oct 2025 17:38:13 -0700 Subject: [PATCH] [Add]: perf summary (#15458) * add: perf summary * update docs * fix: focus on p99 --- .../release_notes/v1.78.0-stable/index.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/my-website/release_notes/v1.78.0-stable/index.md b/docs/my-website/release_notes/v1.78.0-stable/index.md index 85ec6847244..0ea4008b96b 100644 --- a/docs/my-website/release_notes/v1.78.0-stable/index.md +++ b/docs/my-website/release_notes/v1.78.0-stable/index.md @@ -67,6 +67,51 @@ pip install litellm==1.78.0.rc.1 --- +## Performance + +This release cuts p99 latency by 70% on LiteLLM AI Gateway, making it even better for low-latency use cases. + +These gains come from two key enhancements: + +**Reliable Sessions** + +`shared_session` is now consistently used across all calls, resolving JSON errors, enabling connection pooling, and actively managing session lifecycles—detecting stale connections, retrying expired sessions, and preventing 'Session is closed' errors. + +**Faster Routing** + +A new `model_name_to_deployment_indices` hash map replaces O(n) list scans in `_get_all_deployments()` with O(1) hash lookups, boosting routing performance and scalability. + +As a result, performance improved across all latency percentiles: + +- **Median latency:** 110 ms → **100 ms** (−9.1%) +- **p95 latency:** 440 ms → **150 ms** (−65.9%) +- **p99 latency:** 810 ms → **240 ms** (−70.4%) +- **Average latency:** 310 ms → **111.73 ms** (−64.0%) + +### **Test Setup** + +**Locust** + +- **Concurrent users:** 1,000 +- **Ramp-up:** 500 + +**System Specs** + +- **Database was used** +- **CPU:** 4 vCPUs +- **Memory:** 8 GB RAM +- **LiteLLM Workers:** 4 +- **Instances**: 4 + +**Configuration (config.yaml)** + +View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4) + +**Load Script (no_cache_hits.py)** + +View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42) + +--- ### MCP Gateway - Control Tool Access by Team, Key