diff --git a/docs/my-website/docs/benchmarks.md b/docs/my-website/docs/benchmarks.md index f60fa4fcd14..40285b63d71 100644 --- a/docs/my-website/docs/benchmarks.md +++ b/docs/my-website/docs/benchmarks.md @@ -48,6 +48,47 @@ In these tests the baseline latency characteristics are measured against a fake- - High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms. - Setting workers equal to CPU count gives optimal performance. +## LiteLLM vs Portkey Performance Comparison + +**Test Configuration**: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up + +### Multi-Instance (4×) Performance + +| Metric | Portkey (no DB) | LiteLLM (with DB) | +| ------------------- | --------------- | ----------------- | +| **Total Requests** | 293,796 | 312,405 | +| **Failed Requests** | 0 | 0 | +| **Median Latency** | 100 ms | 100 ms | +| **p95 Latency** | 230 ms | 150 ms | +| **p99 Latency** | 500 ms | 240 ms | +| **Average Latency** | 123 ms | 111 ms | +| **Current RPS** | 1,170.9 | 1,170 | + +### Technical Insights + +**Portkey** + +**Pros** + +* Low memory footprint +* Stable latency with minimal spikes + +**Cons** + +* CPU utilization capped around ~40%, indicating underutilization of available compute resources +* Experienced three I/O timeout outages + +**LiteLLM** + +**Pros** + +* Fully utilizes available CPU capacity +* Strong connection handling and low latency after initial warm-up spikes + +**Cons** + +* High memory usage during initialization and per request + ## Machine Spec used for testing Each machine deploying LiteLLM had the following specs: