Add: benchmark comparison with other AI gateways (#16248)

* fix: benchmark comparison location & content

* add: pros and cons

* remove
This commit is contained in:
Alexsander Hamir 2025-11-05 15:44:30 -08:00 committed by GitHub
parent 3ef210e5f4
commit e06d1d623f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -48,6 +48,47 @@ In these tests the baseline latency characteristics are measured against a fake-
- High-percentile latencies drop significantly: P95 630ms → 150ms, P99 1,200ms → 240ms.
- Setting workers equal to CPU count gives optimal performance.
## LiteLLM vs Portkey Performance Comparison
**Test Configuration**: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up
### Multi-Instance (4×) Performance
| Metric | Portkey (no DB) | LiteLLM (with DB) |
| ------------------- | --------------- | ----------------- |
| **Total Requests** | 293,796 | 312,405 |
| **Failed Requests** | 0 | 0 |
| **Median Latency** | 100 ms | 100 ms |
| **p95 Latency** | 230 ms | 150 ms |
| **p99 Latency** | 500 ms | 240 ms |
| **Average Latency** | 123 ms | 111 ms |
| **Current RPS** | 1,170.9 | 1,170 |
### Technical Insights
**Portkey**
**Pros**
* Low memory footprint
* Stable latency with minimal spikes
**Cons**
* CPU utilization capped around ~40%, indicating underutilization of available compute resources
* Experienced three I/O timeout outages
**LiteLLM**
**Pros**
* Fully utilizes available CPU capacity
* Strong connection handling and low latency after initial warm-up spikes
**Cons**
* High memory usage during initialization and per request
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs: