Realtime API benchmarks (#20074)

* Add /realtime API benchmarks to Benchmarks documentation

- Added new section showing performance improvements for /realtime endpoint
- Included before/after metrics showing 182× faster p99 latency
- Added test setup specifications and key optimizations
- Referenced from v1.80.5-stable release notes

Co-authored-by: ishaan <ishaan@berri.ai>

* Update /realtime benchmarks to show current performance only

- Removed before/after comparison, showing only current metrics
- Clarified that benchmarks are e2e latency against fake realtime endpoint
- Simplified table format for better readability

Co-authored-by: ishaan <ishaan@berri.ai>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
This commit is contained in:
Ishaan Jaff 2026-01-30 11:55:29 -08:00 committed by GitHub
parent 0acfcb494b
commit 3910161a02
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -48,6 +48,28 @@ In these tests the baseline latency characteristics are measured against a fake-
- High-percentile latencies drop significantly: P95 630ms → 150ms, P99 1,200ms → 240ms.
- Setting workers equal to CPU count gives optimal performance.
## `/realtime` API Benchmarks
End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake realtime endpoint.
### Performance Metrics
| Metric | Value |
| --------------- | ---------- |
| Median latency | 59 ms |
| p95 latency | 67 ms |
| p99 latency | 99 ms |
| Average latency | 63 ms |
| RPS | 1,207 |
### Test Setup
| Category | Specification |
|----------|---------------|
| **Load Testing** | Locust: 1,000 concurrent users, 500 ramp-up |
| **System** | 4 vCPUs, 8 GB RAM, 4 workers, 4 instances |
| **Database** | PostgreSQL (Redis unused) |
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs: