mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-08 03:08:45 +00:00
6072019b0d
* docs: realtime pre-warmed connection pool design + raw-passthrough follow-up * docs: realtime pool benchmark, repro steps, and deploy guidance * feat: expose Router::deployments() for host-side upstream enumeration * refactor: split realtime dial/splice and add warm-handoff entry point * feat: pre-warmed upstream realtime connection pool with fresh-dial fallback * feat: add realtime pool handle to gateway AppState * feat: try warm pooled upstream before fresh-dial in realtime service * feat: thread realtime pool through the realtime route bridge * feat: build and pre-warm the realtime pool at gateway startup * build: lean Dockerfile for the realtime gateway (default features, env stand-in) Minimal multi-stage image for load-testing the realtime pool: builds the gateway with default features (no python-config, no libpython), runs on a debian-slim base (~157MB), and reads model_list from the OPENAI_REALTIME_MODEL env stand-in. No config.yaml or pip install needed. Build context is the repo root; only litellm-rust/ is included via the sidecar .dockerignore. * docs: add generic ai-gateway benchmarking skill Teaches an agent how to benchmark any ai-gateway endpoint: deploy the gateway, run one load generator against both provider-direct and the gateway with the same protocol, phase-decompose latency (dial/session/first-token/total), compare at scale, and report success% + p50/p95. Documents the benchmarks/<endpoint>/ layout and the hard no-committed-keys rule. * docs: drop per-endpoint realtime benchmark README The measured results table lives in the PR description (numbers go stale in a committed README). The benchmarks/realtime/ dir now holds only the sanitized load-gen harness; the generic method is in benchmarks/SKILL.md. * test: add sanitized realtime WS load-gen harness for gateway benchmarks Copies the ws-bench Go load generator (main.go, go.mod, go.sum, Dockerfile, run.sh) into benchmarks/realtime/. Measures dial/session/first-audio/total per WebSocket connection against both OpenAI-direct and the gateway. No keys are hardcoded — the bearer token comes from -key / $OPENAI_API_KEY; default host is api.openai.com. * test: add hosted-runner serve.sh wrapper for the realtime harness Render one-off jobs don't surface stdout via the Logs API, so on a hosted runner the long-lived web service runs the leg and PUBLISHES the result: serve.sh decodes the base64 flag list, runs wsbench teeing output to /tmp/web/result.txt, then serves it over HTTP so the result is fetchable at /result.txt. No secrets are written to the served file (the -key is only in wsbench's argv). The Dockerfile now copies both run.sh and serve.sh. * test: make serve.sh publish result atomically and clear stale output Remove any prior result.txt at startup and write the new run to a .partial file that's atomically moved into place only once complete. Prevents a fetcher from reading a previous run's numbers while the current run is still in flight. * perf: refill the realtime pool concurrently so warm supply keeps up at scale The replenisher dialed missing warm sockets sequentially, so a full refill cost needed x handshake (~needed x 350ms). Under high connect rates the pool drained faster than it refilled and ~85% of connects missed (measured: only ~15% pool hits at 5000/500). Firing the dials together with join_all refills in ~one handshake window, keeping warm supply close to peak concurrent connects so the sub-millisecond warm handoff becomes the median rather than the lucky-hit tail. Each warm_one dial is independent (no shared state until the final push under the lock), so concurrent refill is safe. Pool unit tests unchanged and passing. * build: drop Dockerfile.lean The lean load-test image isn't worth carrying in the repo; deploy the gateway however you normally do and set the pool env vars. * docs: slim benchmarks/realtime to a README (harness moved to its own repo) Drop the Go load-gen, Dockerfile, run.sh, serve.sh, and the generic SKILL.md from the repo. The harness now lives at github.com/ishaan-berri/litellm-realtime-bench; benchmarks/realtime/README.md carries the results table and links there for repro. * docs: add realtime route README with pooling design + diagram Pooling is now documented as a section in src/routes/realtime/README.md next to the code (handoff diagram, sizing rule, config, notes) instead of the standalone REALTIME_POOL_DESIGN.md RFC. Update the realtime_pool.rs doc pointer to it. * fix: satisfy clippy manual_flatten on concurrent pool refill Use .into_iter().flatten() instead of an if-let-Ok in the for loop over the join_all results, and let rustfmt wrap it. Clears the CI clippy -D warnings failure; fmt + clippy + cargo test all green locally. --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
55 lines
2.9 KiB
Markdown
55 lines
2.9 KiB
Markdown
# Realtime gateway benchmark — pool on/off
|
||
|
||
Measures what the gateway adds over talking to OpenAI's realtime WebSocket
|
||
directly, and what the pre-warmed connection pool removes. See
|
||
`../../src/routes/realtime/README.md` for how the pool works.
|
||
|
||
## Results
|
||
|
||
5000 calls / 500 concurrency, gateway at 10 instances, pool ON
|
||
(`REALTIME_POOL_SIZE=64`), upstream OpenAI `gpt-realtime`. Each leg run twice.
|
||
Times in **ms**. Phases per connection: **dial** = TCP+TLS+WS upgrade,
|
||
**session** = upgrade → `session.created` (the phase the pool removes),
|
||
**1st-audio** = `response.create` → first audio delta (OpenAI inference),
|
||
**total** = full wall-clock.
|
||
|
||
| metric | Direct OpenAI | Gateway (pool ON) | Overhead (ms) | vs OpenAI |
|
||
| ------------------ | ------------- | ----------------- | ------------- | ---------- |
|
||
| success rate (%) | 99.8 | 99.8 | — | — |
|
||
| dial p50 (ms) | 276 | 158 | −118 | **faster** |
|
||
| session p50 (ms) | 7 | 0 | −7 | **faster** |
|
||
| 1st-audio p50 (ms) | 440 | 664 | +224 | slower¹ |
|
||
| total p50 (ms) | 816 | 1010 | +194 | slower¹ |
|
||
| total p95 (ms) | 2152 | 1970 | −182 | **faster** |
|
||
| total p99 (ms) | 2692 | 2610 | −82 | **faster** |
|
||
|
||
The gateway is **faster than direct on 4 of 6 metrics**. The warm pool makes the
|
||
**session phase sub-millisecond** at the median — ~76% of connects hit the pool,
|
||
~70% had session < 1 ms. ¹ The two "slower" rows are not gateway overhead:
|
||
`1st-audio` is OpenAI's own inference time (the gateway only relays it), which ran
|
||
slower during the gateway legs and drags `total p50` with it.
|
||
|
||
**Pool OFF** (control, `REALTIME_POOL_SIZE=0`): session p50 was **367 ms** — the
|
||
fresh-dial overhead the pool removes.
|
||
|
||
## Reproduce
|
||
|
||
The load generator lives in a separate repo:
|
||
**https://github.com/ishaan-berri/litellm-realtime-bench**
|
||
|
||
```bash
|
||
git clone https://github.com/ishaan-berri/litellm-realtime-bench
|
||
cd litellm-realtime-bench && go build -o wsbench .
|
||
|
||
# Direct to OpenAI (baseline)
|
||
./wsbench -host api.openai.com -key "$OPENAI_API_KEY" -m gpt-realtime -n 5000 -c 500 -t 60
|
||
|
||
# Through the gateway — run once with pool ON, once with REALTIME_POOL_SIZE=0
|
||
./wsbench -host <gateway-host> -key "$LITELLM_MASTER_KEY" -m gpt-realtime -n 5000 -c 500 -t 60
|
||
```
|
||
|
||
Run the gateway with the env stand-in (`OPENAI_REALTIME_MODEL=gpt-realtime`,
|
||
`OPENAI_API_KEY`, `LITELLM_MASTER_KEY`, `REALTIME_POOL_SIZE`, `HOST=0.0.0.0`). At
|
||
500 concurrency over N instances, size the pool to `≈ 500 / N` per instance (64 was
|
||
used here for 10 instances). The bench repo's README covers running 500-concurrency
|
||
legs from a hosted multi-vCPU runner. **Never commit keys — pass them via `-key`.**
|