diff --git a/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md b/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md deleted file mode 100644 index 84e926af243..00000000000 --- a/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md +++ /dev/null @@ -1,55 +0,0 @@ -# Realtime gateway benchmark — pool on/off - -Measures what the gateway adds over talking to OpenAI's realtime WebSocket -directly, and what the pre-warmed connection pool removes. See -`../../src/routes/realtime/README.md` for how the pool works. - -## Results - -5000 calls / 500 concurrency, gateway at 10 instances, pool ON -(`REALTIME_POOL_SIZE=64`), upstream OpenAI `gpt-realtime`. Each leg run twice. -Times in **ms**. Phases per connection: **dial** = TCP+TLS+WS upgrade, -**session** = upgrade → `session.created` (the phase the pool removes), -**1st-audio** = `response.create` → first audio delta (OpenAI inference), -**total** = full wall-clock. - -| metric | Direct OpenAI | Gateway (pool ON) | Overhead (ms) | vs OpenAI | -| ------------------ | ------------- | ----------------- | ------------- | ---------- | -| success rate (%) | 99.8 | 99.8 | — | — | -| dial p50 (ms) | 276 | 158 | −118 | **faster** | -| session p50 (ms) | 7 | 0 | −7 | **faster** | -| 1st-audio p50 (ms) | 440 | 664 | +224 | slower¹ | -| total p50 (ms) | 816 | 1010 | +194 | slower¹ | -| total p95 (ms) | 2152 | 1970 | −182 | **faster** | -| total p99 (ms) | 2692 | 2610 | −82 | **faster** | - -The gateway is **faster than direct on 4 of 6 metrics**. The warm pool makes the -**session phase sub-millisecond** at the median — ~76% of connects hit the pool, -~70% had session < 1 ms. ¹ The two "slower" rows are not gateway overhead: -`1st-audio` is OpenAI's own inference time (the gateway only relays it), which ran -slower during the gateway legs and drags `total p50` with it. - -**Pool OFF** (control, `REALTIME_POOL_SIZE=0`): session p50 was **367 ms** — the -fresh-dial overhead the pool removes. - -## Reproduce - -The load generator lives in a separate repo: -**https://github.com/ishaan-berri/litellm-realtime-bench** - -```bash -git clone https://github.com/ishaan-berri/litellm-realtime-bench -cd litellm-realtime-bench && go build -o wsbench . - -# Direct to OpenAI (baseline) -./wsbench -host api.openai.com -key "$OPENAI_API_KEY" -m gpt-realtime -n 5000 -c 500 -t 60 - -# Through the gateway — run once with pool ON, once with REALTIME_POOL_SIZE=0 -./wsbench -host -key "$LITELLM_MASTER_KEY" -m gpt-realtime -n 5000 -c 500 -t 60 -``` - -Run the gateway with the env stand-in (`OPENAI_REALTIME_MODEL=gpt-realtime`, -`OPENAI_API_KEY`, `LITELLM_MASTER_KEY`, `REALTIME_POOL_SIZE`, `HOST=0.0.0.0`). At -500 concurrency over N instances, size the pool to `≈ 500 / N` per instance (64 was -used here for 10 instances). The bench repo's README covers running 500-concurrency -legs from a hosted multi-vCPU runner. **Never commit keys — pass them via `-key`.** diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/mod.rs b/litellm-rust/crates/ai-gateway/src/audio_transcription/mod.rs index 06d53eacf59..dbe55cc15b0 100644 --- a/litellm-rust/crates/ai-gateway/src/audio_transcription/mod.rs +++ b/litellm-rust/crates/ai-gateway/src/audio_transcription/mod.rs @@ -40,6 +40,3 @@ pub async fn audio_transcription(request: AudioTranscriptionRequest<'_>) -> Resu .await .into_result() } - -#[cfg(test)] -mod tests; diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/tests.rs b/litellm-rust/crates/ai-gateway/tests/audio_transcription.rs similarity index 95% rename from litellm-rust/crates/ai-gateway/src/audio_transcription/tests.rs rename to litellm-rust/crates/ai-gateway/tests/audio_transcription.rs index 5df04708b7d..1c60e3ef848 100644 --- a/litellm-rust/crates/ai-gateway/src/audio_transcription/tests.rs +++ b/litellm-rust/crates/ai-gateway/tests/audio_transcription.rs @@ -2,10 +2,9 @@ use std::io::{Read, Write}; use std::net::TcpListener; use std::thread; +use litellm_ai_gateway::audio_transcription::{AudioTranscriptionRequest, audio_transcription}; use serde_json::{Map, json}; -use super::{AudioTranscriptionRequest, audio_transcription}; - #[tokio::test] async fn bedrock_request_is_signed_and_contains_audio() { let listener = TcpListener::bind("127.0.0.1:0").expect("listener"); diff --git a/litellm-rust/crates/core/src/lifecycle/request_body.rs b/litellm-rust/crates/core/src/lifecycle/request_body.rs index aaa915d3ce2..a8552c4922d 100644 --- a/litellm-rust/crates/core/src/lifecycle/request_body.rs +++ b/litellm-rust/crates/core/src/lifecycle/request_body.rs @@ -1,18 +1,12 @@ -/// Representation exposed to `pre_call`. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum CallbackBodyView { - /// A mutable structured object. Structured, - /// An encoded string. Serialized, } -/// When transport captures the request body. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum BodyReadPoint { - /// Before `pre_call`. BuildRequest, - /// After `pre_call`. Send, }