mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-29 01:42:19 +00:00
39372b3922
Fixes #40563. Both GeminiRealtimeConfig and VertexAIRealtimeConfig hardcoded the pcm16 input MIME type to audio/pcm;rate=24000. 24kHz is the Live API's *output* rate. Its documented native *input* rate is 16kHz, and the MIME rate is the only channel the caller has for telling the server what it is actually sending: "Audio output always uses a sample rate of 24kHz. Input audio is natively 16kHz ... To convey the sample rate of input audio, set the MIME type of each audio-containing Blob to a value like audio/pcm;rate=16000." https://ai.google.dev/gemini-api/docs/live-api/capabilities Because the server resamples against whatever the MIME type claims, a client streaming correct 16kHz PCM16 had it relabeled as 24kHz, which corrupts it server-side and degrades transcription with no error anywhere. Three changes: 1. The rate now comes from what the client declared. session.update carries it in the GA shape at audio.input.format.rate, so that value is recorded and used for every subsequent blob. The rate-less beta shape (input_audio_format is a bare codec name), a missing or malformed rate, and a bool (an int subclass, so excluded explicitly) all leave the default alone. 2. That default is now 16000, the documented native input rate, instead of the output rate. 3. VertexAIRealtimeConfig's byte-identical copy of get_audio_mime_type is deleted so it inherits the parent. The duplicate is why patching the parent alone had no effect on the Vertex path, which is the trap the report calls out; a test now asserts the override stays gone. The billed audio duration reads the same rate, so the label and the duration estimate cannot disagree. PCM16_INPUT_AUDIO_BYTES_PER_SECOND (48000, that is 24kHz x 2 bytes) is replaced by the declared rate x PCM16_BYTES_PER_SAMPLE. This does move the estimate for a transcribe-live caller who declares no rate: the same byte count is now billed as 1.5x the duration, because 16kHz audio takes 1.5x as long to send as the 24kHz the old constant assumed. The two existing estimate tests are updated for that, and a new test covers a caller that declares 24kHz and still bills at the old numbers. 14 tests added or updated, each verified to fail against unpatched sources. tests/test_litellm/llms/{gemini,vertex_ai}/realtime: 97 passed. |
||
|---|---|---|
| .. | ||
| a2a_protocol | ||
| anthropic_interface | ||
| batches | ||
| chat_completions | ||
| completion_extras | ||
| compression | ||
| endpoints | ||
| enterprise | ||
| integrations | ||
| litellm_core_utils | ||
| llms | ||
| messages | ||
| models | ||
| ocr | ||
| passthrough | ||
| rag | ||
| realtime_api | ||
| repositories | ||
| router_strategy | ||
| router_utils | ||
| rust_bridge | ||
| sandbox | ||
| skills | ||
| test_router | ||
| types | ||
| videos | ||
| __init__.py | ||
| AGENTS.md | ||
| conftest.py | ||
| test_package_layout.py | ||
| test_socket_policy.py | ||