mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-30 06:43:27 +00:00
cd58014315
* feat(soniox): add realtime speech-to-text on /v1/realtime Maps OpenAI realtime transcription sessions onto the Soniox websocket API (stt-rt-v5): server VAD uses Soniox endpoint detection, commit sends a manual finalize, deployment litellm_params pass Soniox options through, translation sessions transcribe the translated tokens, idle streams get keepalives, and cost is tracked from processed audio duration. * fix(soniox): bill realtime usage from integer milliseconds * fix(soniox): bound session-start queue and cover realtime backend edge paths * fix(soniox): bill realtime stream time, start sessions on connect and surface client protocol errors Soniox charges for the whole open stream, so session close now bills the connection's open time. The backend announces the session on connect, which flushes audio sent before any session.update, and a single upstream pump makes recv safe to cancel. Invalid client frames now get an OpenAI error event instead of a silent disconnect * fix(soniox): skip deferred setup so client frames reach Soniox in order Drops the connect marker frame, which counted as an upstream frame and hid refusals, and could start a buffered stream at the default rate before the client's session.update. Soniox now opts out of deferred setup, the handler announces a synthetic session.created whenever no setup went upstream, a session.update that would change a started stream gets an error event, and stream time is billed only once a stream started * fix(soniox): judge a follow-up session.update only on the settings it sends A partial follow-up session.update was rebuilt with defaults for the settings it left out, so restating part of a 16 kHz or VAD-off stream was rejected as a change * fix(soniox): treat an omitted rate in a follow-up session.update as unchanged * chore(soniox): drop comments that restate the code * fix(soniox): return speaker labels, log refused sessions as failures, and probe realtime health Deltas carry speaker and language when diarization or language identification is on, and the completed event lists segments per speaker and language. A session Soniox refuses before any audio is processed is logged as a failure with no spend. Billing starts at the start request. Deferred-setup providers (Meta, Vertex Chirp) send the error event and still end the session. Soniox realtime models get a websocket /health probe and cost rows for the stt-rt-v5 aliases. * fix(soniox): bill failed streams, probe health with deployment options, and keep token accumulation linear Only request rejections (400, 401, 402, 403, 429) before any processed audio count as a refused session, so a 408 or 5xx still bills the open stream. The /health probe starts Soniox with the deployment's options, sends 100 ms of silence, and passes only on Soniox's finished frame. A turn's final tokens append in constant time instead of copying the tuple per token. * fix(soniox): log every Soniox error before any processed audio as a failed session * fix(soniox): keep untranslated speech, support input_audio_buffer.clear, ignore an empty language * fix(soniox): reject frames without a type, keep strict language hints, queue finalizes in a deque * fix(soniox): answer a commit with no speech with an empty completed turn * fix(soniox): let a server VAD turn answer a commit that is still in flight * fix(soniox): let a server VAD turn answer only the oldest pending commit --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_main.py | ||