* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply
Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its
default audio setup at connect, so a client's session.update (GA output_modalities or
beta modalities) was dropped and every answer came back as audio. The setup now goes
out on the client's first message: a session.update becomes that setup, anything else
gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a
provider hook. Text setups label deltas response.output_text.delta, and downgrading
TEXT to AUDIO on an audio-only model logs a warning
* fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup
* fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup
Gemini and Vertex Live accept one setup per connection, so the bridge used to
turn only the first session.update into that setup and drop the rest without
telling the client. Now every session.update that arrives before the first
content frame is held, deep-merged (later values win per field), acknowledged
with a session.updated built from the setup it would produce, and the merged
session becomes the one setup sent with the first conversation item, audio
frame, or tool result. The session events also surface the systemInstruction
text as instructions, which the old string check never matched because the
setup carries it as a Content object.
* fix(realtime): cap the session.update merge depth so the recursive detector accepts it
_merge_session_fields recurses once per nested mapping level, so CI's
recursive_detector flagged it. Bound the descent at eight levels (the OpenAI
session object nests four deep at most), replace anything deeper wholesale
with the later session.update's value, and list the helper with the other
depth-capped functions the detector allows.
* fix(gemini-live): accept a string output modality and drop values Live cannot produce
A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once.
* fix(gemini realtime): answer an error for session updates Gemini Live cannot apply
An output modality the model cannot produce, or a session.update sent after the
first content, now gets an invalid_request_error event instead of a silent swap
or drop, unless drop_params is set. The synthetic session.created reports the
model's default modality, setupComplete no longer adds an extra session.updated,
and merging nested session updates is linear in their size
* fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns
A pre_call guardrail blocking the first typed message now goes out after the
setup, so Gemini and Vertex no longer close the session with 1007.
Push-to-talk sessions (turn_detection null or create_response false) wrap each
audio turn in empty activityStart and activityEnd messages, which Gemini needs
when automatic activity detection is off.
A session with no content after 60 seconds sends its setup, since Vertex
refuses a setup 300 seconds after connect. Held session.update frames past
10 MB answer session_update_too_large, and deferred mode now errors on a
second session.update unless drop_params is set.
* fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection
An update that leaves the applied setup unchanged now gets session.updated
instead of session_update_not_applied, so clients that resend their session
with the first response keep working. Deferred setup mode merges held updates
like the default mode, turn_detection null turns Gemini's activity detection
off, drop_params drops a voice outside the documented Live voices, and the
lazy setup deadline moves to 240 seconds
* fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content
* fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices
A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name.
* fix(realtime): hold first content so a following session.update joins the Gemini setup
Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too
* fix(realtime): check AI Studio Live voices like Vertex
* fix(realtime): forward any voice to AI Studio Live models that accept it
* fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config
The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup.
* fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock
The deadline and grace tests waited on real 50 ms timers, and the merge cost
check compared wall-clock runs. The session now reads an injected clock,
time.monotonic by default, the two tests step it, and the merge cost check
moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes
the backend through wrap_backend, which main added after the test was written
4c8b90945f