Peek at the pending session.update instead of popping it, so an eager fallback failure before the bridge starts does not lose the replay for the next attempt. Move the websocket scope keys to constants.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Bedrock realtime caught every exception inside both forwarding tasks and
gathered them with return_exceptions=True, so a provider failure surfacing
after the websocket handshake (lazy duplex stream: 503/429/validation only
show up on await_output or the input publisher) made async_realtime return
normally and the router recorded a success instead of running fallbacks and
cooldown accounting. session.updated is now acked only after Bedrock is
ready, provider failures escape as BedrockError with the AWS status code,
a failure after the client disconnected is not reported as a provider
failure, and a fallback attempt on the same websocket replays the pending
session.update instead of emitting a second session.created.
Resolves LIT-6484
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Muse partials carry no turnId and belong to the most recent speechStart,
and the docs say the model may keep post processing a turn after speechEnd
until speechComplete. Releasing the active turn on speechEnd made any
partial arriving in that window raise and get dropped in ENDPOINTING mode.
The turn now stays active until its speechComplete or final transcript.
Gate the reasoning_effort translation on the cost-map flag or the model name containing
claude, so unmapped Claude serving endpoints keep translating. Flag the newer Claude
entries that were missing it. Expose supports_anthropic_thinking_payload as a public
helper next to the other supports_* wrappers instead of importing the private factory.
Drop the adaptive-only guard, since the adaptive flags only ever match Claude ids, and
add regression tests for an unmapped Claude endpoint and an adaptive Claude model
A duplicate speechStart for a turn that already stopped used to make that
closed turn active again, so the next turnless PUSH_TO_TALK transcript was
routed to the finished item and dropped.
Turns no longer wait behind each other in a FIFO queue, so an empty
server_vad turn (speechStart then speechEnd with no transcript) cannot
stall every later turn, and a PUSH_TO_TALK speechComplete now closes its
turn without waiting for a speechEnd that never arrives. Each turn keeps
its own idempotent emit state, so late or duplicate speechEnd,
speechComplete and transcript frames are no-ops, and finished turns are
remembered in a bounded map instead of a separate tombstone deque.
The session.created ack and the sanitized error frame are now typed as
members of OpenAIRealtimeEvents, which removes the typing.cast calls
that the strict ruff budget flagged.
Replace the hand-rolled Meta realtime handler with a MetaRealtimeConfig
that plugs into the shared realtime handler and RealTimeStreaming relay.
Clients keep speaking the OpenAI realtime wire: session.update,
input_audio_buffer.append/commit and the OpenAI transcription events.
Unsupported transcription settings are logged and dropped, matching the
Gemini realtime precedent, and the Meta-specific session.mode, keywords,
language_bias, DIARIZATION and speaker extensions are removed.
Drop the MODEL_API_KEY env var in favor of the standard META_API_KEY,
remove the private-logging flag so spend logs record the transcript the
same way other realtime models do, and add per-second pricing for
muse-voice-transcribe-1.0.
The relay now sends raw bytes from transform_realtime_request straight to
the backend after pace_backend_send, and transcription sessions never
trigger response.create.
Mantle's /openai/v1/responses rejects reasoning.summary values other than "auto" with 400 unsupported_parameter. Drop it with a warning under drop_params, otherwise raise UnsupportedParamsError naming the remedy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Both reasoning_effort mappers ended their if/elif chain in a bare ValueError.
exception_type() has no branch for ValueError, so it fell through to the shared
APIConnectionError fallback and the proxy answered a malformed client request
with a retryable HTTP 500 carrying no hint of the accepted values.
Raise UnsupportedParamsError (400) instead, listing the supported set, matching
what the Anthropic and Bedrock transforms already do and what this same file
already does at its five other param-validation sites.
This also covers 'xhigh' and 'max', which are members of litellm's own
REASONING_EFFORT literal but have no Gemini mapping, so callers bridging from
OpenAI-shaped code were hitting the 500 without typing anything wrong.
Fixes#40474
Claude-Session: https://claude.ai/code/session_01XT1qsbjLwnhiN5sQ2hNUxr
* fix(passthrough): parse Bedrock stream spend incrementally instead of buffering the whole response
Bedrock pass-through streaming kept every relayed chunk in memory until EOF and
then decoded, parsed and translated the whole stream again for spend logging.
Large or concurrent streams could exhaust proxy worker memory.
Sync and async passthrough wrappers now hand each chunk to a provider stream
collector as it is relayed. Bedrock decodes event-stream frames incrementally,
folds consecutive text deltas, and keeps only what stream_chunk_builder needs
for usage, tool calls and metadata. Text deltas are no longer retained in the
Bedrock and Anthropic stream decoders either. Providers without a collector
keep the previous raw-bytes behavior. Collector failures are isolated so spend
tracking can never interrupt the customer stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): assert the spend payload the collector builds instead of mock internals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): type the Bedrock collector helpers by the collector protocol instead of asserting the class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
W&B's serverless catalog grows faster than the registry names it, so a model
they ship today resolves as non-reasoning here until someone edits the cost map,
and the caller's reasoning_effort is dropped or rejected.
Add a wandb-reasoning-baseline capability rule to fallback_generalizations so any
wandb/ id the map has not described defaults to supports_reasoning. Rules lose to
exact entries, so mapped non-reasoning models such as
wandb/meta-llama/Llama-3.1-8B-Instruct are unaffected.
The rule carries no mode and no pricing, so cost stays on the standard unpriced
behavior and the deployment does not read as catalog-mapped to the router's
reasoning-effort resolver.
Claude-Session: https://claude.ai/code/session_01A6SkwJdfZUmkzfUkrEkqX8
