diff --git a/docs/my-website/docs/providers/chatgpt.md b/docs/my-website/docs/providers/chatgpt.md new file mode 100644 index 00000000000..bc971be087e --- /dev/null +++ b/docs/my-website/docs/providers/chatgpt.md @@ -0,0 +1,121 @@ +# ChatGPT, Codex, and GPT-Live + +The proxy exposes the public GPT-Live session routes and retains the Codex-compatible `POST /live` route. Clients authenticate to LiteLLM with a LiteLLM virtual key. A `chatgpt` deployment uses one proxy-wide ChatGPT OAuth record, while an `openai` deployment uses its configured OpenAI API key. Do not send an upstream OAuth token as the proxy key + +## Configure LiteLLM deployments + +Codex text, image, and voice requests need separate LiteLLM aliases because their backends have different capabilities. The canonical aliases below keep the primary Qwen model separate from the ChatGPT OAuth models + +```yaml +model_list: + - model_name: qwen3.8-flash-next-codex + litellm_params: + model: openai/qwen3.8-flash-next + api_base: https://qwen.example.com/v1 + api_key: os.environ/QWEN_API_KEY + + - model_name: gpt-image-2 + litellm_params: + model: chatgpt/gpt-image-2 + + - model_name: gpt-image-2.5-flare + litellm_params: + model: chatgpt/gpt-image-2.5-flare + + - model_name: gpt-image-2.5-sunburst + litellm_params: + model: chatgpt/gpt-image-2.5-sunburst + + - model_name: gpt-realtime-1.5 + litellm_params: + model: chatgpt/gpt-realtime-1.5 + + - model_name: gpt-live-1-codex + litellm_params: + model: chatgpt/gpt-live-1-codex +``` + +If clients use shorter local aliases, publish separate aliases such as `qwen-codex`, `images`, and `voice` that point to the corresponding deployments. Authorize the exact alias sent by the client in the key or team policy, and for a restricted team member set `allowed_models` to contain the aliases used for text, image, or voice requests. Selecting the Qwen alias does not give it image-generation or voice capabilities. The Qwen endpoint in this example uses the OpenAI-compatible adapter and must expose `/v1/responses`; a native vLLM deployment can use `hosted_vllm/qwen3.8-flash-next` when that endpoint is available. LiteLLM does not promise provider-specific tool, reasoning, or stream behavior parity. For a deployment using the public OpenAI API instead, configure `model: openai/gpt-live-1` and `api_key: os.environ/OPENAI_API_KEY` under its own alias. The public API documentation uses `gpt-live-1`; the Codex alias and its backend capabilities are separate + +### Configure global ChatGPT OAuth + +The ChatGPT provider reads one auth file for the LiteLLM process. Set these environment variables before starting the proxy when the default location is not suitable + +```bash +export CHATGPT_TOKEN_DIR=/var/lib/litellm/chatgpt +export CHATGPT_AUTH_FILE=auth.json +``` + +The defaults are `~/.config/litellm/chatgpt` and `auth.json`. Persist the directory and complete the provider's device-code OAuth flow. The provider refreshes the stored record when it expires. All `chatgpt/...` deployments in that process share this record. Live rejects per-deployment `chatgpt_auth_profile`, `chatgpt_token_dir`, and `chatgpt_auth_file` overrides + +### Image generation and editing + +Use the `gpt-image-2` alias for both `/v1/images/generations` and `/v1/images/edits`. Keep the exact image aliases requested by your Codex version; a shorter `images` alias only works for clients configured to request it. Codex sends image requests to its active provider's `base_url`, with no separate image URL override. LiteLLM then selects the image deployment independently of the primary text model. ChatGPT image generation and editing require a valid global ChatGPT OAuth login, but a successful login does not establish that the account or backend supports every image operation. Image editing accepts JSON reference images and multipart files, but not masks. Image 2.5 aliases preserve the requested model name; an accepted name does not prove which backend model executed + +## Configure Codex through LiteLLM + +Codex sends its Responses requests to LiteLLM's `/v1/responses` endpoint. Point the Codex provider at the proxy and select the primary Qwen alias (or the shorter alias you published) + +```toml +model = "qwen3.8-flash-next-codex" +model_provider = "litellm" +experimental_realtime_ws_base_url = "https://litellm.example.com/v1" +experimental_realtime_webrtc_call_base_url = "https://litellm.example.com/v1" + +[model_providers.litellm] +name = "LiteLLM" +base_url = "https://litellm.example.com/v1" +wire_api = "responses" +requires_openai_auth = true +experimental_bearer_token = "" +``` + +Keep Codex signed in with ChatGPT for its client-side capability checks. `experimental_bearer_token` is the LiteLLM virtual key issued by the proxy. It must never contain the upstream ChatGPT OAuth access or refresh token. `requires_openai_auth = true` enables the Codex OpenAI-auth capability path while LiteLLM remains responsible for the upstream provider credentials + +## Configure voice routing + +The two realtime settings in the TOML example are root-level Codex settings, not fields inside `[model_providers.litellm]`. `experimental_realtime_ws_base_url` routes the Realtime WebSocket and its sideband through LiteLLM. `experimental_realtime_webrtc_call_base_url` is optional and separately routes HTTP WebRTC call creation. The optional root setting `experimental_realtime_ws_model` overrides the voice model; leave it unset to retain your client's default. An override must match the active protocol: `gpt-realtime-1.5` for legacy Realtime v1/v2 or `gpt-live-1-codex` for frameless Live v3. The base URLs end at `/v1`; LiteLLM adds the protocol-specific path + +| Voice operation | LiteLLM path | +| --- | --- | +| Legacy Realtime WebSocket | `/v1/realtime` | +| HTTP WebRTC call creation | `/v1/realtime/calls` | +| Frameless Live v3 signaling | `/v1/live` and `/v1/live/{call_id}` | +| Public Live session APIs | `/v1/live/sessions...` | + +WebSocket authentication uses `Authorization: Bearer ` by default. If the proxy sets `litellm_key_header_name`, send the virtual key in that configured header instead. Voice is experimental and pending retest: an observed mobile `POST /live` returned 201, but its sideband used the default `api.openai.com` and returned 404. The WebSocket base URL above addresses that routing gap; full bidirectional voice is not verified + +## Public Live routes + +Use the proxy host in place of `api.openai.com`. Send the configured LiteLLM alias in `session.model` when creating a session, or in the first `session.start` event for a primary WebSocket. Keep the returned session ID unchanged for subsequent operations + +| Method | Path | Request and response | +| --- | --- | --- | +| POST | `/v1/live/sessions` | JSON `session` and `transport: {type: "webrtc", sdp: ""}`; returns 201 JSON with `session.id` and `transport.sdp` | +| POST | `/v1/live/sessions/{session_id}/fork` | JSON WebRTC `transport` and optional `session` overrides; returns 200 JSON with the new session ID and SDP answer | +| GET | `/v1/live/sessions/{session_id}/content` | Downloads stored recording content without converting it to JSON | +| POST | `/v1/live/sessions/{session_id}/accept` | JSON `session` with `type: "live"` and model; successful SIP acceptance returns an empty body | +| POST | `/v1/live/sessions/{session_id}/reject` | JSON with required integer `status_code` from 300 through 699 | +| POST | `/v1/live/sessions/{session_id}/refer` | JSON with `target_uri` for the SIP destination | +| POST | `/v1/live/sessions/{session_id}/hangup` | No request body | +| WebSocket | `/v1/live/sessions` | Start with `session.start`, then wait for `session.started` before sending audio or commands | +| WebSocket | `/v1/live/sessions/{session_id}/attach` | Attach to an existing session; do not send `session.start` or input audio | +| WebSocket | `/v1/live/sessions/{session_id}/fork` | Start with `session.start` and a required `session` overrides object, which may be empty | + +Public WebRTC creation uses JSON, not the multipart or raw SDP formats used by the Codex compatibility route. `POST /live` and its existing aliases remain available for Codex clients using that format. WebRTC audio travels on media tracks; its data channel carries Live JSON events. Primary WebSocket audio uses base64 chunks in `session.input_audio.append` and `session.output_audio.delta` + +The proxy preserves Live event payloads, including nested Responses events inside `response.event`, rather than translating them into Realtime events. Session routing rewrites the configured model alias to the selected upstream model. Audio, transcript, delegation and usage events retain their upstream format. Send `session.close` and wait for `session.closed` to obtain final usage; a disconnected socket alone does not confirm successful finalization + +## Availability and verification + +Route support does not establish that every configured backend or account supports every operation. The official API describes project API-key authentication; it does not guarantee equivalent capabilities for ChatGPT OAuth. An OAuth request reaching SDP validation proves only that the request reached that validation step. It does not prove a working audio session, recording, fork or SIP call. The routes listed here have not all been tested against a real upstream service + +Session controls require a session known to the proxy and owned by the authenticated caller. Incoming SIP calls originate upstream. A proxy administrator can accept or reject the raw ID from a verified incoming-call webhook by supplying `x-litellm-live-model` with an alias that resolves to exactly one deployment. Successful acceptance returns the proxy-owned handle in `x-litellm-live-session-id`, preserving the API's empty response body. Use that handle for subsequent controls. Ordinary virtual keys cannot enroll arbitrary upstream session IDs; a trusted webhook-to-owner enrollment flow is still required for those keys + +Live duration uses cumulative `usage.seconds`; legacy Codex milliseconds remain supported. WebRTC initialization has a 15-second minimum credited against running duration, not added to it. Nested terminal Responses usage is charged separately using its backend model and deduplicated by response ID. A failed observation connection cannot establish complete usage. Managed delegation also depends on receiving its backend usage events; the upstream sideband does not replay events emitted before attachment + +Managed Responses delegation authorizes its backend model as well as the voice model. Use client delegation when a key's per-model budgets or token/request limits require admission checks for each backend invocation. Restricted-model WebRTC keys must explicitly exclude `session.update` from frontend client events when using managed delegation, because that data channel connects directly to OpenAI and could otherwise change the backend model outside the proxy's checks + +For both HTTP and WebSocket forks, restricted-model keys must explicitly set `session.delegation` to `{ "type": "client" }` or `{ "type": "responses", "responses": { "model": "authorized-backend" } }`. Empty overrides cannot safely authorize an inherited backend: the session handle records startup configuration, while later updates may have changed the upstream model. Upstream rules still determine which delegation overrides a source session permits + +See the official [Live overview](https://developers.openai.com/api/docs/guides/live), [Live API reference](https://developers.openai.com/api/reference/resources/live), [session management](https://developers.openai.com/api/docs/guides/live-conversations), [WebRTC guide](https://developers.openai.com/api/docs/guides/voice-webrtc?api=live), [WebSocket guide](https://developers.openai.com/api/docs/guides/voice-websockets?api=live), [server controls](https://developers.openai.com/api/docs/guides/voice-server-controls?api=live) and [SIP guide](https://developers.openai.com/api/docs/guides/voice-sip?api=live) for the upstream contract. The voice guides also contain Realtime tabs with different routes and formats diff --git a/litellm/cost_calculator.py b/litellm/cost_calculator.py index 87807d83efd..000b1b9de8a 100644 --- a/litellm/cost_calculator.py +++ b/litellm/cost_calculator.py @@ -8,7 +8,7 @@ from types import MappingProxyType from typing import TYPE_CHECKING, Any, Final, Literal, cast from httpx import Response -from pydantic import BaseModel, Field, ValidationError +from pydantic import BaseModel, ValidationError from typing_extensions import ReadOnly, TypedDict import litellm @@ -2471,6 +2471,13 @@ class RealtimeAPITokenUsageProcessor(BaseTokenUsageProcessor): result["response"].get("usage", {}) ) usage_objects.append(usage_object) + usage_objects.extend( + ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage( # pyright: ignore[reportPrivateUsage] # reuse the existing Responses usage conversion for nested Live events + response.usage.model_dump() + ) + for response in _live_backend_responses(results) + if response.usage is not None + ) return usage_objects @staticmethod @@ -2621,7 +2628,13 @@ def handle_realtime_stream_cost_calculation( potential_model_names.append(litellm_model_name) input_cost_per_token, output_cost_per_token = _first_priced_realtime_token_costs( potential_model_names=potential_model_names, - combined_usage_object=combined_usage_object, + combined_usage_object=( + RealtimeAPITokenUsageProcessor.collect_and_combine_usage_from_realtime_stream_results( + [event for event in results if event.get("type") != "response.event"] + ) + if any(event.get("type") == "response.event" for event in results) + else combined_usage_object + ), custom_llm_provider=custom_llm_provider, data_residency=data_residency, ) @@ -2639,7 +2652,13 @@ def handle_realtime_stream_cost_calculation( custom_llm_provider=custom_llm_provider, litellm_model_name=litellm_model_name, ) - total_cost: Final = input_cost_per_token + output_cost_per_token + transcription_cost + live_audio_cost + backend_cost: Final = sum( + _live_backend_response_cost(response, litellm_logging_obj) + for response in _live_backend_responses(results, litellm_logging_obj) + ) + total_cost: Final = ( + input_cost_per_token + output_cost_per_token + transcription_cost + live_audio_cost + backend_cost + ) _store_cost_breakdown_in_logging_obj( litellm_logging_obj=litellm_logging_obj, @@ -2649,7 +2668,11 @@ def handle_realtime_stream_cost_calculation( total_cost_usd_dollar=total_cost, additional_costs={ # mutable-ok: logging cost breakdown requires a concrete dict name: cost - for name, cost in (("transcription_cost", transcription_cost), ("live_audio_cost", live_audio_cost)) + for name, cost in ( + ("transcription_cost", transcription_cost), + ("live_audio_cost", live_audio_cost), + ("live_backend_cost", backend_cost), + ) if cost > 0 } or None, @@ -2659,12 +2682,61 @@ def handle_realtime_stream_cost_calculation( return total_cost -class _LiveSessionDurationUsage(BaseModel): - audio_duration_ms: float = Field(strict=True, ge=0, allow_inf_nan=False) +class _LiveBackendEvent(BaseModel): + type: str + response: object = None -class _LiveSessionClosedEvent(BaseModel): - usage: _LiveSessionDurationUsage +class _LiveBackendEnvelope(BaseModel): + event: _LiveBackendEvent + + +def _live_backend_responses( + results: OpenAIRealtimeStreamList, logging_obj: LitellmLoggingObject | None = None +) -> tuple[ResponsesAPIResponse, ...]: + responses: Final = { + response.id: response + for result in results + if result.get("type") == "response.event" + and (response := _live_backend_response(result, logging_obj)) is not None + } + return tuple(responses.values()) + + +def _mark_live_backend_accounting_incomplete(logging_obj: LitellmLoggingObject | None) -> None: + verbose_logger.warning("Live backend accounting incomplete: missing valid terminal usage or model pricing") + if logging_obj is not None: + logging_obj.model_call_details["realtime_backend_accounting_incomplete"] = True + + +def _live_backend_response( + result: Mapping[str, object], logging_obj: LitellmLoggingObject | None +) -> ResponsesAPIResponse | None: + try: + event: Final = _LiveBackendEnvelope.model_validate(result).event + except ValidationError: + return None + if event.type not in ("response.completed", "response.incomplete", "response.failed"): + return None + try: + response: Final = ResponsesAPIResponse.model_validate(event.response) + except ValidationError: + _mark_live_backend_accounting_incomplete(logging_obj) + return None + if response.usage is None: + _mark_live_backend_accounting_incomplete(logging_obj) + return None + return response + + +def _live_backend_response_cost(response: ResponsesAPIResponse, logging_obj: LitellmLoggingObject | None) -> float: + try: + return completion_cost( + completion_response=response, model=response.model, custom_llm_provider="openai", call_type="aresponses" + ) + except Exception: # noqa: BLE001 # preserve measured voice cost when backend pricing cannot be resolved + _mark_live_backend_accounting_incomplete(logging_obj) + return 0.0 def handle_live_session_duration_cost( @@ -2672,18 +2744,40 @@ def handle_live_session_duration_cost( custom_llm_provider: str, litellm_model_name: str, ) -> float: - terminal: Final = next((event for event in reversed(results) if event.get("type") == "session.closed"), None) - if terminal is None: - return 0.0 - try: - usage: Final = _LiveSessionClosedEvent.model_validate(terminal).usage - except ValidationError: - return 0.0 + seconds: Final = max( + ( + duration + for event in results + if event.get("type") in ("session.closed", "session.usage.updated") + and (duration := _live_duration_seconds(event)) is not None + ), + default=0.0, + ) + initialization_seconds: Final = max( + ( + duration + for event in results + if event.get("type") == "litellm.live.initialization" + and (duration := _live_duration_seconds(event)) is not None + ), + default=0.0, + ) try: model_info: Final = litellm.get_model_info(model=litellm_model_name, custom_llm_provider=custom_llm_provider) except Exception: return 0.0 - return usage.audio_duration_ms / 1000 * (model_info.get("input_cost_per_second") or 0.0) + return max(seconds, initialization_seconds) * (model_info.get("input_cost_per_second") or 0.0) + + +def _live_duration_seconds(event: Mapping[str, object]) -> float | None: + from litellm.types.realtime import LiveSessionUsageEvent + + try: + usage: Final = LiveSessionUsageEvent.model_validate(event).usage + except ValidationError: + return None + raw_usage: Final = cast(Mapping[str, object], event.get("usage")) + return usage.duration / (1 if "seconds" in raw_usage else 1000) def handle_realtime_transcription_cost_calculation( diff --git a/litellm/litellm_core_utils/realtime_streaming.py b/litellm/litellm_core_utils/realtime_streaming.py index 70ba3341ac9..dfbc89ce193 100644 --- a/litellm/litellm_core_utils/realtime_streaming.py +++ b/litellm/litellm_core_utils/realtime_streaming.py @@ -15,6 +15,7 @@ from litellm._logging import redact_internal_details_from_client_message, verbos from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig from litellm.types.llms.openai import ( + OpenAILiveResponseEvent, OpenAIRealtimeEvents, OpenAIRealtimeOutputItemDone, OpenAIRealtimeResponseDelta, @@ -148,6 +149,7 @@ class RealTimeStreaming: logging_worker: _LoggingWorker = GLOBAL_LOGGING_WORKER, *, account_usage: bool = True, + live_initialization_seconds: float = 0, ): self.websocket: _ClientWebSocket = websocket self.backend_ws = backend_ws @@ -155,6 +157,10 @@ class RealTimeStreaming: self._logging_worker = logging_worker self._account_usage = account_usage self.messages: list[OpenAIRealtimeEvents] = [] + if account_usage and live_initialization_seconds > 0: + self.messages.append( + {"type": "litellm.live.initialization", "usage": {"seconds": live_initialization_seconds}} + ) self._backend_sent_frames: bool = False self.input_message: dict = {} self.input_messages: list[dict[str, str]] = [] @@ -266,9 +272,16 @@ class RealTimeStreaming: else: message_obj = cast(dict[str, Any], json.loads(cast(str, message))) self._collect_tool_calls_from_response_done(cast(dict, message_obj)) - if message_obj.get("type") == "session.closed" and isinstance(message_obj.get("usage"), dict): + if message_obj.get("type") in ("session.closed", "session.usage.updated") and isinstance( + message_obj.get("usage"), dict + ): self.messages.append(TypeAdapter(OpenAIRealtimeSessionClosed).validate_python(message_obj)) return + if message_obj.get("type") == "response.event" and isinstance(message_obj.get("event"), dict): + nested: Final = message_obj["event"] + if nested.get("type") in ("response.completed", "response.incomplete", "response.failed"): + self.messages.append(TypeAdapter(OpenAILiveResponseEvent).validate_python(message_obj)) + return if not self._should_store_message(message_obj): return try: diff --git a/litellm/llms/chatgpt/live.py b/litellm/llms/chatgpt/live.py new file mode 100644 index 00000000000..696ac00f118 --- /dev/null +++ b/litellm/llms/chatgpt/live.py @@ -0,0 +1,193 @@ +import re +from collections.abc import Mapping +from dataclasses import dataclass, field +from types import MappingProxyType +from typing import TYPE_CHECKING, Final, Literal, TypeAlias +from unicodedata import category +from urllib.parse import quote, unquote + +import httpx +from pydantic import JsonValue + +from litellm.constants import REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES +from litellm.llms.chatgpt.realtime import ( + ChatGPTRealtime, + configured_realtime_headers, + configured_realtime_query, + realtime_headers, +) +from litellm.llms.custom_httpx.http_handler import ( + get_async_httpx_client, # pyright: ignore[reportUnknownVariableType] # shared client has legacy untyped optional params + get_shared_realtime_ssl_context, +) +from litellm.types.router import GenericLiteLLMParams +from litellm.types.utils import LlmProviders + +if TYPE_CHECKING: + from websockets.asyncio.client import ClientConnection + +LiveQuery: TypeAlias = Mapping[str, str | int | float | bool | None | tuple[str | int | float | bool | None, ...]] +LiveBody: TypeAlias = Mapping[str, JsonValue] +LiveOperation: TypeAlias = Literal["fork", "accept", "reject", "refer", "hangup", "content", "attach"] +_PATH: Final = re.compile(r"live/sessions(?:/([^/]+)/(fork|accept|reject|refer|hangup|content|attach))?\Z") +_ROUTING_QUERY: Final = frozenset(("model", "session_id", "call_id", "api_key", "api_base", "authorization")) + + +@dataclass(frozen=True, slots=True) +class LiveDeployment: + model: str + model_id: str | None = None + provider: Literal["chatgpt", "openai"] = "chatgpt" + api_base: str | None = None + api_key: str | None = field(default=None, repr=False) + extra_headers: Mapping[str, str] = field(default_factory=lambda: MappingProxyType({}), repr=False) + extra_query: LiveQuery = field(default_factory=lambda: MappingProxyType({}), repr=False) + + +def _validate_session_id(session_id: str) -> None: + candidate: str = session_id # rebind-ok: inspect every decoding layer without recursive stack exhaustion + while True: + if ( + not candidate + or candidate in (".", "..") + or any(char in ("/", "\\") or category(char) in ("Cc", "Cs") for char in candidate) + ): + raise ValueError("Invalid Live session ID") + decoded: str = unquote(candidate, errors="strict") # rebind-ok: validate successive decoding layers iteratively + if decoded == candidate: + return + candidate = decoded # rebind-ok: each percent-decoding pass reduces the input length + + +def _validate_path(path: str) -> None: + match: Final = _PATH.fullmatch(path) + if match is None: + raise ValueError("Invalid Live endpoint") + if match.group(1) is None: + return + session_id: Final = unquote(match.group(1), errors="strict") + if quote(session_id, safe="") != match.group(1): + raise ValueError("Noncanonical Live session path") + _validate_session_id(session_id) + + +def live_session_path(session_id: str, operation: LiveOperation) -> str: + _validate_session_id(session_id) + path: Final = f"live/sessions/{quote(session_id, safe='')}/{operation}" + _validate_path(path) + return path + + +class LiveTransport: + def __init__( + self, + deployment: LiveDeployment, + inbound_headers: Mapping[str, str], + *, + http_client: httpx.AsyncClient | None = None, + ) -> None: + self.deployment = deployment + self._http_client = http_client + params: Final = GenericLiteLLMParams.model_validate( + MappingProxyType( + { + "api_base": deployment.api_base, + "extra_query": deployment.extra_query, + } + ) + ) + self._query = configured_realtime_query(params) + self._headers = ( + realtime_headers(params, inbound_headers, deployment.extra_headers) + if deployment.provider == "chatgpt" + else MappingProxyType( + { + **MappingProxyType( + { + key.lower(): value + for key, value in inbound_headers.items() + if key.lower() in ("openai-alpha", "openai-beta", "x-session-id", "x-oai-attestation") + } + ), + **configured_realtime_headers(deployment.extra_headers), + "authorization": f"Bearer {deployment.api_key or ''}", + } + ) + ) + + def _url(self, path: str, query: LiveQuery | None, *, websocket: bool) -> str: + _validate_path(path) + base: Final = httpx.URL( + ChatGPTRealtime.get_api_base(self.deployment.api_base) + if self.deployment.provider == "chatgpt" + else self.deployment.api_base or "https://api.openai.com/v1" + ) + if base.scheme not in ("https", "http", "wss", "ws") or not base.host or base.userinfo or base.fragment: + raise ValueError("Invalid Live API base") + merged: Final = base.params.merge(query or MappingProxyType({})).merge(self._query) + safe_query: Final = tuple( + (key, value) for key, value in merged.multi_items() if key.lower() not in _ROUTING_QUERY + ) + return str( + base.copy_with( + scheme=("wss" if base.scheme in ("https", "wss") else "ws") + if websocket + else ("https" if base.scheme in ("https", "wss") else "http"), + path=f"{base.path.rstrip('/')}/{path}", + params=safe_query, + ) + ) + + async def request( + self, + method: str, + path: str, + body: LiveBody | None = None, + query: LiveQuery | None = None, + ) -> httpx.Response: + if (method, path.rsplit("/", 1)[-1]) not in ( + ("POST", "sessions"), + ("POST", "fork"), + ("POST", "accept"), + ("POST", "reject"), + ("POST", "refer"), + ("POST", "hangup"), + ("GET", "content"), + ): + raise ValueError("Invalid Live HTTP operation") + url: Final = self._url(path, query, websocket=False) + client: Final = ( + self._http_client + or get_async_httpx_client( + llm_provider=LlmProviders.CHATGPT if self.deployment.provider == "chatgpt" else LlmProviders.OPENAI + ).client + ) + return await client.request( + method, + url, + headers=MappingProxyType({**self._headers, "content-type": "application/json"}), + json=dict(body) if body is not None else None, # mutable-ok: JSON encoder requires a concrete dict + timeout=60, + follow_redirects=False, + ) + + async def connect(self, path: str, query: LiveQuery | None = None) -> "ClientConnection": + import websockets + + class DirectConnect(websockets.connect): + def process_redirect(self, exc: Exception) -> Exception: + return exc + + if path != "live/sessions" and path.rsplit("/", 1)[-1] not in ("attach", "fork"): + raise ValueError("Invalid Live WebSocket operation") + url: Final = self._url(path, query, websocket=True) + ssl_context: Final = get_shared_realtime_ssl_context() if url.startswith("wss://") else None + return await DirectConnect( + url, + additional_headers=self._headers, + max_size=REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES, + max_queue=16, + ssl=True if ssl_context is False else ssl_context, + open_timeout=20, + close_timeout=10, + ) diff --git a/litellm/proxy/_lazy_features.py b/litellm/proxy/_lazy_features.py index dd1180b30ad..172be2250fe 100644 --- a/litellm/proxy/_lazy_features.py +++ b/litellm/proxy/_lazy_features.py @@ -213,10 +213,15 @@ LAZY_FEATURES: Final[tuple[LazyFeature, ...]] = ( "/watsonx/", ), ), + LazyFeature( + name="live", + module_path="litellm.proxy.realtime_endpoints.live", + path_prefixes=("/openai/v1/live/sessions", "/v1/live/sessions", "/live/sessions"), + ), LazyFeature( name="realtime", module_path="litellm.proxy.realtime_endpoints.endpoints", - path_prefixes=("/openai/v1/realtime", "/v1/realtime", "/realtime"), + path_prefixes=("/openai/v1/realtime", "/v1/realtime", "/realtime", "/openai/v1/live", "/v1/live", "/live"), ), LazyFeature( name="anthropic_passthrough", diff --git a/litellm/proxy/_lazy_openapi_snapshot.json b/litellm/proxy/_lazy_openapi_snapshot.json index 3be39265de7..3fbb0e48a0b 100644 --- a/litellm/proxy/_lazy_openapi_snapshot.json +++ b/litellm/proxy/_lazy_openapi_snapshot.json @@ -15890,6 +15890,844 @@ } } }, + "live": { + "components": { + "schemas": { + "HTTPValidationError": { + "properties": { + "detail": { + "items": { + "$ref": "#/components/schemas/ValidationError" + }, + "title": "Detail", + "type": "array" + } + }, + "title": "HTTPValidationError", + "type": "object" + }, + "ValidationError": { + "properties": { + "ctx": { + "title": "Context", + "type": "object" + }, + "input": { + "title": "Input" + }, + "loc": { + "items": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + } + ] + }, + "title": "Location", + "type": "array" + }, + "msg": { + "title": "Message", + "type": "string" + }, + "type": { + "title": "Error Type", + "type": "string" + } + }, + "required": [ + "loc", + "msg", + "type" + ], + "title": "ValidationError", + "type": "object" + } + } + }, + "paths": { + "/live/sessions": { + "post": { + "operationId": "create_live_session_live_sessions_post", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "live" + ] + } + }, + "/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__accept_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_live_sessions__session_id__content_get", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_live_sessions__session_id__fork_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "live" + ] + } + }, + "/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__hangup_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__refer_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__reject_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions": { + "post": { + "operationId": "create_live_session_openai_v1_live_sessions_post_2", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__accept_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__content_get_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_openai_v1_live_sessions__session_id__fork_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__hangup_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__refer_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__reject_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions": { + "post": { + "operationId": "create_live_session_v1_live_sessions_post", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__accept_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_v1_live_sessions__session_id__content_get", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_v1_live_sessions__session_id__fork_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__hangup_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__refer_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + }, + "/v1/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__reject_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "live" + ] + } + } + } + }, "llm_passthrough": { "components": { "schemas": { @@ -19222,6 +20060,284 @@ ] } }, + "/openai/v1/live": { + "post": { + "operationId": "proxy_live_calls_openai_v1_live_post", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Proxy Live Calls", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions": { + "post": { + "operationId": "create_live_session_openai_v1_live_sessions_post", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__accept_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__content_get", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_openai_v1_live_sessions__session_id__fork_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__hangup_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__refer_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__reject_post", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "llm_passthrough" + ] + } + }, "/openai/v1/realtime/calls": { "post": { "operationId": "proxy_realtime_calls_openai_v1_realtime_calls_post", @@ -37024,6 +38140,19 @@ "realtime": { "components": { "schemas": { + "HTTPValidationError": { + "properties": { + "detail": { + "items": { + "$ref": "#/components/schemas/ValidationError" + }, + "title": "Detail", + "type": "array" + } + }, + "title": "HTTPValidationError", + "type": "object" + }, "RealtimeClientSecretResponse": { "description": "Response from POST /v1/realtime/client_secrets.\n\nBoth the top-level `value` and `session.client_secret.value`\nwill contain the encrypted token instead of the raw ephemeral key.\nThe `session` field is kept as a raw dict so unknown fields pass through.", "properties": { @@ -37080,10 +38209,606 @@ }, "title": "RealtimeTranscriptionSessionResponse", "type": "object" + }, + "ValidationError": { + "properties": { + "ctx": { + "title": "Context", + "type": "object" + }, + "input": { + "title": "Input" + }, + "loc": { + "items": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + } + ] + }, + "title": "Location", + "type": "array" + }, + "msg": { + "title": "Message", + "type": "string" + }, + "type": { + "title": "Error Type", + "type": "string" + } + }, + "required": [ + "loc", + "msg", + "type" + ], + "title": "ValidationError", + "type": "object" } } }, "paths": { + "/live": { + "post": { + "operationId": "proxy_live_calls_live_post", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Proxy Live Calls", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions": { + "post": { + "operationId": "create_live_session_live_sessions_post_2", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__accept_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_live_sessions__session_id__content_get_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_live_sessions__session_id__fork_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__hangup_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__refer_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_live_sessions__session_id__reject_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live": { + "post": { + "operationId": "proxy_live_calls_openai_v1_live_post_2", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Proxy Live Calls", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions": { + "post": { + "operationId": "create_live_session_openai_v1_live_sessions_post_3", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__accept_post_3", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__content_get_3", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_openai_v1_live_sessions__session_id__fork_post_3", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__hangup_post_3", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__refer_post_3", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/openai/v1/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_openai_v1_live_sessions__session_id__reject_post_3", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, "/openai/v1/realtime/calls": { "post": { "operationId": "proxy_realtime_calls_openai_v1_realtime_calls_post_2", @@ -37228,6 +38953,284 @@ ] } }, + "/v1/live": { + "post": { + "operationId": "proxy_live_calls_v1_live_post", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Proxy Live Calls", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions": { + "post": { + "operationId": "create_live_session_v1_live_sessions_post_2", + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + } + }, + "summary": "Create Live Session", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions/{session_id}/accept": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__accept_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions/{session_id}/content": { + "get": { + "operationId": "control_live_session_v1_live_sessions__session_id__content_get_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions/{session_id}/fork": { + "post": { + "operationId": "fork_live_session_v1_live_sessions__session_id__fork_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Fork Live Session", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions/{session_id}/hangup": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__hangup_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions/{session_id}/refer": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__refer_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, + "/v1/live/sessions/{session_id}/reject": { + "post": { + "operationId": "control_live_session_v1_live_sessions__session_id__reject_post_2", + "parameters": [ + { + "in": "path", + "name": "session_id", + "required": true, + "schema": { + "title": "Session Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Control Live Session", + "tags": [ + "realtime" + ] + } + }, "/v1/realtime/calls": { "post": { "operationId": "proxy_realtime_calls_v1_realtime_calls_post", diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index 7b111c1513e..a757a02c6ec 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -412,6 +412,33 @@ class LiteLLMRoutes(enum.Enum): "/live", "/v1/live", "/v1/live/{call_id}", + "/openai/v1/live", + "/live/{call_id}", + "/openai/v1/live/{call_id}", + "/live/sessions", + "/live/sessions/{session_id}/attach", + "/live/sessions/{session_id}/fork", + "/live/sessions/{session_id}/content", + "/live/sessions/{session_id}/accept", + "/live/sessions/{session_id}/reject", + "/live/sessions/{session_id}/refer", + "/live/sessions/{session_id}/hangup", + "/v1/live/sessions", + "/v1/live/sessions/{session_id}/attach", + "/v1/live/sessions/{session_id}/fork", + "/v1/live/sessions/{session_id}/content", + "/v1/live/sessions/{session_id}/accept", + "/v1/live/sessions/{session_id}/reject", + "/v1/live/sessions/{session_id}/refer", + "/v1/live/sessions/{session_id}/hangup", + "/openai/v1/live/sessions", + "/openai/v1/live/sessions/{session_id}/attach", + "/openai/v1/live/sessions/{session_id}/fork", + "/openai/v1/live/sessions/{session_id}/content", + "/openai/v1/live/sessions/{session_id}/accept", + "/openai/v1/live/sessions/{session_id}/reject", + "/openai/v1/live/sessions/{session_id}/refer", + "/openai/v1/live/sessions/{session_id}/hangup", # realtime (GA WebRTC HTTP routes) "/realtime/client_secrets", "/v1/realtime/client_secrets", diff --git a/litellm/proxy/auth/auth_utils.py b/litellm/proxy/auth/auth_utils.py index 2cb739878bd..1b096e60b09 100644 --- a/litellm/proxy/auth/auth_utils.py +++ b/litellm/proxy/auth/auth_utils.py @@ -1705,6 +1705,7 @@ _MODEL_ROUTING_COMPLETION_MODEL_ROUTE_MARKERS: Final = ("/evals",) _MODEL_ROUTING_SESSION_MODEL_ROUTE_MARKERS: Final = ( "/realtime/client_secrets", "/realtime/calls", + "/live/sessions", ) _MODEL_ROUTING_ID_FIELDS: Final = ( "file_id", @@ -1866,15 +1867,15 @@ def _extract_model_candidates_from_request( uses_completion_model_sources: Final = _route_matches_any_marker( route=route, markers=_MODEL_ROUTING_COMPLETION_MODEL_ROUTE_MARKERS ) - session: Final[object] = ( - request_data.get("session") - if _route_matches_any_marker(route=route, markers=_MODEL_ROUTING_SESSION_MODEL_ROUTE_MARKERS) - else None - ) + uses_session_model: Final = _route_matches_any_marker( + route=route, markers=_MODEL_ROUTING_SESSION_MODEL_ROUTE_MARKERS + ) or route.rstrip("/") in ("/live", "/v1/live", "/openai/v1/live") + session: Final[object] = request_data.get("session") if uses_session_model else None parsed_session: Final[object] = safe_json_loads(session) if isinstance(session, str) else session session_model: Final[object] = parsed_session.get("model") if isinstance(parsed_session, dict) else None if ( - _route_matches_any_marker(route=route, markers=("/realtime/calls",)) + uses_session_model + and not _route_matches_any_marker(route=route, markers=("/realtime/client_secrets",)) and isinstance(session_model, str) and session_model ): @@ -1885,7 +1886,7 @@ def _extract_model_candidates_from_request( _append_model_candidates(candidates, body_model) if uses_body_target_model_sources or not body_model: _append_model_candidates(candidates, request_data.get("target_model_names")) - if _route_matches_any_marker(route=route, markers=_MODEL_ROUTING_SESSION_MODEL_ROUTE_MARKERS): + if uses_session_model: _append_model_candidates(candidates, session_model) if uses_completion_model_sources and isinstance(request_data.get("completion"), dict): _append_model_candidates(candidates, request_data["completion"].get("model")) diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py index 6c17701e961..d51574de8b8 100644 --- a/litellm/proxy/proxy_server.py +++ b/litellm/proxy/proxy_server.py @@ -11920,7 +11920,12 @@ async def _reject_realtime_session( _CODEX_LIVE_AUTH_DEPENDENCY: Final = Depends(user_api_key_auth_websocket) +reserve_lazy_slot(app, "live") +reserve_lazy_slot(app, "realtime") + +@app.websocket("/openai/v1/live/{call_id}") +@app.websocket("/live/{call_id}") @app.websocket("/v1/live/{call_id}") async def codex_live_sideband_endpoint( websocket: WebSocket, @@ -11934,6 +11939,7 @@ async def codex_live_sideband_endpoint( @app.websocket("/v1/live") @app.websocket("/live") +@app.websocket("/openai/v1/live") @app.websocket("/openai/v1/realtime") @app.websocket("/v1/realtime") @app.websocket("/realtime") diff --git a/litellm/proxy/realtime_endpoints/call_sessions.py b/litellm/proxy/realtime_endpoints/call_sessions.py index 8989fadfead..d1908919503 100644 --- a/litellm/proxy/realtime_endpoints/call_sessions.py +++ b/litellm/proxy/realtime_endpoints/call_sessions.py @@ -356,7 +356,13 @@ async def _create_codex_realtime_call(request: Request) -> Response: valid_token=auth, llm_router=server.llm_router, ) - data: Final = build_call_request(offer, request.query_params, request.headers) + live_signaling: Final = request.url.path.rstrip("/") in ("/live", "/v1/live", "/openai/v1/live") + query: Final = ( + MappingProxyType({"intent": "quicksilver", "architecture": "avas", **request.query_params}) + if live_signaling + else request.query_params + ) + data: Final = build_call_request(offer, query, request.headers) signaling_auth: Final = auth.model_copy(update=MappingProxyType({"budget_reservation": None})) if isinstance(limiter, _PROXY_MaxParallelRequestsHandler) and ( auth.max_parallel_requests is not None @@ -409,7 +415,9 @@ async def _create_codex_realtime_call(request: Request) -> Response: response.content, status_code=response.status_code, media_type="application/sdp", - headers=MappingProxyType({"Location": f"/v1/realtime/calls/{token}"}), + headers=MappingProxyType( + {"Location": f"/v1/live/{token}" if live_signaling else f"/v1/realtime/calls/{token}"} + ), ) finally: try: diff --git a/litellm/proxy/realtime_endpoints/call_supervision.py b/litellm/proxy/realtime_endpoints/call_supervision.py index f511d943c3a..3f9ed521624 100644 --- a/litellm/proxy/realtime_endpoints/call_supervision.py +++ b/litellm/proxy/realtime_endpoints/call_supervision.py @@ -3,7 +3,7 @@ from collections.abc import AsyncIterator, Awaitable, Callable from contextlib import suppress from typing import Final, Protocol -from pydantic import BaseModel, Field, ValidationError +from pydantic import BaseModel, ValidationError from websockets.exceptions import ConnectionClosedOK from litellm._logging import verbose_proxy_logger @@ -16,6 +16,7 @@ from litellm.proxy.spend_tracking.budget_reservation import ( invalidate_budget_reservation_counters, release_or_invalidate_budget_reservation, ) +from litellm.types.realtime import LiveSessionUsageEvent class ObserverSocket(Protocol): @@ -34,14 +35,6 @@ class _ObserverEvent(BaseModel): type: str -class _LiveDurationUsage(BaseModel): - audio_duration_ms: float = Field(strict=True, ge=0, allow_inf_nan=False) - - -class _LiveTerminalEvent(BaseModel): - usage: _LiveDurationUsage - - class CallSupervisor: def __init__( self, @@ -57,6 +50,7 @@ class CallSupervisor: termination_timeout: float = 60, logging_timeout: float = LOGGING_WORKER_MAX_TIME_PER_COROUTINE, terminal_usage_required: bool = True, + connected_ready: bool = False, force_close_call: Callable[[], Awaitable[None]] | None = None, lease: RealtimeCallLease | None = None, ) -> None: @@ -75,7 +69,9 @@ class CallSupervisor: self._terminal_usage_required = terminal_usage_required self._ready = asyncio.Event() self._stop = asyncio.Event() - self._started = False + self._started = connected_ready + if connected_ready: + self._ready.set() self._terminal = False self._terminal_usage_valid = False self._close_confirmed = False @@ -128,7 +124,7 @@ class CallSupervisor: if event.type == "session.closed": self._terminal = True try: - _LiveTerminalEvent.model_validate_json(message) + LiveSessionUsageEvent.model_validate_json(message) except ValidationError: self._terminal_usage_valid = False else: @@ -206,7 +202,9 @@ class CallSupervisor: await asyncio.wait_for( self._stream.log_messages(wait_for_dispatch=True), timeout=self._logging_timeout ) - self._accounting_complete = True + self._accounting_complete = not bool( + self._logging.model_call_details.get("realtime_backend_accounting_incomplete") + ) except asyncio.TimeoutError: verbose_proxy_logger.error("Realtime observer timed out dispatching usage accounting") finally: diff --git a/litellm/proxy/realtime_endpoints/endpoints.py b/litellm/proxy/realtime_endpoints/endpoints.py index 82041b4274f..c1da14cb667 100644 --- a/litellm/proxy/realtime_endpoints/endpoints.py +++ b/litellm/proxy/realtime_endpoints/endpoints.py @@ -362,6 +362,15 @@ async def create_realtime_client_secret( return RealtimeClientSecretResponse(**upstream_json) +@router.post("/v1/live", tags=["realtime"]) +@router.post("/live", tags=["realtime"]) +@router.post("/openai/v1/live", tags=["realtime"]) +async def proxy_live_calls(request: Request) -> Response: + from litellm.proxy.realtime_endpoints.call_sessions import create_codex_realtime_call + + return await create_codex_realtime_call(request) + + @router.post( "/v1/realtime/calls", tags=["realtime"], diff --git a/litellm/proxy/realtime_endpoints/live.py b/litellm/proxy/realtime_endpoints/live.py new file mode 100644 index 00000000000..77e732f65fa --- /dev/null +++ b/litellm/proxy/realtime_endpoints/live.py @@ -0,0 +1,1070 @@ +import asyncio +import base64 +import hashlib +import json +import time +from collections.abc import AsyncGenerator, Mapping +from contextlib import asynccontextmanager, nullcontext +from types import MappingProxyType +from typing import TYPE_CHECKING, Final + +import httpx +from fastapi import APIRouter, HTTPException, Request, Response, WebSocket, WebSocketDisconnect +from pydantic import BaseModel, Field, JsonValue, TypeAdapter +from starlette.types import Message + +if TYPE_CHECKING: + from websockets.asyncio.client import ClientConnection + +from litellm.litellm_core_utils.litellm_logging import Logging +from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming +from litellm.llms.chatgpt.live import LiveDeployment, LiveOperation, LiveTransport, live_session_path +from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth +from litellm.proxy.auth.auth_checks import ( + can_key_call_resolved_model, # pyright: ignore[reportUnknownVariableType] # legacy authorization accepts untyped deployment lists +) +from litellm.proxy.auth.user_api_key_auth import get_websocket_api_key, user_api_key_auth +from litellm.proxy.common_utils.encrypt_decrypt_utils import decrypt_value_helper, encrypt_value_helper +from litellm.proxy.hooks.parallel_request_limiter import ( + _PROXY_MaxParallelRequestsHandler, # pyright: ignore[reportPrivateUsage] # limiter class is the existing hook identity +) +from litellm.proxy.hooks.parallel_request_limiter_v3 import ( + _PROXY_MaxParallelRequestsHandler_v3, # pyright: ignore[reportPrivateUsage] # slot transfer is available on this existing hook + isolated_request_stash, +) +from litellm.proxy.hooks.realtime_call_lease import RealtimeCallLease, realtime_call_attachment +from litellm.proxy.realtime_endpoints.call_sessions import process_codex_request +from litellm.proxy.realtime_endpoints.call_supervision import CALL_SUPERVISORS, CallSupervisor +from litellm.proxy.spend_tracking.budget_reservation import ( + release_or_invalidate_budget_reservation, # pyright: ignore[reportUnknownVariableType] # budget helper accepts legacy reservation dicts +) + +_routes: Final = APIRouter() +_JSON: Final = TypeAdapter[JsonValue](JsonValue) +_EMPTY: Final[Mapping[str, JsonValue]] = MappingProxyType({}) +_MAPPING: Final = TypeAdapter(Mapping[str, object]) +_OBJECT: Final = TypeAdapter(Mapping[str, JsonValue]) +_DEPLOYMENT: Final = TypeAdapter(LiveDeployment) +_PREFIX: Final = "live_litellm_" + + +def _json_value(value: object) -> JsonValue: + if isinstance(value, Mapping): + entries: Final = _MAPPING.validate_python(value) + return {key: _json_value(item) for key, item in entries.items()} # mutable-ok: JSON wire objects require dicts + if isinstance(value, (tuple, list)): + items: Final = TypeAdapter(tuple[object, ...]).validate_python(value) + return [_json_value(item) for item in items] # mutable-ok: JSON wire arrays require lists + return _JSON.validate_python(value) + + +def _object(value: object) -> Mapping[str, JsonValue]: + return _OBJECT.validate_python(_json_value(value)) + + +def _mutable( + value: Mapping[str, object], +) -> dict[str, object]: # mutable-ok: legacy proxy and ASGI contracts mutate inputs + return dict(value) # mutable-ok: make the mutable copy at the framework boundary + + +def _encode_json(value: object) -> str: + return json.dumps(_json_value(value)) + + +class _ConnectionState: + def __init__(self) -> None: + self.connection: ClientConnection | None = None + + +class LiveHandle(BaseModel): + session_id: str + alias: str + deployment: Mapping[str, JsonValue] + owner: str + expires_at: float + parallel_reserved: bool = False + initialization_seconds: float = 0 + policy: Mapping[str, JsonValue] = Field(default_factory=lambda: _EMPTY) + + +def encode_session(handle: LiveHandle) -> str: + encrypted: Final = encrypt_value_helper(handle.model_dump_json()) + return _PREFIX + base64.urlsafe_b64encode(encrypted.encode()).decode().rstrip("=") + + +def decode_session(token: str, owner: str) -> LiveHandle: + try: + if not token.startswith(_PREFIX): + raise ValueError("Invalid prefix") + encoded: Final = token[len(_PREFIX) :] + encrypted: Final = base64.b64decode(encoded + "=" * (-len(encoded) % 4), altchars=b"-_", validate=True) + handle: Final = LiveHandle.model_validate_json( + decrypt_value_helper(encrypted.decode(), key="live_session") or "" + ) + if handle.owner != owner or handle.expires_at <= time.time(): + raise ValueError("Invalid ownership or expiry") + live_session_path(handle.session_id, "attach") + return handle + except (ValueError, TypeError, UnicodeError) as exc: + raise HTTPException(403, "Invalid or expired Live session") from exc + + +def rewrite_session_ids(value: JsonValue | Mapping[str, JsonValue], raw_id: str, public_id: str) -> JsonValue: + if not isinstance(value, Mapping): + return _json_value(value) + session: Final = value.get("session") + return _json_value( + MappingProxyType( + { + **value, + **(MappingProxyType({"session_id": public_id}) if value.get("session_id") == raw_id else _EMPTY), + **( + MappingProxyType({"session": MappingProxyType({**session, "id": public_id})}) + if isinstance(session, Mapping) and session.get("id") == raw_id + else _EMPTY + ), + } + ) + ) + + +def _owner(auth: UserAPIKeyAuth) -> str: + if not auth.api_key: + raise HTTPException(403, "Live sessions require an authenticated API key") + return hashlib.sha256(auth.api_key.encode()).hexdigest() + + +async def _auth(request: Request) -> UserAPIKeyAuth: + return await user_api_key_auth( + request=request, + api_key=request.headers.get("authorization", ""), + azure_api_key_header=request.headers.get("api-key", ""), + anthropic_api_key_header=None, + google_ai_studio_api_key_header=None, + azure_apim_header=None, + custom_litellm_key_header=request.headers.get("x-litellm-api-key"), + ) + + +async def _body(request: Request) -> Mapping[str, JsonValue]: + from litellm.proxy.realtime_endpoints.call_sessions import MAX_REALTIME_OFFER_BYTES + + chunks: Final = bytearray() + async for chunk in request.stream(): + if len(chunks) + len(chunk) > MAX_REALTIME_OFFER_BYTES: + raise HTTPException(413, "Live request exceeds the 8 MiB limit") + chunks.extend(chunk) + try: + return _OBJECT.validate_json(bytes(chunks)) if chunks else _EMPTY + except ValueError as exc: + raise HTTPException(400, "Expected a JSON object") from exc + + +def _request(source: Request | WebSocket, body: Mapping[str, JsonValue]) -> Request: + async def receive() -> Message: + return _mutable( + MappingProxyType({"type": "http.request", "body": _encode_json(body).encode(), "more_body": False}) + ) + + return Request(_mutable(MappingProxyType({**source.scope, "type": "http", "method": "POST"})), receive=receive) + + +def _session_model(body: Mapping[str, JsonValue], fallback: str | None = None) -> str: + session: Final = body.get("session", _EMPTY) + if not isinstance(session, Mapping): + raise HTTPException(400, "session must be a JSON object") + model: Final = session.get("model", fallback) + if not isinstance(model, str) or not model: + raise HTTPException(400, "session.model is required") + if fallback is not None and "model" in session: + raise HTTPException(400, "A session fork cannot change the authorized model") + return model + + +async def _authorize(model: str, auth: UserAPIKeyAuth) -> None: + from litellm.proxy import proxy_server as server + + await can_key_call_resolved_model( + model=model, + llm_model_list=list( # mutable-ok: legacy authorization requires a concrete list + TypeAdapter(tuple[Mapping[str, object], ...]).validate_python(server.llm_model_list or ()) # pyright: ignore[reportUnknownMemberType] # legacy registry is validated here + ), + valid_token=auth, + llm_router=server.llm_router, + ) + + +async def _deployment(model: str, processed: Mapping[str, object]) -> LiveDeployment: + from litellm.proxy import proxy_server as server + + if server.llm_router is None: + raise HTTPException(503, "Live requires a configured model deployment") + selected: Final = _MAPPING.validate_python( + await server.llm_router.async_get_available_deployment( # pyright: ignore[reportUnknownMemberType] # validate the legacy router result at this boundary + model=model, request_kwargs=_mutable(processed) + ) + ) + await server.llm_router.async_routing_strategy_pre_call_checks(_mutable(selected), None) # pyright: ignore[reportUnknownMemberType] # existing routing strategy has an untyped deployment contract + params: Final = _object(selected["litellm_params"]) + qualified: Final = str(params["model"]) + prefix, _, suffix = qualified.partition("/") + provider: Final = prefix if prefix in ("openai", "chatgpt") else "openai" + upstream: Final = suffix if prefix in ("openai", "chatgpt") else qualified + if provider == "chatgpt" and any( + params.get(key) is not None for key in ("chatgpt_auth_profile", "chatgpt_token_dir", "chatgpt_auth_file") + ): + raise HTTPException( + 400, "ChatGPT Live uses the proxy OAuth credentials; deployment auth overrides are unsupported" + ) + if prefix not in ("openai", "chatgpt"): + if "/" in qualified: + raise HTTPException(400, "Live requires an OpenAI or ChatGPT deployment") + return _DEPLOYMENT.validate_python( + _mutable( + MappingProxyType( + { + "model": upstream, + "provider": provider, + "model_id": str(_MAPPING.validate_python(selected["model_info"])["id"]), + "api_base": params.get("api_base"), + "api_key": params.get("api_key"), + "extra_headers": params.get("extra_headers") or _EMPTY, + "extra_query": params.get("extra_query") or _EMPTY, + } + ) + ) + ) + + +def _pinned(handle: LiveHandle) -> LiveDeployment: + return _DEPLOYMENT.validate_python(_mutable(handle.deployment)) + + +def _new_handle( + session_id: str, + alias: str, + deployment: LiveDeployment, + auth: UserAPIKeyAuth, + lease: RealtimeCallLease | None, + initialization_seconds: float = 0, + policy: Mapping[str, JsonValue] | None = None, +) -> LiveHandle: + live_session_path(session_id, "attach") + routing: Final = _object( + MappingProxyType( + { + "model": deployment.model, + "provider": deployment.provider, + "model_id": deployment.model_id, + "api_base": deployment.api_base, + "api_key": deployment.api_key, + "extra_headers": deployment.extra_headers, + "extra_query": deployment.extra_query, + } + ) + ) + return LiveHandle( + session_id=session_id, + alias=alias, + deployment=routing, + owner=_owner(auth), + expires_at=time.time() + 30 * 86400, + parallel_reserved=lease is not None, + initialization_seconds=initialization_seconds, + policy=policy or _EMPTY, + ) + + +def _session_id(payload: Mapping[str, JsonValue]) -> str: + session: Final = payload.get("session") + if isinstance(session, Mapping) and isinstance(session.get("id"), str): + return TypeAdapter(str).validate_python(session["id"]) + raise HTTPException(502, "Upstream did not return a Live session ID") + + +class _BudgetOwnership: + def __init__(self, auth: UserAPIKeyAuth) -> None: + self.auth = auth + self.transferred = False + + def replace_auth(self, auth: UserAPIKeyAuth) -> None: + self.auth = auth + + +@asynccontextmanager +async def _budget_scope(auth: UserAPIKeyAuth) -> AsyncGenerator[_BudgetOwnership]: + ownership: Final = _BudgetOwnership(auth) + try: + yield ownership + finally: + if not ownership.transferred: + await release_or_invalidate_budget_reservation(budget_reservation=ownership.auth.budget_reservation) + + +async def _reauth(ownership: _BudgetOwnership, request: Request, body: Mapping[str, JsonValue], model: str) -> None: + await release_or_invalidate_budget_reservation(budget_reservation=ownership.auth.budget_reservation) + ownership.replace_auth(await _auth(_request(request, MappingProxyType({**body, "model": model})))) + + +def _policy_object(value: object) -> Mapping[str, JsonValue]: + try: + return _object(value) + except ValueError as exc: + raise HTTPException(400, "Live session delegation and responses must be JSON objects") from exc + + +def _policy_body(body: Mapping[str, JsonValue], source: LiveHandle | None) -> Mapping[str, JsonValue]: + current: Final = _policy_object(body.get("session", _EMPTY)) + inherited: Final = source.policy if source is not None else _EMPTY + parent_delegation: Final = _policy_object(inherited.get("delegation") or _EMPTY) + child_delegation: Final = _policy_object(current.get("delegation") or _EMPTY) + merged_delegation: Final = MappingProxyType( + { + **parent_delegation, + **child_delegation, + "responses": MappingProxyType( + { + **_policy_object(parent_delegation.get("responses") or _EMPTY), + **_policy_object(child_delegation.get("responses") or _EMPTY), + } + ), + } + ) + return _policy_object( + MappingProxyType( + { + **body, + "session": MappingProxyType( + { + **inherited, + **current, + **( + MappingProxyType({"delegation": merged_delegation}) + if parent_delegation or child_delegation + else _EMPTY + ), + } + ), + } + ) + ) + + +def _session_policy(body: Mapping[str, JsonValue], source: LiveHandle | None) -> Mapping[str, JsonValue]: + session: Final = _object(_policy_body(body, source)["session"]) + delegation: Final = _object(session.get("delegation") or _EMPTY) + responses: Final = _object(delegation.get("responses") or _EMPTY) + return _object( + MappingProxyType( + { + **( + MappingProxyType( + { + "delegation": MappingProxyType( + { + "type": delegation.get("type"), + "responses": MappingProxyType({"model": responses.get("model")}), + } + ) + } + ) + if delegation.get("type") == "responses" or responses.get("model") + else _EMPTY + ), + **(MappingProxyType({"client": session["client"]}) if "client" in session else _EMPTY), + } + ) + ) + + +def _managed_constraints(auth: UserAPIKeyAuth) -> bool: + values: Final = _object( + auth.model_dump( + include=MappingProxyType( + { + name: True + for name in ( + "model_max_budget", + "user_model_max_budget", + "end_user_model_max_budget", + "rpm_limit_per_model", + "tpm_limit_per_model", + "rpm_limit", + "tpm_limit", + "team_rpm_limit", + "team_tpm_limit", + "user_rpm_limit", + "user_tpm_limit", + "team_metadata", + "metadata", + "organization_metadata", + "project_metadata", + ) + } + ) + ) + ) + + def constrained(value: JsonValue | Mapping[str, JsonValue]) -> bool: + if not isinstance(value, Mapping): + return False + return any( + bool(item) + if any(marker in key for marker in ("rpm_limit", "tpm_limit", "model_max_budget")) + else constrained(item) + for key, item in value.items() + ) + + return constrained(values) + + +def _restricted_models(auth: UserAPIKeyAuth) -> bool: + key_models: Final = TypeAdapter(tuple[str, ...]).validate_python(getattr(auth, "models", ()) or ()) + team_models: Final = TypeAdapter(tuple[str, ...]).validate_python(getattr(auth, "team_models", ()) or ()) + return any( + models and "*" not in models and "all-proxy-models" not in models for models in (key_models, team_models) + ) + + +async def _authorize_delegation(body: Mapping[str, JsonValue], auth: UserAPIKeyAuth) -> None: + session: Final = body.get("session") + if not isinstance(session, Mapping): + return + delegation: Final = session.get("delegation") + if not isinstance(delegation, dict) or delegation.get("type") == "client": + return + responses: Final = delegation.get("responses") + if delegation.get("type") != "responses" and not isinstance(responses, dict): + return + if _managed_constraints(auth): + raise HTTPException( + 400, + "Managed Live delegation cannot enforce configured backend model budgets or rate limits; use client delegation", + ) + if not isinstance(responses, dict): + if _restricted_models(auth): + raise HTTPException(400, "Restricted keys require an explicit authorized delegation.responses.model") + return + model: Final = responses.get("model") + if isinstance(model, str): + await _authorize(model, auth) + elif body.get("type") != "session.update" and _restricted_models(auth): + raise HTTPException(400, "Restricted keys require an explicit authorized delegation.responses.model") + transport: Final = body.get("transport") + if isinstance(transport, dict) and transport.get("type") == "webrtc" and _restricted_models(auth): + client: Final = session.get("client") + channel: Final = client.get("data_channel") if isinstance(client, dict) else None + events: Final = channel.get("allowed_client_events") if isinstance(channel, dict) else None + if not isinstance(events, list) or "session.update" in events: + raise HTTPException( + 400, "Restricted keys must explicitly exclude session.update from WebRTC allowed_client_events" + ) + + +async def _authorize_fork_policy( + body: Mapping[str, JsonValue], source: LiveHandle | None, auth: UserAPIKeyAuth +) -> None: + if source is not None and (_restricted_models(auth) or _managed_constraints(auth)): + # Handles contain startup policy; later sideband or WebRTC updates can change the backend model. + session: Final = _policy_object(body.get("session", _EMPTY)) + delegation: Final = _policy_object(session.get("delegation") or _EMPTY) + responses: Final = _policy_object(delegation.get("responses") or _EMPTY) + if delegation.get("type") != "client" and not ( + delegation.get("type") == "responses" and isinstance(responses.get("model"), str) and responses.get("model") + ): + raise HTTPException( + 400, "Constrained-key forks require explicit client delegation or an authorized responses model" + ) + await _authorize_delegation(_policy_body(body, source), auth) + + +class _Prepared: + def __init__( + self, + processed: Mapping[str, object], + logger: Logging, + lease: RealtimeCallLease | None, + ownership: _BudgetOwnership | None = None, + ) -> None: + self.processed = processed + self.logger = logger + self.lease = lease + self.transferred = False + self.ownership = ownership + + def transfer(self) -> None: + self.transferred = True + if self.ownership is not None: + self.ownership.transferred = True + + +class _PrecallState: + def __init__(self) -> None: + self.prepared: _Prepared | None = None + self.lease: RealtimeCallLease | None = None + + +@asynccontextmanager +async def _precall( + request: Request, + auth: UserAPIKeyAuth, + model: str, + *, + attachment: object | None = None, + parallel_reserved: bool = False, + ownership: _BudgetOwnership | None = None, +) -> AsyncGenerator[_Prepared]: + from litellm.proxy import proxy_server as server + + limiter: Final = server.proxy_logging_obj.get_proxy_hook("parallel_request_limiter") + with isolated_request_stash(): + state: Final = _PrecallState() + signaling_auth: Final = auth.model_copy(update=MappingProxyType({"budget_reservation": None})) + try: + await _authorize(model, auth) + if isinstance(limiter, _PROXY_MaxParallelRequestsHandler) and ( + auth.max_parallel_requests is not None + or _MAPPING.validate_python(server.general_settings).get("global_max_parallel_requests") # pyright: ignore[reportUnknownMemberType] # legacy settings are validated here is not None + ): + raise HTTPException(400, "Live requires the V3 rate limiter") + payload: Final = _OBJECT.validate_json(await request.body()) + await _authorize_delegation(payload, auth) + data: Final = MappingProxyType( + {key: value for key, value in payload.items() if key in ("session", "transport")} + ) + with ( + realtime_call_attachment(attachment) if attachment is not None and parallel_reserved else nullcontext() + ): + processed, logger = await process_codex_request( + request, + _mutable( + MappingProxyType( + { + **data, + "model": model, + **(MappingProxyType({"websocket": attachment}) if attachment is not None else _EMPTY), + } + ) + ), + signaling_auth, + model, + "_arealtime" if attachment is not None else "arealtime_calls", + ) + if attachment is None and isinstance(limiter, _PROXY_MaxParallelRequestsHandler_v3): + state.lease = limiter.transfer_realtime_call_slot(processed) + if state.lease is not None: + state.lease.start() + if not await state.lease.renew(): + raise HTTPException(503, "Live quota reservation was lost") + await _authorize_delegation(_processed_body(payload, processed), auth) + state.prepared = _Prepared(processed, logger, state.lease, ownership) + yield state.prepared + finally: + try: + if state.prepared is None or not state.prepared.transferred: + if state.lease is not None: + await state.lease.close() + if ownership is None: + await release_or_invalidate_budget_reservation(budget_reservation=auth.budget_reservation) + finally: + if isinstance(limiter, _PROXY_MaxParallelRequestsHandler_v3): + await limiter.async_post_call_failure_hook( # pyright: ignore[reportUnknownMemberType] # hook accepts legacy mutable request data + request_data=_mutable(_EMPTY), + original_exception=Exception("Live signaling complete"), + user_api_key_dict=signaling_auth, + ) + + +async def _supervise( + request: Request, handle: LiveHandle, auth: UserAPIKeyAuth, logger: Logging, lease: RealtimeCallLease | None +) -> RealTimeStreaming: + with isolated_request_stash(): + return await _start_supervisor(request, handle, auth, lease) + + +async def _start_supervisor( + request: Request, handle: LiveHandle, auth: UserAPIKeyAuth, lease: RealtimeCallLease | None +) -> RealTimeStreaming: + deployment: Final = _pinned(handle) + transport: Final = LiveTransport(deployment, request.headers) + state: Final = _ConnectionState() + try: + processed, logger = await process_codex_request( + _request(request, MappingProxyType({"model": handle.alias})), + _mutable(MappingProxyType({"model": handle.alias})), + auth, + handle.alias, + "_arealtime", + internal_realtime_observer=True, + ) + import litellm + + metadata: Final = _mutable( + MappingProxyType( + { + **_MAPPING.validate_python(processed.get("litellm_metadata") or _EMPTY), + **( + MappingProxyType( + { + "model_info": _mutable( + MappingProxyType( + { + **litellm.get_model_info(model=deployment.model_id), + "id": deployment.model_id, + } + ) + ) + } + ) + if deployment.model_id is not None + else _EMPTY + ), + } + ) + ) + pinned: Final = _mutable(MappingProxyType({**processed, "litellm_metadata": metadata})) + logger.update_from_kwargs( # pyright: ignore[reportUnknownMemberType] # logging accepts legacy mutable provider parameters + kwargs=pinned, + model=deployment.model, + user=None, + optional_params=_mutable(_EMPTY), + litellm_params=_mutable( + MappingProxyType( + { + **_MAPPING.validate_python(logger.litellm_params), # pyright: ignore[reportUnknownMemberType] # legacy logging parameters are validated here + "litellm_metadata": metadata, + "arealtime": True, + } + ) + ), + custom_llm_provider=deployment.provider, + ) + state.connection = await transport.connect(live_session_path(handle.session_id, "attach")) + + async def receive() -> Message: + return _mutable(MappingProxyType({"type": "websocket.disconnect", "code": 1000})) + + async def send(message: Message) -> None: + return None + + frontend: Final = WebSocket( + _mutable(MappingProxyType({**request.scope, "type": "websocket"})), receive=receive, send=send + ) + stream: Final = RealTimeStreaming( + frontend, + state.connection, + logger, + model=_pinned(handle).model, + user_api_key_dict=auth, + live_initialization_seconds=handle.initialization_seconds, + ) + + async def hangup() -> None: + result: Final = await transport.request("POST", live_session_path(handle.session_id, "hangup")) + result.raise_for_status() + + supervisor: Final = CallSupervisor( + state.connection, + stream, + logger, + auth, + hangup, + force_close_call=hangup, + lease=lease, + terminal_usage_required=True, + connected_ready=True, + ) + await CALL_SUPERVISORS.start(supervisor) + return stream + except BaseException: + from litellm.proxy.spend_tracking.budget_reservation import ( + invalidate_budget_reservation_counters, # pyright: ignore[reportUnknownVariableType] # reservation helper has a legacy dict contract + ) + + try: + result: Final = await transport.request("POST", live_session_path(handle.session_id, "hangup")) + result.raise_for_status() + except Exception: + await invalidate_budget_reservation_counters(budget_reservation=auth.budget_reservation) + finally: + if state.connection is not None: + await state.connection.close() + raise + + +def _processed_body(body: Mapping[str, JsonValue], processed: Mapping[str, object]) -> Mapping[str, JsonValue]: + return _object( + MappingProxyType( + { + **body, + **_object( + MappingProxyType( + {key: value for key, value in processed.items() if key in ("session", "transport")} + ) + ), + } + ) + ) + + +def _provider_body(body: Mapping[str, JsonValue], model: str) -> Mapping[str, JsonValue]: + session: Final = _object(body.get("session", _EMPTY)) + return _object(MappingProxyType({**body, "session": MappingProxyType({**session, "model": model})})) + + +def _response(response: httpx.Response, handle: LiveHandle | None = None) -> Response: + if handle is None: + return Response( + response.content, + status_code=response.status_code, + headers=MappingProxyType( + { + key: value + for key, value in response.headers.items() + if key.lower() in ("content-type", "content-disposition", "content-range", "accept-ranges") + } + ), + ) + return Response( + _encode_json( + rewrite_session_ids(_OBJECT.validate_json(response.content), handle.session_id, encode_session(handle)) + ), + status_code=response.status_code, + media_type="application/json", + ) + + +async def _create(request: Request, token: str | None = None) -> Response: + body: Final = await _body(request) + auth: Final = await _auth( + _request(request, _EMPTY if token else MappingProxyType({**body, "model": _session_model(body)})) + ) + async with _budget_scope(auth) as ownership: + source: Final = decode_session(token, _owner(auth)) if token else None + model: Final = _session_model(body, source.alias if source else None) + if source is not None: + await _reauth(ownership, request, body, model) + async with _precall( + _request(request, MappingProxyType({**body, "model": model})), ownership.auth, model, ownership=ownership + ) as prepared: + await _authorize_fork_policy(_processed_body(body, prepared.processed), source, ownership.auth) + deployment: Final = _pinned(source) if source else await _deployment(model, prepared.processed) + transport: Final = LiveTransport(deployment, request.headers) + path: Final = live_session_path(source.session_id, "fork") if source else "live/sessions" + response: Final = await transport.request( + "POST", + path, + body=_processed_body(body, prepared.processed) + if source + else _provider_body(_processed_body(body, prepared.processed), deployment.model), + ) + if response.is_error: + return _response(response) + handle: Final = _new_handle( + _session_id(_OBJECT.validate_json(response.content)), + model, + deployment, + ownership.auth, + prepared.lease, + initialization_seconds=15, + policy=_session_policy(_processed_body(body, prepared.processed), source), + ) + await _supervise(request, handle, ownership.auth, prepared.logger, prepared.lease) + prepared.transfer() + return _response(response, handle) + + +@_routes.post("/sessions") +async def create_live_session(request: Request) -> Response: + return await _create(request) + + +@_routes.post("/sessions/{session_id}/fork") +async def fork_live_session(request: Request, session_id: str) -> Response: + return await _create(request, session_id) + + +@_routes.get("/sessions/{session_id}/content") +@_routes.post("/sessions/{session_id}/accept") +@_routes.post("/sessions/{session_id}/reject") +@_routes.post("/sessions/{session_id}/refer") +@_routes.post("/sessions/{session_id}/hangup") +async def control_live_session(request: Request, session_id: str) -> Response: + body: Final = await _body(request) + auth: Final = await _auth(_request(request, _EMPTY)) + async with _budget_scope(auth) as ownership: + operation: Final = TypeAdapter[LiveOperation](LiveOperation).validate_python( + request.url.path.rsplit("/", 1)[-1] + ) + if not session_id.startswith(_PREFIX): + return await _incoming_sip(request, session_id, operation, body, auth, ownership) + handle: Final = decode_session(session_id, _owner(auth)) + await _reauth(ownership, request, body, handle.alias) + async with _precall( + _request(request, MappingProxyType({"model": handle.alias})), + ownership.auth, + handle.alias, + attachment=request, + parallel_reserved=handle.parallel_reserved, + ownership=ownership, + ): + response: Final = await LiveTransport(_pinned(handle), request.headers).request( + request.method, live_session_path(handle.session_id, operation), body=body if body else None + ) + return _response(response) + + +async def _incoming_sip( + request: Request, + session_id: str, + operation: LiveOperation, + body: Mapping[str, JsonValue], + auth: UserAPIKeyAuth, + ownership: _BudgetOwnership, +) -> Response: + from litellm.proxy import proxy_server as server + + if auth.user_role != LitellmUserRoles.PROXY_ADMIN or operation not in ("accept", "reject"): + raise HTTPException(403, "Incoming SIP enrollment requires a proxy administrator") + alias: Final = request.headers.get("x-litellm-live-model") + if not alias: + raise HTTPException(400, "Incoming SIP requires x-litellm-live-model identifying one deployment") + configured: Final = tuple( + item + for item in TypeAdapter(tuple[Mapping[str, object], ...]).validate_python( + getattr(server, "llm_model_list", ()) or () + ) + if item.get("model_name") == alias + ) + if len(configured) != 1: + raise HTTPException(400, "Incoming SIP requires a model alias with exactly one deployment") + live_session_path(session_id, "attach") + if operation == "accept" and _session_model(body) != alias: + raise HTTPException(400, "session.model must match x-litellm-live-model") + await _reauth(ownership, request, body, alias) + async with _precall( + _request(request, MappingProxyType({**body, "model": alias})), ownership.auth, alias, ownership=ownership + ) as prepared: + deployment: Final = await _deployment(alias, prepared.processed) + response: Final = await LiveTransport(deployment, request.headers).request( + "POST", + live_session_path(session_id, operation), + body=_provider_body(_processed_body(body, prepared.processed), deployment.model) + if operation == "accept" + else body, + ) + if response.is_error or operation == "reject": + return _response(response) + handle: Final = _new_handle( + session_id, + alias, + deployment, + ownership.auth, + prepared.lease, + policy=_session_policy(_processed_body(body, prepared.processed), None), + ) + await _supervise(request, handle, ownership.auth, prepared.logger, prepared.lease) + prepared.transfer() + return Response( + response.content, + status_code=response.status_code, + headers=MappingProxyType({"x-litellm-live-session-id": encode_session(handle)}), + ) + + +class _PublicSocket: + def __init__( + self, + websocket: WebSocket, + handle: LiveHandle, + public_id: str, + auth: UserAPIKeyAuth, + observer: RealTimeStreaming | None = None, + ) -> None: + self.websocket = websocket + self.handle = handle + self.public_id = public_id + self.auth = auth + self.observer = observer + self.scope = websocket.scope + self.headers = websocket.headers + + async def send_text(self, data: str) -> None: + if self.observer is not None: + self.observer.store_message(data) # pyright: ignore[reportUnknownMemberType] # stream also accepts legacy dict events + await self.websocket.send_text( + _encode_json(rewrite_session_ids(_OBJECT.validate_json(data), self.handle.session_id, self.public_id)) + ) + + async def receive_text(self) -> str: + data: Final = await self.websocket.receive_text() + payload: Final = _OBJECT.validate_json(data) + if payload.get("type") == "session.start": + raise HTTPException(400, "Session has already started") + session: Final = payload.get("session") + if isinstance(session, Mapping) and "model" in session: + raise HTTPException(400, "Session model cannot change") + await _authorize_delegation(payload, self.auth) + return _encode_json(rewrite_session_ids(payload, self.public_id, self.handle.session_id)) + + async def close(self, code: int = 1000, reason: str | None = None) -> None: + await self.websocket.close(code=code, reason=reason) + + +class _StartupEvents: + def __init__(self) -> None: + self.messages: tuple[str, ...] = () + self.size = 0 + + def store(self, event: Mapping[str, JsonValue]) -> None: + message: Final = _encode_json(event) + if len(self.messages) >= 128 or self.size + len(message) > 8 * 1024 * 1024: + raise HTTPException(502, "Upstream exceeded the Live startup event limit") + self.messages = (*self.messages, message) + self.size += len(message) + + +async def _wait_started( + connection: "ClientConnection", websocket: WebSocket, startup: _StartupEvents | None = None +) -> Mapping[str, JsonValue]: + async def receive_started() -> Mapping[str, JsonValue]: + while True: + event: Mapping[str, JsonValue] = _OBJECT.validate_json( + await connection.recv() + ) # rebind-ok: each received event has a new value + if event.get("type") == "session.started": + return event + if startup is not None: + startup.store(event) + await websocket.send_json(event) + if event.get("type") in ("error", "session.closed"): + raise HTTPException(502, "Upstream did not start the session") + + return await asyncio.wait_for(receive_started(), 20) + + +@_routes.websocket("/sessions") +@_routes.websocket("/sessions/{session_id}/attach") +@_routes.websocket("/sessions/{session_id}/fork") +async def websocket_live_session(websocket: WebSocket, session_id: str | None = None) -> None: + state: Final = _ConnectionState() + try: + api_key: Final = get_websocket_api_key(websocket) + if not api_key: + raise HTTPException(403, "API key required") + auth_request: Final = _request(websocket, _EMPTY) + inbound_headers: Final = TypeAdapter(tuple[tuple[bytes, bytes], ...]).validate_python( + websocket.scope["headers"] + ) + auth_request.scope["headers"] = ( + *(item for item in inbound_headers if item[0].lower() != b"authorization"), + (b"authorization", f"Bearer {api_key}".encode()), + ) + auth: Final = await _auth(auth_request) + async with _budget_scope(auth) as ownership: + source: Final = decode_session(session_id, _owner(auth)) if session_id else None + attached: Final = source is not None and websocket.url.path.endswith("/attach") + await websocket.accept() + first: Final = ( + _EMPTY if attached else _OBJECT.validate_json(await asyncio.wait_for(websocket.receive_text(), 20)) + ) + if not attached and first.get("type") != "session.start": + raise HTTPException(400, "First message must be session.start") + model: Final = ( + source.alias + if attached and source is not None + else _session_model(first, source.alias if source else None) + ) + await _reauth(ownership, auth_request, first, model) + async with _precall( + _request(websocket, MappingProxyType({**first, "model": model})), + ownership.auth, + model, + attachment=websocket if attached else None, + parallel_reserved=source.parallel_reserved if attached and source is not None else False, + ownership=ownership, + ) as prepared: + if attached: + await _authorize_delegation( + _policy_body(_processed_body(first, prepared.processed), source), ownership.auth + ) + else: + await _authorize_fork_policy(_processed_body(first, prepared.processed), source, ownership.auth) + deployment: Final = _pinned(source) if source else await _deployment(model, prepared.processed) + path: Final = ( + live_session_path(source.session_id, "attach" if attached else "fork") + if source + else "live/sessions" + ) + state.connection = await LiveTransport(deployment, websocket.headers).connect(path) + + async def start_session() -> tuple[ + LiveHandle, RealTimeStreaming | None, Mapping[str, JsonValue] | None + ]: + if attached and source is not None: + return source, None, None + if state.connection is None: + raise RuntimeError("Live connection was not established") + await state.connection.send( + _encode_json( + _processed_body(first, prepared.processed) + if source + else _provider_body(_processed_body(first, prepared.processed), deployment.model) + ) + ) + startup: Final = _StartupEvents() + initial: Final = await _wait_started(state.connection, websocket, startup) + handle: Final = _new_handle( + _session_id(initial), + model, + deployment, + ownership.auth, + prepared.lease, + policy=_session_policy(_processed_body(first, prepared.processed), source), + ) + observer: Final = await _supervise( + _request(websocket, MappingProxyType({"model": model})), + handle, + ownership.auth, + prepared.logger, + prepared.lease, + ) + for buffered in startup.messages: + observer.store_message(buffered) # pyright: ignore[reportUnknownMemberType] # stream also accepts legacy dict events + prepared.transfer() + return handle, observer, initial + + handle, observer, initial = await start_session() + public_id: Final = session_id if attached and session_id is not None else encode_session(handle) + if initial is not None: + await websocket.send_json(rewrite_session_ids(initial, handle.session_id, public_id)) + frontend: Final = _PublicSocket(websocket, handle, public_id, ownership.auth, observer) + stream: Final = RealTimeStreaming( + frontend, + state.connection, + prepared.logger, + model=deployment.model, + user_api_key_dict=auth, + request_data=_mutable(prepared.processed), + account_usage=False, + ) + await stream.bidirectional_forward() + except (HTTPException, ValueError, WebSocketDisconnect, asyncio.TimeoutError): + try: + await websocket.close(code=1008, reason="Live session rejected") + except RuntimeError: + pass + except Exception: + try: + await websocket.close(code=1011, reason="Live upstream connection failed") + except RuntimeError: + pass + finally: + if state.connection is not None: + await state.connection.close() + + +router: Final = APIRouter() +for _prefix in ("/v1/live", "/live", "/openai/v1/live"): + router.include_router(_routes, prefix=_prefix) diff --git a/litellm/types/llms/openai.py b/litellm/types/llms/openai.py index f448244706c..c5e9fe360f0 100644 --- a/litellm/types/llms/openai.py +++ b/litellm/types/llms/openai.py @@ -2037,10 +2037,15 @@ class OpenAIRealtimeStreamResponseBaseObject(TypedDict): class OpenAIRealtimeSessionClosed(TypedDict): - type: ReadOnly[Literal["session.closed"]] + type: ReadOnly[Literal["session.closed", "session.usage.updated", "litellm.live.initialization"]] usage: ReadOnly[Mapping[str, object]] +class OpenAILiveResponseEvent(TypedDict): + type: ReadOnly[Literal["response.event"]] + event: ReadOnly[Mapping[str, object]] + + class OpenAIRealtimeConversationObject(TypedDict, total=False): id: str object: Required[Literal["realtime.conversation"]] @@ -2295,6 +2300,7 @@ class OpenAIRealtimeEventTypes(Enum): OpenAIRealtimeEvents = ( OpenAIRealtimeStreamResponseBaseObject | OpenAIRealtimeSessionClosed + | OpenAILiveResponseEvent | OpenAIRealtimeStreamSessionEvents | OpenAIRealtimeStreamResponseOutputItemAdded | OpenAIRealtimeResponseContentPartAdded diff --git a/litellm/types/realtime.py b/litellm/types/realtime.py index 4c3df385ab9..b0655f36fad 100644 --- a/litellm/types/realtime.py +++ b/litellm/types/realtime.py @@ -1,6 +1,6 @@ from typing import Any, Literal -from pydantic import BaseModel +from pydantic import AliasChoices, BaseModel, Field from typing_extensions import ReadOnly, TypedDict from .llms.openai import ( @@ -12,6 +12,16 @@ from .llms.openai import ( ALL_DELTA_TYPES = Literal["text", "audio"] +class LiveSessionDurationUsage(BaseModel): + duration: float = Field( + strict=True, ge=0, allow_inf_nan=False, validation_alias=AliasChoices("seconds", "audio_duration_ms") + ) + + +class LiveSessionUsageEvent(BaseModel): + usage: LiveSessionDurationUsage + + class RealtimeResponseTransformInput(TypedDict): session_configuration_request: str | None current_output_item_id: ( diff --git a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py index 7c514db3e38..b3a41d6cf81 100644 --- a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py +++ b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py @@ -3590,3 +3590,43 @@ async def test_provider_bytes_are_sent_raw_after_pacing(): assert [call.args[0] for call in backend_ws.send.await_args_list] == [b"\x00\x01", '{"type":"endStream"}'] provider_config.pace_backend_send.assert_awaited_once_with(b"\x00\x01") + + +def test_public_live_accounting_survives_filtered_logging(monkeypatch): + monkeypatch.setattr(litellm, "logged_real_time_event_types", []) + stream = RealTimeStreaming(MagicMock(), MagicMock(), MagicMock()) + events = [ + {"type": "session.usage.updated", "usage": {"seconds": 15}}, + { + "type": "response.event", + "event": { + "type": "response.completed", + "response": { + "id": "resp_one", + "model": "gpt-backend", + "usage": {"total_tokens": 12}, + }, + }, + }, + {"type": "session.closed", "usage": {"seconds": 30}}, + ] + for event in events: + stream.store_message({**event, "private_transcript": "do not retain"}) + stream.store_message( + {"type": "response.event", "event": {"type": "response.output_text.delta", "delta": "private"}} + ) + assert stream.messages == events + + +@pytest.mark.parametrize("account_usage,expected", [(True, 1), (False, 0)]) +def test_live_initialization_is_retained_only_by_accounting_owner(account_usage, expected): + stream = RealTimeStreaming( + MagicMock(), + MagicMock(), + MagicMock(), + account_usage=account_usage, + live_initialization_seconds=15, + ) + assert len(stream.messages) == expected + if account_usage: + assert stream.messages == [{"type": "litellm.live.initialization", "usage": {"seconds": 15}}] diff --git a/tests/test_litellm/llms/chatgpt/test_live.py b/tests/test_litellm/llms/chatgpt/test_live.py new file mode 100644 index 00000000000..2e1a35473ef --- /dev/null +++ b/tests/test_litellm/llms/chatgpt/test_live.py @@ -0,0 +1,194 @@ +import json + +import httpx +import pytest + +from litellm.llms.chatgpt.live import LiveDeployment, LiveTransport, live_session_path + + +@pytest.mark.asyncio +@pytest.mark.parametrize("provider", ["chatgpt", "openai"]) +@pytest.mark.parametrize("status", [201, 403, 429, 503]) +async def test_live_request_preserves_payload_status_and_selected_credentials(provider, status, chatgpt_tokens): + payload = { + "session": {"model": "deployment-model", "tools": [{"type": "function", "name": "lookup"}]}, + "transport": {"type": "webrtc", "sdp": "v=0\r\n"}, + "future_option": {"nested": [True, None, 3]}, + } + + def respond(request): + assert request.url.path == "/custom/v1/live/sessions" + assert request.headers["authorization"] == ( + "Bearer test-token-default" if provider == "chatgpt" else "Bearer deployment-key" + ) + assert request.headers.get("chatgpt-account-id") == ("test-account-default" if provider == "chatgpt" else None) + assert request.headers["x-gateway"] == "configured" + assert request.headers["openai-beta"] == "feature=v1" + assert "cookie" not in request.headers + assert json.loads(request.content) == payload + assert request.url.params.get_list("tag") == ["a +/&", "b"] + assert request.url.params["gateway"] == "trusted" + assert request.url.params["cursor"] == "opaque +/&" + assert not {"model", "call_id", "session_id", "api_key"}.intersection(request.url.params) + return httpx.Response(status, json={"result": "upstream"}, headers={"x-request-id": "provider-id"}) + + async with httpx.AsyncClient(transport=httpx.MockTransport(respond)) as client: + transport = LiveTransport( + LiveDeployment( + model="deployment-model", + provider=provider, + api_key="deployment-key", + api_base="https://gateway.example/custom/v1/?gateway=base", + extra_headers={"x-gateway": "configured", "Authorization": "bad", "ChatGPT-Account-Id": "bad"}, + extra_query={"gateway": "trusted", "tag": ("a +/&", "b"), "model": "bad", "session_id": "bad"}, + ), + {"Authorization": "Bearer proxy-key", "Cookie": "private", "OpenAI-Beta": "feature=v1"}, + http_client=client, + ) + response = await transport.request( + "POST", + "live/sessions", + payload, + {"gateway": "untrusted", "cursor": "opaque +/&", "call_id": "bad", "api_key": "bad"}, + ) + assert response.status_code == status + assert response.json() == {"result": "upstream"} + assert response.headers["x-request-id"] == "provider-id" + assert not client.is_closed + + +@pytest.mark.asyncio +@pytest.mark.parametrize("operation", ["fork", "accept", "reject", "refer", "hangup", "content"]) +async def test_live_all_http_operations(operation): + def respond(request): + assert request.url.path == f"/v1/live/sessions/sess_new-ID/{operation}" + assert request.method == ("GET" if operation == "content" else "POST") + assert request.url.params["output_format"] == "json" + return httpx.Response(204) + + async with httpx.AsyncClient(transport=httpx.MockTransport(respond)) as client: + transport = LiveTransport(LiveDeployment("model", provider="openai", api_key="key"), {}, http_client=client) + response = await transport.request( + "GET" if operation == "content" else "POST", + live_session_path("sess_new-ID", operation), + query={"output_format": "json"}, + ) + assert response.status_code == 204 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("path", ["live/sessions", "live/sessions/sess_1/attach", "live/sessions/sess_1/fork"]) +async def test_live_websocket_paths_bounds_and_auth(path, chatgpt_tokens): + from websockets.asyncio.server import serve + + async def observe(connection): + assert connection.request.headers["Authorization"] == "Bearer test-token-default" + await connection.send(connection.request.path) + + async with serve(observe, "127.0.0.1", 0) as server: + port = server.sockets[0].getsockname()[1] + transport = LiveTransport( + LiveDeployment("model", api_base=f"http://127.0.0.1:{port}/v1", extra_query={"route": "a+&b"}), + {"Authorization": "Bearer proxy-key"}, + ) + connection = await transport.connect(path, {"checkpoint": "opaque+value"}) + try: + received = await connection.recv() + url = httpx.URL(f"http://127.0.0.1{received}") + assert url.path == f"/v1/{path}" + assert url.params["route"] == "a+&b" + assert url.params["checkpoint"] == "opaque+value" + finally: + await connection.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + [ + "https://evil.example/live/sessions", + "//evil.example/live/sessions", + "live/sessions/../accept", + "live/sessions/sess%2Fbad/accept", + "live/sessions/sess%5Cbad/accept", + "live/sessions/sess%252Fbad/accept", + "live/sessions/%252e%252e/accept", + "live/sessions/%2E%2E/accept", + "live/sessions/sess%00bad/accept", + "live/sessions/sess%0Abad/accept", + "live/sessions/sess.foo?query/accept", + "live/sessions/sess_1/accept?url=https://evil.example", + "live/sessions/sess_1/accept#fragment", + "live/sessions/sess_1/accept\n", + ], +) +async def test_live_rejects_noncanonical_paths_before_network(path): + transport = LiveTransport(LiveDeployment("model", provider="openai", api_key="key"), {}) + with pytest.raises(ValueError, match=r"(?:Invalid|Noncanonical) Live"): + await transport.request("POST", path) + with pytest.raises(ValueError, match=r"(?:Invalid|Noncanonical) Live"): + await transport.connect(path) + + +@pytest.mark.parametrize( + "session_id", + ["../x", "sess/x", "sess%2Fx", "sess\\x", "sess%255cx", ".", "..", "%252e%252e", "", "sess\n", "sess\x00"], +) +def test_live_session_ids_cannot_inject_path_or_query(session_id): + with pytest.raises(ValueError, match="Invalid Live session ID"): + live_session_path(session_id, "content") + + +@pytest.mark.asyncio +@pytest.mark.parametrize("session_id", ["sess.foo", "sess-\u00f1\u4e2d", "sess?x#y", "sess 50%", "x" * 1024]) +async def test_live_preserves_opaque_session_ids(session_id): + from urllib.parse import quote + + def respond(request): + assert request.url.raw_path == f"/v1/live/sessions/{quote(session_id, safe='')}/content".encode() + assert request.url.params == httpx.QueryParams() + return httpx.Response(200, json={"session_id": session_id}) + + async with httpx.AsyncClient(transport=httpx.MockTransport(respond)) as client: + transport = LiveTransport(LiveDeployment("model", provider="openai", api_key="key"), {}, http_client=client) + response = await transport.request("GET", live_session_path(session_id, "content")) + assert response.json()["session_id"] == session_id + + +@pytest.mark.asyncio +async def test_live_does_not_redirect_credentials(): + def respond(request): + assert request.url.host == "api.openai.com" + return httpx.Response(307, headers={"location": "https://elsewhere.example/collect"}) + + async with httpx.AsyncClient(transport=httpx.MockTransport(respond), follow_redirects=True) as client: + transport = LiveTransport(LiveDeployment("model", provider="openai", api_key="key"), {}, http_client=client) + response = await transport.request("POST", "live/sessions", {}) + assert response.status_code == 307 + + +@pytest.mark.asyncio +async def test_live_websocket_does_not_redirect_credentials(): + from websockets.asyncio.server import serve + from websockets.datastructures import Headers + from websockets.exceptions import InvalidStatus + from websockets.http11 import Response + + async def unused(connection): + pytest.fail("Redirected WebSocket must never open") + + def redirect(connection, request): + assert request.headers["authorization"] == "Bearer deployment-key" + return Response(307, "Temporary Redirect", Headers({"Location": "/elsewhere"})) + + async with serve(unused, "127.0.0.1", 0, process_request=redirect) as server: + port = server.sockets[0].getsockname()[1] + transport = LiveTransport( + LiveDeployment( + "model", provider="openai", api_key="deployment-key", api_base=f"http://127.0.0.1:{port}/v1" + ), + {}, + ) + with pytest.raises(InvalidStatus) as failure: + await transport.connect("live/sessions") + assert failure.value.response.status_code == 307 diff --git a/tests/test_litellm/proxy/auth/test_auth_utils.py b/tests/test_litellm/proxy/auth/test_auth_utils.py index 8f41fc22d71..fc45ca7c699 100644 --- a/tests/test_litellm/proxy/auth/test_auth_utils.py +++ b/tests/test_litellm/proxy/auth/test_auth_utils.py @@ -572,7 +572,11 @@ def _azure_relay_router(): model_list=[ { "model_name": "gpt", - "litellm_params": {"model": "azure_ai/gpt-5.4-mini", "api_base": "https://a.services.ai.azure.com", "api_key": "k"}, + "litellm_params": { + "model": "azure_ai/gpt-5.4-mini", + "api_base": "https://a.services.ai.azure.com", + "api_key": "k", + }, }, { "model_name": "other-group", @@ -966,11 +970,24 @@ def test_get_model_from_request_extracts_realtime_session_model(route, encoded): @pytest.mark.parametrize("session", ['{"model":"actual-voice"}', {"model": "actual-voice"}]) -def test_realtime_calls_auth_uses_executed_session_model_despite_decoys(session): +@pytest.mark.parametrize( + "route", + [ + "/v1/realtime/calls", + "/v1/live", + "/live", + "/openai/v1/live", + "/v1/live/sessions", + "/live/sessions", + "/openai/v1/live/sessions", + "/v1/live/sessions/incoming/accept", + ], +) +def test_realtime_calls_auth_uses_executed_session_model_despite_decoys(session, route): assert ( get_model_from_request( request_data={"model": "body-decoy", "session": session}, - route="/v1/realtime/calls", + route=route, request_query_params={"model": "query-decoy"}, request_headers={"x-litellm-model": "header-decoy"}, ) diff --git a/tests/test_litellm/proxy/realtime_endpoints/test_call_sessions.py b/tests/test_litellm/proxy/realtime_endpoints/test_call_sessions.py index dffd4500ed7..cf3be1e40b8 100644 --- a/tests/test_litellm/proxy/realtime_endpoints/test_call_sessions.py +++ b/tests/test_litellm/proxy/realtime_endpoints/test_call_sessions.py @@ -178,7 +178,9 @@ async def test_mixed_case_offer_preserves_boundary_metadata_and_closes_extra_fil @pytest.mark.parametrize("policy", ["budget", "personal_models"]) @pytest.mark.parametrize("mixed_case", [False, True]) @pytest.mark.parametrize("pre_read", [False, True]) -async def test_offer_auth_enforces_session_model_policy_before_upstream(monkeypatch, multipart, policy, mixed_case, pre_read): +async def test_offer_auth_enforces_session_model_policy_before_upstream( + monkeypatch, multipart, policy, mixed_case, pre_read +): import json from unittest.mock import AsyncMock, MagicMock @@ -557,8 +559,9 @@ async def test_realtime_endpoint_rejects_untrusted_call_ids(monkeypatch, call_id @pytest.mark.parametrize("multipart", [False, True]) @pytest.mark.parametrize("credential", ["authorization", "api-key", "subprotocol", "x-litellm-api-key", "custom"]) @pytest.mark.parametrize("signaling_credential", ["authorization", "api-key", "x-litellm-api-key", "mixed"]) +@pytest.mark.parametrize("signaling_path", ["/v1/realtime/calls", "/v1/live", "/live", "/openai/v1/live"]) async def test_offer_exchange_wraps_call_and_filters_client_headers( - monkeypatch, multipart, credential, signaling_credential + monkeypatch, multipart, credential, signaling_credential, signaling_path ): import json from unittest.mock import AsyncMock @@ -596,7 +599,7 @@ async def test_offer_exchange_wraps_call_and_filters_client_headers( { "type": "http", "method": "POST", - "path": "/v1/realtime/calls", + "path": signaling_path, "scheme": "http", "server": ("localhost", 80), "query_string": b"intent=quicksilver&architecture=avas&untrusted=bad", @@ -672,6 +675,8 @@ async def test_offer_exchange_wraps_call_and_filters_client_headers( monkeypatch.setattr(codex, "supervise_codex_call", supervise) response = await codex.create_codex_realtime_call(request) assert response.status_code == 201 + expected_prefix = "/v1/realtime/calls/" if signaling_path == "/v1/realtime/calls" else "/v1/live/" + assert response.headers["location"].startswith(expected_prefix) assert response.body == b"v=0\r\nanswer" token = response.headers["location"].rsplit("/", 1)[-1] call = codex.decode_call(token, "Bearer owner") @@ -1147,7 +1152,9 @@ async def test_signaling_rejection_after_admission_refunds_parallel_slot(monkeyp class Reject(CustomLogger): async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type): - current = await proxy.internal_usage_cache.async_get_cache(key, litellm_parent_otel_span=None, local_only=True) + current = await proxy.internal_usage_cache.async_get_cache( + key, litellm_parent_otel_span=None, local_only=True + ) assert limiter._gauge_in_flight_from_cache_value(current) == 1 raise RuntimeError("Policy rejected after admission") diff --git a/tests/test_litellm/proxy/realtime_endpoints/test_call_supervision.py b/tests/test_litellm/proxy/realtime_endpoints/test_call_supervision.py index 6603b4a0cdb..4455ef25cd7 100644 --- a/tests/test_litellm/proxy/realtime_endpoints/test_call_supervision.py +++ b/tests/test_litellm/proxy/realtime_endpoints/test_call_supervision.py @@ -187,7 +187,8 @@ class Sink: @pytest.mark.parametrize( "duration,valid", [(0, True), (1000, True), (None, False), (-1, False), (True, False), ("1000", False)] ) -async def test_live_terminal_requires_valid_duration_for_accounting(monkeypatch, duration, valid): +@pytest.mark.parametrize("duration_field", ["audio_duration_ms", "seconds"]) +async def test_live_terminal_requires_valid_duration_for_accounting(monkeypatch, duration, valid, duration_field): from litellm.proxy.realtime_endpoints import call_supervision socket = Socket() @@ -201,7 +202,7 @@ async def test_live_terminal_requires_valid_duration_for_accounting(monkeypatch, await socket.messages.put({"type": "session.started"}) await supervisor.start() await socket.messages.put( - {"type": "session.closed", **({"usage": {"audio_duration_ms": duration}} if duration is not None else {})} + {"type": "session.closed", **({"usage": {duration_field: duration}} if duration is not None else {})} ) await supervisor.wait() close.assert_not_awaited() @@ -695,3 +696,58 @@ async def test_ga_observer_disconnect_requires_confirmed_hangup(closure, hangup_ assert bool(logger.model_call_details.get("realtime_usage_incomplete")) == (not hangup_succeeds) assert sink.logs == 1 assert socket.closed + + +@pytest.mark.asyncio +async def test_live_connected_attach_is_ready_without_session_started_and_bills_once(): + socket = Socket() + logger = MagicMock(spec=Logging) + logger.model_call_details = {} + sink = Sink(logger) + + async def close(): + await socket.messages.put({"type": "session.closed", "usage": {"seconds": 30}}) + + supervisor = CallSupervisor( + socket, + sink, + logger, + UserAPIKeyAuth(), + close, + connected_ready=True, + ready_timeout=0.1, + ) + await supervisor.start() + await socket.messages.put({"type": "session.usage.updated", "usage": {"seconds": 15}}) + await supervisor.close() + await supervisor.close() + assert sink.logs == 1 + assert sink.events[-1] == {"type": "session.closed", "usage": {"seconds": 30}} + assert "realtime_usage_incomplete" not in logger.model_call_details + assert socket.closed + + +@pytest.mark.asyncio +async def test_live_missing_backend_accounting_invalidates_budget_after_dispatch(monkeypatch): + from litellm.proxy.realtime_endpoints import call_supervision + + socket = Socket() + logger = MagicMock(spec=Logging) + logger.model_call_details = {} + invalidate = AsyncMock() + monkeypatch.setattr(call_supervision, "invalidate_budget_reservation_counters", invalidate) + + class IncompleteSink(Sink): + async def log_messages(self, *, wait_for_dispatch=False): + await super().log_messages(wait_for_dispatch=wait_for_dispatch) + logger.model_call_details["realtime_backend_accounting_incomplete"] = True + + sink = IncompleteSink(logger) + supervisor = CallSupervisor(socket, sink, logger, UserAPIKeyAuth(), AsyncMock(), connected_ready=True) + await supervisor.start() + await socket.messages.put({"type": "session.closed", "usage": {"seconds": 30}}) + await supervisor.wait() + assert sink.logs == 1 + assert logger.model_call_details["realtime_accounting_incomplete"] is True + assert "realtime_usage_incomplete" not in logger.model_call_details + invalidate.assert_awaited_once() diff --git a/tests/test_litellm/proxy/realtime_endpoints/test_live.py b/tests/test_litellm/proxy/realtime_endpoints/test_live.py new file mode 100644 index 00000000000..a1bc9106f84 --- /dev/null +++ b/tests/test_litellm/proxy/realtime_endpoints/test_live.py @@ -0,0 +1,930 @@ +import json +import time +from contextlib import asynccontextmanager +from types import MappingProxyType, SimpleNamespace +from unittest.mock import AsyncMock, Mock + +import httpx +import pytest +from fastapi import FastAPI, HTTPException, Request +from fastapi.testclient import TestClient + +from litellm.llms.chatgpt.live import LiveDeployment +from litellm.proxy._types import UserAPIKeyAuth +from litellm.proxy.realtime_endpoints import live + + +@pytest.fixture(autouse=True) +def encryption_key(monkeypatch): + monkeypatch.setenv("LITELLM_SALT_KEY", "test-live-encryption-key") + + +def handle(owner="owner"): + return live._new_handle( + "sess_upstream", + "voice", + LiveDeployment(model="gpt-live"), + UserAPIKeyAuth(api_key=owner), + None, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("field", ["chatgpt_auth_profile", "chatgpt_token_dir", "chatgpt_auth_file"]) +async def test_live_rejects_unsupported_deployment_credentials(monkeypatch, field): + from litellm.proxy import proxy_server + + router = SimpleNamespace( + async_get_available_deployment=AsyncMock( + return_value={ + "litellm_params": {"model": "chatgpt/gpt-live-1", field: "other-account"}, + "model_info": {"id": "voice"}, + } + ), + async_routing_strategy_pre_call_checks=AsyncMock(), + ) + monkeypatch.setattr(proxy_server, "llm_router", router) + with pytest.raises(HTTPException) as rejected: + await live._deployment("voice", {}) + assert rejected.value.status_code == 400 + assert "deployment auth overrides are unsupported" in rejected.value.detail + + +def test_session_tokens_hide_credentials_and_enforce_owner_expiry_and_integrity(): + original = handle() + token = live.encode_session(original) + assert "deployment-a" not in token and "sess_upstream" not in token + assert live.decode_session(token, live._owner(UserAPIKeyAuth(api_key="owner"))) == original + for candidate, owner in ( + (token, "other-owner"), + ("sess_upstream", original.owner), + (token[:-8] + "aaaaaaaa", original.owner), + (live.encode_session(original.model_copy(update={"expires_at": time.time() - 1})), original.owner), + ): + with pytest.raises(HTTPException) as rejected: + live.decode_session(candidate, owner) + assert rejected.value.status_code == 403 + + +def test_handle_serializes_mappingproxy_without_losing_pinned_deployment(): + deployment = LiveDeployment( + model="gpt-live", + model_id="deployment-a", + api_base="https://upstream.test/v1", + extra_headers=MappingProxyType({"openai-beta": "test"}), + extra_query=MappingProxyType({"architecture": "test"}), + ) + original = live._new_handle("sess_upstream", "voice", deployment, UserAPIKeyAuth(api_key="owner"), None) + assert live._pinned(live.decode_session(live.encode_session(original), original.owner)) == deployment + + +def test_only_protocol_session_ids_are_rewritten_and_application_values_survive(): + event = { + "type": "session.started", + "session": {"id": "raw", "instructions": "raw"}, + "session_id": "raw", + "delta": "raw", + "event": {"type": "response.output_text.delta", "delta": "raw", "session_id": "raw"}, + } + rewritten = live.rewrite_session_ids(event, "raw", "public") + assert rewritten["session"]["id"] == "public" + assert rewritten["session_id"] == "public" + assert rewritten["session"]["instructions"] == "raw" + assert rewritten["delta"] == "raw" + assert rewritten["event"] == event["event"] + assert event["session"]["id"] == "raw" + + +@pytest.fixture +def route_client(monkeypatch): + auth = UserAPIKeyAuth(api_key="owner") + deployment = LiveDeployment(model="gpt-live", provider="openai", api_key="upstream-key", model_id="deployment-a") + transport = SimpleNamespace( + request=AsyncMock( + return_value=httpx.Response( + 201, json={"session": {"id": "sess_upstream"}, "transport": {"type": "webrtc", "sdp": "answer"}} + ) + ) + ) + selected = AsyncMock(return_value=deployment) + supervised = AsyncMock() + authenticated_bodies = [] + + async def authenticate(request): + authenticated_bodies.append(await request.json()) + return auth + + @asynccontextmanager + async def precall(request, auth, model, **kwargs): + yield live._Prepared({"model": model}, Mock(), None, kwargs.get("ownership")) + + monkeypatch.setattr(live, "_auth", authenticate) + monkeypatch.setattr(live, "_precall", precall) + monkeypatch.setattr(live, "_deployment", selected) + monkeypatch.setattr(live, "_supervise", supervised) + factory = Mock(return_value=transport) + monkeypatch.setattr(live, "LiveTransport", factory) + app = FastAPI() + app.include_router(live.router) + return SimpleNamespace( + client=TestClient(app), + transport=transport, + selected=selected, + supervised=supervised, + auth=auth, + factory=factory, + bodies=authenticated_bodies, + ) + + +@pytest.mark.parametrize("prefix", ["/v1/live", "/live", "/openai/v1/live"]) +def test_create_preserves_configuration_and_returns_owned_json_session(route_client, prefix): + body = { + "session": { + "model": "voice", + "instructions": "hello", + "input": [{"role": "user", "content": "hi"}], + "audio": {"output": {"voice": "marin"}}, + "delegation": {"type": "client"}, + "future_option": {"enabled": True}, + }, + "transport": {"type": "webrtc", "sdp": "offer"}, + "api_base": "https://untrusted.test", + } + result = route_client.client.post(prefix + "/sessions", json=body) + assert result.status_code == 201 + output = result.json() + assert output["transport"] == {"type": "webrtc", "sdp": "answer"} + original = live.decode_session(output["session"]["id"], live._owner(route_client.auth)) + assert original.session_id == "sess_upstream" + assert original.deployment["api_key"] == "upstream-key" + assert original.initialization_seconds == 15 + forwarded = route_client.transport.request.await_args.kwargs["body"] + assert forwarded["session"] == {**body["session"], "model": "gpt-live"} + assert route_client.bodies[0] == {**body, "model": "voice"} + route_client.supervised.assert_awaited_once() + assert route_client.factory.call_args.args[0].api_base is None + + +def test_fork_preserves_empty_overrides_and_pins_source_deployment(route_client): + source = handle() + source = source.model_copy(update={"deployment": {**source.deployment, "model_id": "deployment-a"}}) + token = live.encode_session(source) + body = {"session": {}, "transport": {"type": "webrtc", "sdp": "offer"}} + result = route_client.client.post(f"/v1/live/sessions/{token}/fork", json=body) + assert result.status_code == 201 + assert route_client.transport.request.await_args.args == ("POST", "live/sessions/sess_upstream/fork") + assert route_client.transport.request.await_args.kwargs["body"] == body + assert route_client.factory.call_args.args[0].model_id == "deployment-a" + route_client.selected.assert_not_awaited() + + +def test_fork_cannot_change_model_even_to_same_alias(route_client): + token = live.encode_session(handle()) + response = route_client.client.post(f"/v1/live/sessions/{token}/fork", json={"session": {"model": "voice"}}) + assert response.status_code == 400 + route_client.transport.request.assert_not_awaited() + + +def test_cross_key_followup_and_raw_incoming_ids_never_contact_upstream(route_client): + token = live.encode_session(handle("different-key")) + for path in (f"{token}/hangup", "sess_other/accept", "sess_other/reject"): + response = route_client.client.post(f"/v1/live/sessions/{path}", json={}) + assert response.status_code == 403 + route_client.transport.request.assert_not_awaited() + + +def test_recording_preserves_binary_body_status_and_content_headers(route_client): + token = live.encode_session(handle()) + route_client.transport.request.return_value = httpx.Response( + 206, + content=b"\x00\xffrecording", + headers={ + "content-type": "video/mp4", + "content-disposition": "attachment; filename=recording.mp4", + "content-range": "bytes 0-10/20", + }, + ) + result = route_client.client.get(f"/v1/live/sessions/{token}/content") + assert result.status_code == 206 + assert result.content == b"\x00\xffrecording" + assert result.headers["content-type"] == "video/mp4" + assert result.headers["content-range"] == "bytes 0-10/20" + + +@pytest.mark.asyncio +async def test_pre_call_preserves_safe_body_options_and_refunds_failed_signaling(monkeypatch): + from litellm.proxy import proxy_server + + auth = UserAPIKeyAuth(api_key="owner") + authorize = AsyncMock() + process = AsyncMock(return_value=({"model": "voice"}, Mock())) + release = AsyncMock() + monkeypatch.setattr(live, "_authorize", authorize) + monkeypatch.setattr(live, "process_codex_request", process) + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", release) + monkeypatch.setattr(proxy_server, "proxy_logging_obj", SimpleNamespace(get_proxy_hook=lambda _: None)) + request = Request({"type": "http", "headers": []}) + synthetic = live._request( + request, + { + "session": {"model": "voice", "instructions": "safe"}, + "api_base": "https://untrusted.test", + "extra_headers": {"x-admin": "true"}, + }, + ) + with pytest.raises(RuntimeError, match="signaling failed"): + async with live._precall(synthetic, auth, "voice"): + raise RuntimeError("signaling failed") + authorize.assert_awaited_once_with("voice", auth) + data = process.await_args.args[1] + assert data["session"]["instructions"] == "safe" + assert "api_base" not in data and "extra_headers" not in data + release.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_transferred_supervisor_retains_budget_on_client_disconnect(monkeypatch): + from litellm.proxy import proxy_server + + monkeypatch.setattr(live, "_authorize", AsyncMock()) + monkeypatch.setattr(live, "process_codex_request", AsyncMock(return_value=({"model": "voice"}, Mock()))) + release = AsyncMock() + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", release) + monkeypatch.setattr(proxy_server, "proxy_logging_obj", SimpleNamespace(get_proxy_hook=lambda _: None)) + request = live._request(Request({"type": "http", "headers": []}), {"model": "voice"}) + + async def disconnect_after_transfer(): + async with live._precall(request, UserAPIKeyAuth(api_key="owner"), "voice") as prepared: + prepared.transferred = True + raise RuntimeError("client disconnected") + + with pytest.raises(RuntimeError, match="client disconnected"): + await disconnect_after_transfer() + release.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_info_before_started_is_forwarded_without_rejecting_session(): + info = {"type": "info", "code": "data_channel_permissions", "message": "ready"} + started = {"type": "session.started", "session": {"id": "sess_upstream"}} + socket = SimpleNamespace(recv=AsyncMock(side_effect=[json.dumps(info), json.dumps(started)])) + client = SimpleNamespace(send_json=AsyncMock()) + assert await live._wait_started(socket, client) == started + client.send_json.assert_awaited_once_with(info) + + +@pytest.mark.asyncio +async def test_upstream_startup_error_is_forwarded_without_creating_session(): + error = {"type": "error", "error": {"code": "forbidden", "message": "Voice access denied"}} + socket = SimpleNamespace(recv=AsyncMock(return_value=json.dumps(error))) + client = SimpleNamespace(send_json=AsyncMock()) + with pytest.raises(HTTPException) as rejected: + await live._wait_started(socket, client) + assert rejected.value.status_code == 502 + client.send_json.assert_awaited_once_with(error) + + +@pytest.mark.asyncio +async def test_session_events_keep_stable_public_id_and_feed_shared_usage_sink(): + original = handle() + token = live.encode_session(original) + client = SimpleNamespace(send_text=AsyncMock(), scope={}, headers={}) + observer = SimpleNamespace(store_message=Mock()) + frontend = live._PublicSocket(client, original, token, UserAPIKeyAuth(api_key="owner"), observer) + event = { + "type": "session.updated", + "session": {"id": original.session_id}, + "event": {"type": "response.completed", "response": {"id": "resp_a"}}, + } + for _ in range(2): + await frontend.send_text(json.dumps(event)) + assert json.loads(client.send_text.await_args.args[0])["session"]["id"] == token + assert observer.store_message.call_count == 2 + + +@pytest.mark.asyncio +async def test_websocket_delegation_model_update_is_authorized_before_forwarding(monkeypatch): + authorize = AsyncMock(side_effect=HTTPException(403, "Model forbidden")) + monkeypatch.setattr(live, "_authorize", authorize) + message = {"type": "session.update", "session": {"delegation": {"responses": {"model": "unauthorized"}}}} + client = SimpleNamespace(receive_text=AsyncMock(return_value=json.dumps(message)), scope={}, headers={}) + auth = UserAPIKeyAuth(api_key="owner") + frontend = live._PublicSocket(client, handle(), "public", auth) + with pytest.raises(HTTPException) as rejected: + await frontend.receive_text() + assert rejected.value.status_code == 403 + authorize.assert_awaited_once_with("unauthorized", auth) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "limits", + [ + {"rpm_limit": 1}, + {"model_max_budget": {"backend": 1}}, + {"team_tpm_limit": 10}, + {"team_metadata": {"model_rpm_limit": {"backend": 1}}}, + ], +) +async def test_managed_delegation_fails_closed_for_unenforceable_constraints(limits): + body = {"session": {"delegation": {"type": "responses", "responses": {"model": "backend"}}}} + with pytest.raises(HTTPException) as rejected: + await live._authorize_delegation(body, UserAPIKeyAuth(api_key="owner", **limits)) + assert rejected.value.status_code == 400 + assert "client delegation" in rejected.value.detail + + +@pytest.mark.asyncio +async def test_restricted_webrtc_cannot_change_managed_model_outside_proxy(monkeypatch): + monkeypatch.setattr(live, "_authorize", AsyncMock()) + body = { + "session": {"delegation": {"type": "responses", "responses": {"model": "backend"}}}, + "transport": {"type": "webrtc", "sdp": "offer"}, + } + auth = UserAPIKeyAuth(api_key="owner", models=["voice", "backend"]) + with pytest.raises(HTTPException) as rejected: + await live._authorize_delegation(body, auth) + assert rejected.value.status_code == 400 + body["session"]["client"] = {"data_channel": {"allowed_client_events": ["session.close"]}} + await live._authorize_delegation(body, auth) + + +@pytest.mark.asyncio +async def test_budget_scope_releases_on_ownership_decode_error(monkeypatch): + release = AsyncMock() + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", release) + auth = UserAPIKeyAuth(api_key="owner") + with pytest.raises(HTTPException): + async with live._budget_scope(auth): + live.decode_session("raw-session-id", live._owner(auth)) + release.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_model_authorization_rejection_still_releases_budget(monkeypatch): + from litellm.proxy import proxy_server + + monkeypatch.setattr(live, "_authorize", AsyncMock(side_effect=HTTPException(403, "Model forbidden"))) + release = AsyncMock() + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", release) + monkeypatch.setattr(proxy_server, "proxy_logging_obj", SimpleNamespace(get_proxy_hook=lambda _: None)) + request = live._request(Request({"type": "http", "headers": []}), {"model": "forbidden"}) + with pytest.raises(HTTPException): + async with live._precall(request, UserAPIKeyAuth(api_key="owner"), "forbidden"): + pytest.fail("Upstream must not be reached") + release.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_precall_guardrail_mutations_are_used_without_forwarding_routing_options(monkeypatch): + from litellm.proxy import proxy_server + + monkeypatch.setattr(live, "_authorize", AsyncMock()) + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", AsyncMock()) + process = AsyncMock( + return_value=( + { + "model": "voice", + "session": {"model": "voice", "instructions": "redacted"}, + "api_base": "https://internal.test", + }, + Mock(), + ) + ) + monkeypatch.setattr(live, "process_codex_request", process) + monkeypatch.setattr(proxy_server, "proxy_logging_obj", SimpleNamespace(get_proxy_hook=lambda _: None)) + body = {"type": "session.start", "session": {"model": "voice", "instructions": "sensitive"}} + request = live._request(Request({"type": "http", "headers": []}), body) + async with live._precall(request, UserAPIKeyAuth(api_key="owner"), "voice") as prepared: + forwarded = live._processed_body(body, prepared.processed) + assert forwarded["session"]["instructions"] == "redacted" + assert forwarded["type"] == "session.start" + assert "api_base" not in forwarded + assert process.await_args.args[1]["session"]["instructions"] == "sensitive" + + +@pytest.mark.asyncio +async def test_failed_live_signaling_releases_real_parallel_limiter_and_can_retry(monkeypatch): + import litellm + from litellm.proxy import proxy_server as server + from litellm.proxy.common_utils.user_api_key_cache import UserApiKeyCache + from litellm.proxy.utils import ProxyLogging + + proxy = ProxyLogging(UserApiKeyCache()) + monkeypatch.setattr(litellm, "callbacks", []) + proxy._add_proxy_hooks() + model_list = [{"model_name": "voice", "litellm_params": {"model": "openai/gpt-live-1", "api_key": "upstream-key"}}] + monkeypatch.setattr(server, "proxy_logging_obj", proxy) + monkeypatch.setattr(server, "general_settings", {}) + monkeypatch.setattr(server, "llm_model_list", model_list) + monkeypatch.setattr(server, "llm_router", litellm.Router(model_list=model_list)) + auth = UserAPIKeyAuth(api_key="live-limiter-owner", max_parallel_requests=1) + authenticate = AsyncMock(return_value=auth) + monkeypatch.setattr(live, "user_api_key_auth", authenticate) + transport = SimpleNamespace( + request=AsyncMock(return_value=httpx.Response(403, json={"error": {"code": "forbidden"}})) + ) + monkeypatch.setattr(live, "LiveTransport", Mock(return_value=transport)) + supervisor = AsyncMock() + monkeypatch.setattr(live, "_supervise", supervisor) + for _ in range(2): + request = live._request( + Request( + { + "type": "http", + "method": "POST", + "path": "/v1/live/sessions", + "query_string": b"", + "headers": [(b"content-type", b"application/json")], + } + ), + {"session": {"model": "voice"}, "transport": {"type": "webrtc", "sdp": "offer"}}, + ) + result = await live.create_live_session(request) + assert result.status_code == 403 + current = await proxy.internal_usage_cache.async_get_cache( + "{api_key:live-limiter-owner}:max_parallel_requests", litellm_parent_otel_span=None, local_only=True + ) + limiter = proxy.get_proxy_hook("parallel_request_limiter") + assert limiter._gauge_in_flight_from_cache_value(current) == 0 + assert transport.request.await_count == 2 + supervisor.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_inherited_managed_fork_cannot_bypass_new_key_constraints(): + source = handle().model_copy( + update={"policy": {"delegation": {"type": "responses", "responses": {"model": "backend"}}}} + ) + payload = live._policy_body({"session": {}, "transport": {"type": "webrtc", "sdp": "offer"}}, source) + assert payload["session"]["delegation"]["responses"]["model"] == "backend" + with pytest.raises(HTTPException) as rejected: + await live._authorize_delegation(payload, UserAPIKeyAuth(api_key="owner", rpm_limit_per_model={"backend": 1})) + assert rejected.value.status_code == 400 + + +@pytest.mark.parametrize("protocol", ["http", "websocket"]) +@pytest.mark.parametrize( + "startup_policy", [{}, {"delegation": {"type": "responses", "responses": {"model": "allowed"}}}] +) +@pytest.mark.parametrize("overrides", [{}, {"delegation": {"responses": {}}}]) +def test_restricted_fork_never_trusts_startup_delegation(route_client, protocol, startup_policy, overrides): + from starlette.websockets import WebSocketDisconnect + + # The source may now use a revoked backend, including after an unrestricted WebRTC update. + route_client.auth.models = ["voice", "allowed"] + token = live.encode_session(handle().model_copy(update={"policy": startup_policy})) + path = f"/v1/live/sessions/{token}/fork" + if protocol == "http": + response = route_client.client.post(path, json={"session": overrides}) + assert response.status_code == 400 + else: + with route_client.client.websocket_connect(path, headers={"Authorization": "Bearer owner"}) as ws: + ws.send_json({"type": "session.start", "session": overrides}) + with pytest.raises(WebSocketDisconnect) as rejected: + ws.receive_json() + assert rejected.value.code == 1008 + route_client.factory.assert_not_called() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("startup_policy", [{}, {"delegation": {"type": "responses", "responses": {"model": "old"}}}]) +async def test_explicit_fork_backend_is_authorized_even_when_startup_policy_differs(monkeypatch, startup_policy): + authorize = AsyncMock() + monkeypatch.setattr(live, "_authorize", authorize) + source = handle().model_copy(update={"policy": startup_policy}) + auth = UserAPIKeyAuth(api_key="owner", models=["voice", "allowed"]) + body = {"session": {"delegation": {"type": "responses", "responses": {"model": "allowed"}}}} + await live._authorize_fork_policy(body, source, auth) + authorize.assert_awaited_once_with("allowed", auth) + authorize.side_effect = HTTPException(403, "Model revoked") + with pytest.raises(HTTPException) as rejected: + await live._authorize_fork_policy(body, source, auth) + assert rejected.value.status_code == 403 + + +def test_restricted_fork_can_explicitly_select_client_delegation(route_client): + route_client.auth.models = ["voice"] + body = {"session": {"delegation": {"type": "client"}}} + token = live.encode_session(handle()) + route_client.transport.request.return_value = httpx.Response( + 200, json={"session": {"id": "sess_fork"}, "transport": {"type": "webrtc", "sdp": "answer"}} + ) + response = route_client.client.post(f"/v1/live/sessions/{token}/fork", json=body) + assert response.status_code == 200 + assert response.json()["transport"]["sdp"] == "answer" + route_client.transport.request.assert_awaited_once_with("POST", "live/sessions/sess_upstream/fork", body=body) + + +@pytest.mark.parametrize("protocol", ["http", "websocket"]) +@pytest.mark.parametrize("limits", [{"rpm_limit": 10}, {"tpm_limit": 100}, {"model_max_budget": {"backend": 1}}]) +def test_fork_with_new_limits_cannot_trust_old_client_policy(route_client, protocol, limits): + from starlette.websockets import WebSocketDisconnect + + # An unrestricted source could have switched to managed delegation after its handle was issued. + for key, value in limits.items(): + setattr(route_client.auth, key, value) + token = live.encode_session(handle()) + path = f"/v1/live/sessions/{token}/fork" + if protocol == "http": + response = route_client.client.post(path, json={"session": {}}) + assert response.status_code == 400 + else: + with route_client.client.websocket_connect(path, headers={"Authorization": "Bearer owner"}) as ws: + ws.send_json({"type": "session.start", "session": {}}) + with pytest.raises(WebSocketDisconnect) as rejected: + ws.receive_json() + assert rejected.value.code == 1008 + route_client.factory.assert_not_called() + + +@pytest.mark.asyncio +async def test_explicit_managed_fork_still_rejected_for_new_rate_limits(): + with pytest.raises(HTTPException) as rejected: + await live._authorize_fork_policy( + {"session": {"delegation": {"type": "responses", "responses": {"model": "backend"}}}}, + handle(), + UserAPIKeyAuth(api_key="owner", rpm_limit=10), + ) + assert rejected.value.status_code == 400 + assert "cannot enforce" in rejected.value.detail + + +@pytest.mark.asyncio +async def test_startup_usage_buffer_preserves_nested_events_until_supervisor_owns_accounting(): + usage = { + "type": "response.event", + "event": { + "type": "response.completed", + "response": { + "id": "resp_before_start", + "model": "backend", + "usage": {"input_tokens": 7, "output_tokens": 2}, + }, + }, + } + started = {"type": "session.started", "session": {"id": "sess_upstream"}} + backend = SimpleNamespace(recv=AsyncMock(side_effect=[json.dumps(usage), json.dumps(started)])) + client = SimpleNamespace(send_json=AsyncMock()) + startup = live._StartupEvents() + assert await live._wait_started(backend, client, startup) == started + assert [json.loads(message) for message in startup.messages] == [usage] + + +def test_primary_websocket_authenticates_model_and_keeps_public_event_shape(route_client): + from websockets.exceptions import ConnectionClosedOK + from websockets.frames import Close + + event = {"type": "future.live.event", "session_id": "sess_upstream", "payload": {"untouched": ["a", 2]}} + backend = SimpleNamespace( + send=AsyncMock(), + close=AsyncMock(), + recv=AsyncMock( + side_effect=[ + json.dumps({"type": "info", "code": "ready", "message": "Preparing"}), + json.dumps({"type": "session.started", "session": {"id": "sess_upstream"}}), + json.dumps(event), + ConnectionClosedOK(Close(1000, ""), Close(1000, ""), True), + ] + ), + ) + route_client.transport.connect = AsyncMock(return_value=backend) + observer = SimpleNamespace(store_message=Mock()) + route_client.supervised.return_value = observer + with route_client.client.websocket_connect("/v1/live/sessions", headers={"Authorization": "Bearer owner"}) as ws: + ws.send_json({"type": "session.start", "session": {"model": "voice", "instructions": "hello", "unknown": True}}) + assert ws.receive_json()["type"] == "info" + started = ws.receive_json() + public_id = started["session"]["id"] + assert live.decode_session(public_id, live._owner(route_client.auth)).session_id == "sess_upstream" + forwarded = ws.receive_json() + assert forwarded == {**event, "session_id": public_id} + assert route_client.bodies[0] == {} + assert route_client.bodies[1]["model"] == "voice" + assert route_client.bodies[1]["session"]["instructions"] == "hello" + initial = json.loads(backend.send.await_args.args[0]) + assert initial == { + "type": "session.start", + "session": {"model": "gpt-live", "instructions": "hello", "unknown": True}, + } + assert any(json.loads(call.args[0]) == event for call in observer.store_message.call_args_list) + + +@pytest.mark.asyncio +async def test_successful_live_session_holds_real_parallel_slot_until_supervisor_releases(monkeypatch): + import litellm + from litellm.proxy import proxy_server as server + from litellm.proxy.common_utils.user_api_key_cache import UserApiKeyCache + from litellm.proxy.utils import ProxyLogging + + proxy = ProxyLogging(UserApiKeyCache()) + monkeypatch.setattr(litellm, "callbacks", []) + proxy._add_proxy_hooks() + models = [{"model_name": "voice", "litellm_params": {"model": "openai/gpt-live-1", "api_key": "upstream-key"}}] + monkeypatch.setattr(server, "proxy_logging_obj", proxy) + monkeypatch.setattr(server, "general_settings", {}) + monkeypatch.setattr(server, "llm_model_list", models) + monkeypatch.setattr(server, "llm_router", litellm.Router(model_list=models)) + auth = UserAPIKeyAuth(api_key="live-held-slot", max_parallel_requests=1) + monkeypatch.setattr(live, "user_api_key_auth", AsyncMock(return_value=auth)) + transport = SimpleNamespace( + request=AsyncMock( + return_value=httpx.Response( + 201, json={"session": {"id": "sess_upstream"}, "transport": {"type": "webrtc", "sdp": "answer"}} + ) + ) + ) + monkeypatch.setattr(live, "LiveTransport", Mock(return_value=transport)) + leases = [] + + async def supervise(request, handle, auth, logger, lease): + leases.append(lease) + + monkeypatch.setattr(live, "_supervise", supervise) + + def request(): + return live._request( + Request( + { + "type": "http", + "method": "POST", + "path": "/v1/live/sessions", + "query_string": b"", + "headers": [(b"content-type", b"application/json")], + } + ), + {"session": {"model": "voice"}, "transport": {"type": "webrtc", "sdp": "offer"}}, + ) + + try: + result = await live.create_live_session(request()) + assert result.status_code == 201 + assert leases[0] is not None + with pytest.raises(HTTPException) as blocked: + await live.create_live_session(request()) + assert blocked.value.status_code == 429 + assert transport.request.await_count == 1 + finally: + for lease in leases: + if lease is not None: + await lease.close() + current = await proxy.internal_usage_cache.async_get_cache( + "{api_key:live-held-slot}:max_parallel_requests", litellm_parent_otel_span=None, local_only=True + ) + limiter = proxy.get_proxy_hook("parallel_request_limiter") + assert limiter._gauge_in_flight_from_cache_value(current) == 0 + + +@pytest.mark.asyncio +async def test_supervisor_logger_keeps_deployment_pricing(monkeypatch): + import litellm + + original = handle().model_copy( + update={"deployment": {**handle().deployment, "model_id": "deployment-priced"}, "initialization_seconds": 15} + ) + logger = Mock(litellm_params={}, model_call_details={}) + monkeypatch.setattr(live, "process_codex_request", AsyncMock(return_value=({"model": "voice"}, logger))) + monkeypatch.setattr(litellm, "get_model_info", Mock(return_value={"input_cost_per_second": 0.1})) + connection = SimpleNamespace(close=AsyncMock()) + transport = SimpleNamespace(connect=AsyncMock(return_value=connection), request=AsyncMock()) + monkeypatch.setattr(live, "LiveTransport", Mock(return_value=transport)) + started = AsyncMock() + monkeypatch.setattr(live, "CALL_SUPERVISORS", SimpleNamespace(start=started)) + request = Request({"type": "http", "method": "POST", "path": "/v1/live/sessions", "headers": []}) + stream = await live._start_supervisor(request, original, UserAPIKeyAuth(api_key="owner"), None) + assert stream.messages[0]["usage"]["seconds"] == 15 + started.assert_awaited_once() + metadata = logger.update_from_kwargs.call_args.kwargs["kwargs"]["litellm_metadata"] + assert metadata["model_info"]["id"] == "deployment-priced" + assert metadata["model_info"]["input_cost_per_second"] == 0.1 + connection.close.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_supervisor_startup_failure_closes_observer_and_invalidates_unconfirmed_hangup(monkeypatch): + from litellm.proxy.spend_tracking import budget_reservation + + logger = Mock(litellm_params={}, model_call_details={}) + monkeypatch.setattr(live, "process_codex_request", AsyncMock(return_value=({"model": "voice"}, logger))) + connection = SimpleNamespace(close=AsyncMock()) + transport = SimpleNamespace( + connect=AsyncMock(return_value=connection), + request=AsyncMock(side_effect=httpx.ConnectError("upstream unavailable")), + ) + monkeypatch.setattr(live, "LiveTransport", Mock(return_value=transport)) + monkeypatch.setattr( + live, "CALL_SUPERVISORS", SimpleNamespace(start=AsyncMock(side_effect=RuntimeError("cannot start"))) + ) + invalidate = AsyncMock() + monkeypatch.setattr(budget_reservation, "invalidate_budget_reservation_counters", invalidate) + request = Request({"type": "http", "method": "POST", "path": "/v1/live/sessions", "headers": []}) + with pytest.raises(RuntimeError, match="cannot start"): + await live._start_supervisor(request, handle(), UserAPIKeyAuth(api_key="owner"), None) + connection.close.assert_awaited_once() + invalidate.assert_awaited_once() + + +def test_admin_sip_accept_requires_exact_deployment_and_returns_owned_handle(route_client, monkeypatch): + from litellm.proxy import proxy_server + from litellm.proxy._types import LitellmUserRoles + + route_client.auth.user_role = LitellmUserRoles.PROXY_ADMIN + monkeypatch.setattr(proxy_server, "llm_model_list", [{"model_name": "voice"}]) + route_client.transport.request.return_value = httpx.Response(200, content=b"") + body = {"session": {"model": "voice", "type": "live", "instructions": "incoming"}} + missing = route_client.client.post("/v1/live/sessions/sess_incoming/accept", json=body) + assert missing.status_code == 400 + route_client.transport.request.assert_not_awaited() + accepted = route_client.client.post( + "/v1/live/sessions/sess_incoming/accept", json=body, headers={"x-litellm-live-model": "voice"} + ) + assert accepted.status_code == 200 and accepted.content == b"" + token = accepted.headers["x-litellm-live-session-id"] + owned = live.decode_session(token, live._owner(route_client.auth)) + assert owned.session_id == "sess_incoming" and owned.alias == "voice" + assert route_client.transport.request.await_args.kwargs["body"]["session"]["model"] == "gpt-live" + route_client.supervised.assert_awaited_once() + raw_hangup = route_client.client.post("/v1/live/sessions/sess_incoming/hangup") + assert raw_hangup.status_code == 403 + assert route_client.client.post(f"/v1/live/sessions/{token}/hangup").status_code == 200 + + +def test_admin_sip_cannot_enroll_through_alias_with_multiple_accounts(route_client, monkeypatch): + from litellm.proxy import proxy_server + from litellm.proxy._types import LitellmUserRoles + + route_client.auth.user_role = LitellmUserRoles.PROXY_ADMIN + monkeypatch.setattr(proxy_server, "llm_model_list", [{"model_name": "voice"}, {"model_name": "voice"}]) + response = route_client.client.post( + "/v1/live/sessions/sess_incoming/reject", json={"status_code": 603}, headers={"x-litellm-live-model": "voice"} + ) + assert response.status_code == 400 + route_client.transport.request.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_primary_first_frame_timeout_releases_authenticated_reservation(monkeypatch): + import asyncio + + from fastapi import WebSocket + + messages = iter([{"type": "websocket.connect"}]) + + async def receive(): + try: + return next(messages) + except StopIteration: + raise asyncio.TimeoutError("first message timeout") + + sent = [] + + async def send(message): + sent.append(message) + + websocket = WebSocket( + { + "type": "websocket", + "path": "/v1/live/sessions", + "query_string": b"", + "headers": [(b"authorization", b"Bearer owner")], + "scheme": "ws", + "server": ("localhost", 4000), + }, + receive, + send, + ) + monkeypatch.setattr(live, "_auth", AsyncMock(return_value=UserAPIKeyAuth(api_key="owner"))) + release = AsyncMock() + transport = Mock() + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", release) + monkeypatch.setattr(live, "LiveTransport", transport) + await live.websocket_live_session(websocket) + release.assert_awaited_once() + transport.assert_not_called() + assert sent[-1]["type"] == "websocket.close" and sent[-1]["code"] == 1008 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("key_limit,global_limit,allowed", [(None, None, True), (1, None, False), (None, 2, False)]) +async def test_legacy_limiter_only_blocks_sessions_requiring_parallel_leases( + monkeypatch, key_limit, global_limit, allowed +): + from litellm.proxy import proxy_server + from litellm.proxy.hooks.parallel_request_limiter import _PROXY_MaxParallelRequestsHandler + + legacy = Mock(spec=_PROXY_MaxParallelRequestsHandler) + monkeypatch.setattr(proxy_server, "proxy_logging_obj", SimpleNamespace(get_proxy_hook=lambda _: legacy)) + monkeypatch.setattr(proxy_server, "general_settings", {"global_max_parallel_requests": global_limit}) + monkeypatch.setattr(live, "_authorize", AsyncMock()) + processor = AsyncMock(return_value=({"model": "voice"}, Mock())) + monkeypatch.setattr(live, "process_codex_request", processor) + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", AsyncMock()) + request = live._request(Request({"type": "http", "headers": []}), {"model": "voice"}) + auth = UserAPIKeyAuth(api_key="owner", max_parallel_requests=key_limit) + if allowed: + async with live._precall(request, auth, "voice") as prepared: + assert prepared.lease is None + processor.assert_awaited_once() + return + with pytest.raises(HTTPException) as rejected: + async with live._precall(request, auth, "voice"): + pytest.fail("Parallel-limited sessions require renewable leases") + assert rejected.value.status_code == 400 + processor.assert_not_awaited() + + +@pytest.mark.parametrize("delegation", ["invalid", {"type": "responses", "responses": "invalid"}]) +def test_malformed_delegation_returns_client_error_before_provider(route_client, delegation): + result = route_client.client.post( + "/v1/live/sessions", + json={"session": {"model": "voice", "delegation": delegation}, "transport": {"type": "webrtc", "sdp": "offer"}}, + ) + assert result.status_code == 400 + route_client.transport.request.assert_not_awaited() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("parallel_reserved", [False, True]) +async def test_attachment_skips_parallel_admission_only_with_existing_session_lease(monkeypatch, parallel_reserved): + from litellm.proxy import proxy_server + from litellm.proxy.hooks.realtime_call_lease import is_realtime_call_attachment + + attachment = object() + observed = [] + + async def process(request, data, auth, model, route_type): + observed.append(is_realtime_call_attachment(attachment)) + return data, Mock() + + monkeypatch.setattr(proxy_server, "proxy_logging_obj", SimpleNamespace(get_proxy_hook=lambda _: None)) + monkeypatch.setattr(live, "_authorize", AsyncMock()) + monkeypatch.setattr(live, "process_codex_request", process) + monkeypatch.setattr(live, "release_or_invalidate_budget_reservation", AsyncMock()) + request = live._request(Request({"type": "http", "headers": []}), {"model": "voice"}) + async with live._precall( + request, UserAPIKeyAuth(api_key="owner"), "voice", attachment=attachment, parallel_reserved=parallel_reserved + ): + pass + assert observed == [parallel_reserved] + + +@pytest.mark.asyncio +async def test_live_observer_becomes_ready_without_session_started_event(monkeypatch): + import asyncio + + from litellm.proxy.realtime_endpoints.call_supervision import CallSupervisor + + class Observer: + def __init__(self): + self.queue = asyncio.Queue() + self.closed = False + + def __aiter__(self): + return self + + async def __anext__(self): + return await self.queue.get() + + async def close(self): + self.closed = True + + observer = Observer() + logger = Mock(litellm_params={}, model_call_details={}) + monkeypatch.setattr(live, "process_codex_request", AsyncMock(return_value=({"model": "voice"}, logger))) + sink = SimpleNamespace(store_message=Mock(), log_messages=AsyncMock()) + monkeypatch.setattr(live, "RealTimeStreaming", Mock(return_value=sink)) + + async def hangup(*args, **kwargs): + await observer.queue.put(json.dumps({"type": "session.closed", "usage": {"seconds": 0}})) + return httpx.Response(200, request=httpx.Request("POST", "https://upstream.test/hangup")) + + monkeypatch.setattr( + live, + "LiveTransport", + Mock(return_value=SimpleNamespace(connect=AsyncMock(return_value=observer), request=hangup)), + ) + supervisors = [] + + def build_supervisor(*args, **kwargs): + return CallSupervisor( + *args, **kwargs, ready_timeout=0.1, drain_timeout=0.1, termination_timeout=0.2, logging_timeout=0.2 + ) + + async def start(supervisor): + supervisors.append(supervisor) + await supervisor.start() + + monkeypatch.setattr(live, "CallSupervisor", build_supervisor) + monkeypatch.setattr(live, "CALL_SUPERVISORS", SimpleNamespace(start=start)) + request = Request({"type": "http", "method": "POST", "path": "/v1/live/sessions", "headers": []}) + try: + result = await asyncio.wait_for( + live._start_supervisor(request, handle(), UserAPIKeyAuth(api_key="owner"), None), 0.5 + ) + assert result is sink + sink.store_message.assert_not_called() + finally: + for supervisor in supervisors: + await supervisor.close() + assert observer.closed diff --git a/tests/test_litellm/proxy/realtime_endpoints/test_realtime_webrtc_endpoints.py b/tests/test_litellm/proxy/realtime_endpoints/test_realtime_webrtc_endpoints.py index 82f2ef097aa..3fdcf3d4b52 100644 --- a/tests/test_litellm/proxy/realtime_endpoints/test_realtime_webrtc_endpoints.py +++ b/tests/test_litellm/proxy/realtime_endpoints/test_realtime_webrtc_endpoints.py @@ -7,7 +7,7 @@ Tests for LiteLLM proxy realtime WebRTC HTTP endpoints: import json import time from collections.abc import Awaitable -from typing import Protocol +from typing import Final, Protocol from unittest.mock import AsyncMock, MagicMock, patch import httpx @@ -15,7 +15,6 @@ import pytest from fastapi import FastAPI, HTTPException from fastapi.testclient import TestClient - from litellm.proxy._types import UserAPIKeyAuth from litellm.proxy.auth.user_api_key_auth import user_api_key_auth from litellm.proxy.common_utils.encrypt_decrypt_utils import ( @@ -120,6 +119,114 @@ def proxy_app(monkeypatch): return proxy_server.app +@pytest.mark.parametrize("path", ["/v1/live", "/live", "/openai/v1/live"]) +@pytest.mark.parametrize("query", ["", "?intent=custom&architecture=custom"]) +def test_live_multipart_offer_runs_authenticated_call_pipeline( + proxy_app: FastAPI, monkeypatch: pytest.MonkeyPatch, path: str, query: str +) -> None: + from litellm.proxy import proxy_server + from litellm.proxy.realtime_endpoints import call_sessions + + monkeypatch.setenv("LITELLM_SALT_KEY", "test-live-signaling-salt") + session: Final = {"model": "gpt-live-1-codex", "audio": {"output": {"voice": "sol"}}} + authenticate: Final = AsyncMock(wraps=call_sessions.user_api_key_auth) + process: Final = AsyncMock(side_effect=lambda request, data, *args: (data, MagicMock())) + supervise: Final = AsyncMock() + upstream: Final = httpx.Response( + 201, + content=b"v=0\r\nanswer", + headers={"Location": "/v1/live/rtc_private"}, + extensions={"chatgpt_realtime": {"model": "gpt-live-1-codex"}}, + ) + + async def route(**kwargs: object) -> Awaitable[httpx.Response]: + assert kwargs["route_type"] == "arealtime_calls" + assert isinstance(kwargs["data"], dict) + assert kwargs["data"]["sdp_body"] == b"v=0\r\noffer" + assert kwargs["data"]["session"] == session + assert kwargs["data"]["chatgpt_realtime_client_query"] == ( + {"architecture": "custom", "intent": "custom"} + if query + else {"architecture": "avas", "intent": "quicksilver"} + ) + + async def respond() -> httpx.Response: + return upstream + + return respond() + + monkeypatch.setattr(call_sessions, "user_api_key_auth", authenticate) + monkeypatch.setattr(call_sessions, "process_codex_request", process) + monkeypatch.setattr(call_sessions, "supervise_codex_call", supervise) + monkeypatch.setattr(proxy_server, "route_request", route) + response: Final = TestClient(proxy_app).post( + f"{path}{query}", + headers={"Authorization": "Bearer sk-test-master-key"}, + files={ + "sdp": (None, "v=0\r\noffer", "application/sdp"), + "session": (None, json.dumps(session), "application/json"), + }, + ) + + assert response.status_code == 201 + assert response.content == b"v=0\r\nanswer" + assert response.headers["content-type"] == "application/sdp" + authenticate.assert_awaited_once() + process.assert_awaited_once() + assert process.await_args.args[3:] == ("gpt-live-1-codex", "arealtime_calls") + assert isinstance(process.await_args.args[2], UserAPIKeyAuth) + supervise.assert_awaited_once() + assert response.headers["location"].startswith("/v1/live/") + token: Final = response.headers["location"].rsplit("/", 1)[-1] + call: Final = call_sessions.decode_call(token, "Bearer sk-test-master-key") + assert call.call_id == "rtc_private" + assert call.alias == "gpt-live-1-codex" + assert call.usage_supervised + + +@pytest.mark.parametrize("path", ["/v1/live", "/live", "/openai/v1/live"]) +def test_live_multipart_offer_rejects_invalid_credentials_before_routing( + proxy_app: FastAPI, monkeypatch: pytest.MonkeyPatch, path: str +) -> None: + from litellm.proxy import proxy_server + from litellm.proxy.realtime_endpoints import call_sessions + + authenticate: Final = AsyncMock(wraps=call_sessions.user_api_key_auth) + route: Final = AsyncMock() + monkeypatch.setattr(call_sessions, "user_api_key_auth", authenticate) + monkeypatch.setattr(proxy_server, "route_request", route) + response: Final = TestClient(proxy_app).post( + path, + files={"sdp": (None, "v=0\r\n"), "session": (None, '{"model":"gpt-live-1-codex"}')}, + ) + + assert response.status_code == 401 + authenticate.assert_awaited_once() + route.assert_not_awaited() + + +def test_live_multipart_offer_rejects_model_outside_key_scope( + proxy_app: FastAPI, monkeypatch: pytest.MonkeyPatch +) -> None: + from litellm.proxy import proxy_server + from litellm.proxy.realtime_endpoints import call_sessions + + authenticate: Final = AsyncMock(return_value=UserAPIKeyAuth(models=["another-model"])) + route: Final = AsyncMock() + monkeypatch.setattr(call_sessions, "user_api_key_auth", authenticate) + monkeypatch.setattr(proxy_server, "route_request", route) + response: Final = TestClient(proxy_app).post( + "/v1/live", + headers={"Authorization": "Bearer restricted-key"}, + files={"sdp": (None, "v=0\r\n"), "session": (None, '{"model":"gpt-live-1-codex"}')}, + ) + + assert response.status_code == 403 + assert "gpt-live-1-codex" in response.text + authenticate.assert_awaited_once() + route.assert_not_awaited() + + @pytest.fixture def mock_route_request_client_secrets(): """Mock route_request to return a fake upstream client_secrets response.""" diff --git a/tests/test_litellm/proxy/test_live_route_registration.py b/tests/test_litellm/proxy/test_live_route_registration.py new file mode 100644 index 00000000000..d280b9b1a30 --- /dev/null +++ b/tests/test_litellm/proxy/test_live_route_registration.py @@ -0,0 +1,57 @@ +from unittest.mock import AsyncMock + +import httpx +import pytest +from fastapi import HTTPException +from fastapi.testclient import TestClient + + +@pytest.mark.asyncio +@pytest.mark.parametrize("prefix", ["/live", "/v1/live", "/openai/v1/live"]) +@pytest.mark.parametrize( + ("method", "suffix"), + [ + ("POST", ""), + ("POST", "/opaque/fork"), + ("POST", "/opaque/accept"), + ("POST", "/opaque/reject"), + ("POST", "/opaque/refer"), + ("POST", "/opaque/hangup"), + ("GET", "/opaque/content"), + ], +) +async def test_public_live_http_routes_reach_live_auth_before_generic_passthrough(monkeypatch, prefix, method, suffix): + from litellm.proxy import proxy_server + from litellm.proxy.realtime_endpoints import live + + authenticate = AsyncMock(side_effect=HTTPException(401, "Live authentication required")) + monkeypatch.setattr(live, "_auth", authenticate) + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=proxy_server.app), base_url="http://proxy" + ) as client: + response = await client.request( + method, + prefix + "/sessions" + suffix, + json={"session": {"model": "voice"}, "transport": {"type": "webrtc", "sdp": "offer"}}, + ) + assert response.status_code == 401 + assert "Live authentication required" in response.text + authenticate.assert_awaited_once() + + +@pytest.mark.parametrize("prefix", ["/live", "/v1/live", "/openai/v1/live"]) +@pytest.mark.parametrize("suffix", ["", "/opaque/attach", "/opaque/fork"]) +def test_public_live_websockets_reach_live_auth_before_legacy_sideband(monkeypatch, prefix, suffix): + from starlette.websockets import WebSocketDisconnect + + from litellm.proxy import proxy_server + from litellm.proxy.realtime_endpoints import live + + authenticate = AsyncMock(side_effect=HTTPException(403, "Live authentication rejected")) + monkeypatch.setattr(live, "_auth", authenticate) + monkeypatch.setattr(proxy_server, "general_settings", {}) + with TestClient(proxy_server.app) as client: + with pytest.raises(WebSocketDisconnect): + with client.websocket_connect(prefix + "/sessions" + suffix, headers={"authorization": "Bearer test"}): + pass + authenticate.assert_awaited_once() diff --git a/tests/test_litellm/test_cost_calculator.py b/tests/test_litellm/test_cost_calculator.py index 6c4260d5d3e..0364e609a30 100644 --- a/tests/test_litellm/test_cost_calculator.py +++ b/tests/test_litellm/test_cost_calculator.py @@ -1,4 +1,3 @@ - import json from pathlib import Path from typing import Final @@ -1822,7 +1821,6 @@ def test_azure_ai_cache_cost_calculation(_local_model_cost_map): ), f"Output cost mismatch: got {output_cost}, expected {expected_output_cost}" - AZURE_GPT_5_6_MAP_KEYS = ( "azure/gpt-5.6", "azure/gpt-5.6-sol", @@ -1889,6 +1887,7 @@ def test_azure_gpt_5_6_rates_match_azure_price_page(_local_model_cost_map, model for key in token_cost_keys: assert entry[key] == pytest.approx(global_entry[key] * 1.1), key + def test_vertex_regional_deployment_costs_uplift_over_global(monkeypatch): """ Regression for https://github.com/BerriAI/litellm/issues/34393: two Vertex @@ -4221,6 +4220,8 @@ def test_completion_cost_together_metadata_only_model_still_uses_size_bucket(_lo ) assert cost == pytest.approx((23 + 15) * 8e-07, rel=1e-9) + + def test_select_model_name_strips_unregistered_alias_prefix(_local_model_cost_map): """A router-facing model_name alias containing "/" whose leading segment is NOT a registered provider must not be double-prefixed into a non-existent cost key. @@ -4864,13 +4865,16 @@ def test_live_terminal_duration_uses_configured_second_price(monkeypatch, rate, ) == pytest.approx(expected) -def test_live_terminal_duration_honors_deployment_override(monkeypatch): +@pytest.mark.parametrize("public_live", [False, True]) +def test_live_terminal_duration_honors_deployment_override(monkeypatch, public_live): monkeypatch.setitem( litellm.model_cost, "live-deployment-test", {"litellm_provider": "chatgpt", "mode": "realtime", "input_cost_per_second": 0.025}, ) - result = RealtimeAPITokenUsageProcessor.create_logging_realtime_object(Usage(), [_live_terminal_event()]) + result = RealtimeAPITokenUsageProcessor.create_logging_realtime_object( + Usage(), [{"type": "session.closed", "usage": {"seconds": 4}}] if public_live else [_live_terminal_event()] + ) assert completion_cost( completion_response=result, model="gpt-live-1", @@ -5170,3 +5174,216 @@ def test_completion_cost_ocr_ignores_deployment_pricing_without_custom_pricing_f litellm_logging_obj=logging_obj, ) assert cost == 0.0 + + +@pytest.mark.parametrize("terminal", [False, True]) +def test_public_live_seconds_are_cumulative_and_backend_usage_is_separately_priced(monkeypatch, terminal): + monkeypatch.setitem( + litellm.model_cost, + "live-seconds-test", + { + "litellm_provider": "openai", + "mode": "realtime", + "input_cost_per_second": 0.025, + "input_cost_per_token": 100, + "output_cost_per_token": 100, + }, + ) + monkeypatch.setitem( + litellm.model_cost, + "live-backend-test", + { + "litellm_provider": "openai", + "mode": "responses", + "input_cost_per_token": 0.001, + "output_cost_per_token": 0.002, + }, + ) + backend = { + "type": "response.event", + "event": { + "type": "response.completed", + "response": { + "id": "resp_backend", + "created_at": 1, + "model": "live-backend-test", + "output": [], + "usage": {"input_tokens": 20, "output_tokens": 10, "total_tokens": 30}, + }, + }, + } + events = [ + {"type": "session.usage.updated", "usage": {"seconds": 15}}, + {"type": "session.usage.updated", "usage": {"seconds": 30}}, + backend, + backend, + {"type": "session.closed" if terminal else "session.usage.updated", "usage": {"seconds": 30}}, + ] + usage = RealtimeAPITokenUsageProcessor.collect_and_combine_usage_from_realtime_stream_results(events) + assert usage.total_tokens == 30 + assert handle_realtime_stream_cost_calculation(events, usage, "openai", "live-seconds-test") == pytest.approx(0.79) + + +@pytest.mark.parametrize("seconds", [-1, True, "30", float("inf"), float("nan"), None]) +def test_public_live_invalid_seconds_are_not_billed(monkeypatch, seconds): + monkeypatch.setitem( + litellm.model_cost, + "live-seconds-test", + { + "litellm_provider": "openai", + "mode": "realtime", + "input_cost_per_second": 0.025, + }, + ) + assert ( + handle_realtime_stream_cost_calculation( + [{"type": "session.closed", "usage": {"seconds": seconds}}], Usage(), "openai", "live-seconds-test" + ) + == 0 + ) + + +@pytest.mark.parametrize("terminal", [False, True]) +def test_live_duration_does_not_regress_when_primary_and_observer_events_interleave(monkeypatch, terminal): + monkeypatch.setitem( + litellm.model_cost, + "live-interleaved-test", + {"litellm_provider": "openai", "mode": "realtime", "input_cost_per_second": 0.025}, + ) + events = [ + {"type": "session.closed" if terminal else "session.usage.updated", "usage": {"seconds": 30}}, + {"type": "session.usage.updated", "usage": {"seconds": 15}}, + ] + assert handle_realtime_stream_cost_calculation(events, Usage(), "openai", "live-interleaved-test") == pytest.approx( + 0.75 + ) + + +@pytest.mark.parametrize("seconds,expected", [(None, 15), (0, 15), (4, 15), (15, 15), (30, 30)]) +def test_live_webrtc_initialization_is_credited_against_duration(monkeypatch, seconds, expected): + monkeypatch.setitem( + litellm.model_cost, + "live-init-test", + { + "litellm_provider": "openai", + "mode": "realtime", + "input_cost_per_second": 0.025, + }, + ) + events = [{"type": "litellm.live.initialization", "usage": {"seconds": 15}}] + if seconds is not None: + events.append({"type": "session.closed", "usage": {"seconds": seconds}}) + assert handle_realtime_stream_cost_calculation(events, Usage(), "openai", "live-init-test") == pytest.approx( + expected * 0.025 + ) + + +def test_live_invalid_terminal_retains_last_reported_partial_usage(monkeypatch): + monkeypatch.setitem( + litellm.model_cost, + "live-partial-test", + { + "litellm_provider": "openai", + "mode": "realtime", + "input_cost_per_second": 0.025, + }, + ) + events = [ + {"type": "litellm.live.initialization", "usage": {"seconds": 15}}, + {"type": "session.usage.updated", "usage": {"seconds": 30}}, + {"type": "session.closed", "usage": {"seconds": "invalid"}}, + ] + assert handle_realtime_stream_cost_calculation(events, Usage(), "openai", "live-partial-test") == pytest.approx( + 0.75 + ) + + +@pytest.mark.parametrize( + "nested", + [ + {"type": "response.created", "response": {"model": "still-starting"}}, + {"type": "response.in_progress", "response": {}}, + {"type": "future.event", "response": ["unknown", "payload"]}, + {"type": "response.completed", "response": {"id": "broken", "usage": "invalid"}}, + ], +) +def test_live_partial_or_malformed_backend_events_preserve_duration(monkeypatch, nested): + from unittest.mock import MagicMock + + monkeypatch.setitem( + litellm.model_cost, + "live-resilient-test", + { + "litellm_provider": "openai", + "mode": "realtime", + "input_cost_per_second": 0.025, + }, + ) + logger = MagicMock(spec=Logging) + logger.model_call_details = {} + events = [ + {"type": "response.event", "event": nested}, + {"type": "session.closed", "usage": {"seconds": 30}}, + ] + usage = RealtimeAPITokenUsageProcessor.collect_and_combine_usage_from_realtime_stream_results(events) + assert usage.total_tokens == 0 + assert handle_realtime_stream_cost_calculation( + events, + usage, + "openai", + "live-resilient-test", + litellm_logging_obj=logger, + ) == pytest.approx(0.75) + assert bool(logger.model_call_details.get("realtime_backend_accounting_incomplete")) == ( + nested["type"] == "response.completed" + ) + + +@pytest.mark.parametrize("missing_usage_duplicate", [False, True]) +def test_live_missing_backend_price_preserves_duration_and_marks_accounting_incomplete( + monkeypatch, missing_usage_duplicate +): + from unittest.mock import MagicMock + + monkeypatch.setitem( + litellm.model_cost, + "live-resilient-test", + { + "litellm_provider": "openai", + "mode": "realtime", + "input_cost_per_second": 0.025, + }, + ) + logger = MagicMock(spec=Logging) + logger.model_call_details = {} + response = { + "id": "resp_unknown", + "created_at": 1, + "model": "unmapped-live-backend-price-test", + "output": [], + "usage": {"input_tokens": 20, "output_tokens": 10, "total_tokens": 30}, + } + events = [ + {"type": "response.event", "event": {"type": "response.completed", "response": response}}, + *( + [ + { + "type": "response.event", + "event": {"type": "response.completed", "response": {**response, "usage": None}}, + } + ] + if missing_usage_duplicate + else [] + ), + {"type": "session.closed", "usage": {"seconds": 30}}, + ] + usage = RealtimeAPITokenUsageProcessor.collect_and_combine_usage_from_realtime_stream_results(events) + assert usage.total_tokens == 30 + assert handle_realtime_stream_cost_calculation( + events, + usage, + "openai", + "live-resilient-test", + litellm_logging_obj=logger, + ) == pytest.approx(0.75) + assert logger.model_call_details["realtime_backend_accounting_incomplete"] is True diff --git a/ui/litellm-dashboard/src/lib/http/schema.d.ts b/ui/litellm-dashboard/src/lib/http/schema.d.ts index 97545f327a5..1e845ad8847 100644 --- a/ui/litellm-dashboard/src/lib/http/schema.d.ts +++ b/ui/litellm-dashboard/src/lib/http/schema.d.ts @@ -8372,7 +8372,7 @@ export interface paths { * WebSocket: realtime_websocket_endpoint * @description WebSocket connection endpoint */ - get: operations["websocket_realtime_websocket_endpoint_get_4"]; + get: operations["websocket_realtime_websocket_endpoint_get_5"]; put?: never; post?: never; delete?: never; @@ -8381,6 +8381,145 @@ export interface paths { patch?: never; trace?: never; }; + "/live/": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * WebSocket: codex_live_sideband_endpoint + * @description WebSocket connection endpoint + */ + get: operations["websocket_codex_live_sideband_endpoint_get_2"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Create Live Session */ + post: operations["create_live_session_live_sessions_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions/{session_id}/accept": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_live_sessions__session_id__accept_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions/{session_id}/content": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** Control Live Session */ + get: operations["control_live_session_live_sessions__session_id__content_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions/{session_id}/fork": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Fork Live Session */ + post: operations["fork_live_session_live_sessions__session_id__fork_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions/{session_id}/hangup": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_live_sessions__session_id__hangup_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions/{session_id}/refer": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_live_sessions__session_id__refer_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/live/sessions/{session_id}/reject": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_live_sessions__session_id__reject_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/login": { parameters: { query?: never; @@ -9780,6 +9919,165 @@ export interface paths { patch?: never; trace?: never; }; + "/openai/v1/live": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * WebSocket: realtime_websocket_endpoint + * @description WebSocket connection endpoint + */ + get: operations["websocket_realtime_websocket_endpoint_get_4"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * WebSocket: codex_live_sideband_endpoint + * @description WebSocket connection endpoint + */ + get: operations["websocket_codex_live_sideband_endpoint_get_3"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Create Live Session */ + post: operations["create_live_session_openai_v1_live_sessions_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions/{session_id}/accept": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_openai_v1_live_sessions__session_id__accept_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions/{session_id}/content": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** Control Live Session */ + get: operations["control_live_session_openai_v1_live_sessions__session_id__content_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions/{session_id}/fork": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Fork Live Session */ + post: operations["fork_live_session_openai_v1_live_sessions__session_id__fork_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions/{session_id}/hangup": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_openai_v1_live_sessions__session_id__hangup_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions/{session_id}/refer": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_openai_v1_live_sessions__session_id__refer_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/openai/v1/live/sessions/{session_id}/reject": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_openai_v1_live_sessions__session_id__reject_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/openai/v1/realtime": { parameters: { query?: never; @@ -18410,7 +18708,7 @@ export interface paths { * WebSocket: realtime_websocket_endpoint * @description WebSocket connection endpoint */ - get: operations["websocket_realtime_websocket_endpoint_get_5"]; + get: operations["websocket_realtime_websocket_endpoint_get_6"]; put?: never; post?: never; delete?: never; @@ -18430,7 +18728,7 @@ export interface paths { * WebSocket: codex_live_sideband_endpoint * @description WebSocket connection endpoint */ - get: operations["websocket_codex_live_sideband_endpoint"]; + get: operations["websocket_codex_live_sideband_endpoint_get"]; put?: never; post?: never; delete?: never; @@ -18439,6 +18737,125 @@ export interface paths { patch?: never; trace?: never; }; + "/v1/live/sessions": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Create Live Session */ + post: operations["create_live_session_v1_live_sessions_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/v1/live/sessions/{session_id}/accept": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_v1_live_sessions__session_id__accept_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/v1/live/sessions/{session_id}/content": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** Control Live Session */ + get: operations["control_live_session_v1_live_sessions__session_id__content_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/v1/live/sessions/{session_id}/fork": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Fork Live Session */ + post: operations["fork_live_session_v1_live_sessions__session_id__fork_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/v1/live/sessions/{session_id}/hangup": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_v1_live_sessions__session_id__hangup_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/v1/live/sessions/{session_id}/refer": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_v1_live_sessions__session_id__refer_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/v1/live/sessions/{session_id}/reject": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** Control Live Session */ + post: operations["control_live_session_v1_live_sessions__session_id__reject_post"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/v1/mcp/access_groups": { parameters: { query?: never; @@ -51059,7 +51476,7 @@ export interface operations { }; }; }; - websocket_realtime_websocket_endpoint_get_4: { + websocket_realtime_websocket_endpoint_get_5: { parameters: { query?: never; header?: never; @@ -51077,6 +51494,230 @@ export interface operations { }; }; }; + websocket_codex_live_sideband_endpoint_get_2: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description WebSocket Protocol Switched */ + 101: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + create_live_session_live_sessions_post: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + }; + }; + control_live_session_live_sessions__session_id__accept_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_live_sessions__session_id__content_get: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + fork_live_session_live_sessions__session_id__fork_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_live_sessions__session_id__hangup_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_live_sessions__session_id__refer_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_live_sessions__session_id__reject_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; login_login_post: { parameters: { query?: never; @@ -53252,6 +53893,248 @@ export interface operations { }; }; }; + websocket_realtime_websocket_endpoint_get_4: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description WebSocket Protocol Switched */ + 101: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + websocket_codex_live_sideband_endpoint_get_3: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description WebSocket Protocol Switched */ + 101: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + create_live_session_openai_v1_live_sessions_post: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + }; + }; + control_live_session_openai_v1_live_sessions__session_id__accept_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_openai_v1_live_sessions__session_id__content_get: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + fork_live_session_openai_v1_live_sessions__session_id__fork_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_openai_v1_live_sessions__session_id__hangup_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_openai_v1_live_sessions__session_id__refer_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_openai_v1_live_sessions__session_id__reject_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; websocket_realtime_websocket_endpoint_get_3: { parameters: { query?: never; @@ -63397,7 +64280,7 @@ export interface operations { }; }; }; - websocket_realtime_websocket_endpoint_get_5: { + websocket_realtime_websocket_endpoint_get_6: { parameters: { query?: never; header?: never; @@ -63415,7 +64298,7 @@ export interface operations { }; }; }; - websocket_codex_live_sideband_endpoint: { + websocket_codex_live_sideband_endpoint_get: { parameters: { query?: never; header?: never; @@ -63433,6 +64316,212 @@ export interface operations { }; }; }; + create_live_session_v1_live_sessions_post: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + }; + }; + control_live_session_v1_live_sessions__session_id__accept_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_v1_live_sessions__session_id__content_get: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + fork_live_session_v1_live_sessions__session_id__fork_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_v1_live_sessions__session_id__hangup_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_v1_live_sessions__session_id__refer_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + control_live_session_v1_live_sessions__session_id__reject_post: { + parameters: { + query?: never; + header?: never; + path: { + session_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; get_mcp_access_groups_v1_mcp_access_groups_get: { parameters: { query?: never;