diff --git a/docs/my-website/docs/providers/runwayml/text-to-speech.md b/docs/my-website/docs/providers/runwayml/text-to-speech.md new file mode 100644 index 00000000000..020269863c6 --- /dev/null +++ b/docs/my-website/docs/providers/runwayml/text-to-speech.md @@ -0,0 +1,244 @@ +# RunwayML - Text-to-Speech + +## Overview + +| Property | Details | +|-------|-------| +| Description | RunwayML provides high-quality AI-powered text-to-speech with natural-sounding voices | +| Provider Route on LiteLLM | `runwayml/` | +| Supported Operations | [`/audio/speech`](#quick-start) | +| Link to Provider Doc | [RunwayML API ↗](https://docs.dev.runwayml.com/) | + +LiteLLM supports RunwayML's text-to-speech API with automatic task polling, allowing you to generate natural-sounding audio from text. + +## Quick Start + +```python showLineNumbers title="Basic Text-to-Speech" +from litellm import speech +import os + +os.environ["RUNWAYML_API_KEY"] = "your-api-key" + +response = speech( + model="runwayml/eleven_multilingual_v2", + input="Step right up, ladies and gentlemen! Have you ever wished for a toaster that's not just a toaster but a marvel of modern ingenuity?", + voice="alloy" +) + +# Save the audio +with open("output.mp3", "wb") as f: + f.write(response.content) +``` + +## Authentication + +Set your RunwayML API key: + +```python showLineNumbers title="Set API Key" +import os + +os.environ["RUNWAYML_API_KEY"] = "your-api-key" +``` + +## Supported Parameters + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `model` | string | Yes | Model to use (e.g., `runwayml/eleven_multilingual_v2`) | +| `input` | string | Yes | Text to convert to speech | +| `voice` | string or dict | Yes | Voice to use (OpenAI name, RunwayML preset, or voice config) | + +## Voice Options + +### Using OpenAI Voice Names + +OpenAI voice names are automatically mapped to appropriate RunwayML voices: + +```python showLineNumbers title="OpenAI Voice Names" +from litellm import speech + +# These OpenAI voice names work automatically +response = speech( + model="runwayml/eleven_multilingual_v2", + input="Hello, world!", + voice="alloy" # Maya - neutral, balanced female voice +) +``` + +**Voice Mappings:** +- `alloy` → Maya (neutral, balanced female voice) +- `echo` → James (male voice) +- `fable` → Bernard (warm, storytelling voice) +- `onyx` → Vincent (deep male voice) +- `nova` → Serene (warm, expressive female voice) +- `shimmer` → Ella (clear, friendly female voice) + +### Using RunwayML Preset Voices + +You can directly specify any RunwayML preset voice by passing the preset name as a string: + +```python showLineNumbers title="RunwayML Preset Names" +from litellm import speech + +# Pass the RunwayML voice name as a string +response = speech( + model="runwayml/eleven_multilingual_v2", + input="Hello, world!", + voice="Maya" # LiteLLM automatically formats this for RunwayML +) + +# Try different RunwayML voices +response = speech( + model="runwayml/eleven_multilingual_v2", + input="Step right up, ladies and gentlemen!", + voice="Bernard" # Great for storytelling +) +``` + +**Available RunwayML Voices:** + +Maya, Arjun, Serene, Bernard, Billy, Mark, Clint, Mabel, Chad, Leslie, Eleanor, Elias, Elliot, Grungle, Brodie, Sandra, Kirk, Kylie, Lara, Lisa, Malachi, Marlene, Martin, Miriam, Monster, Paula, Pip, Rusty, Ragnar, Xylar, Maggie, Jack, Katie, Noah, James, Rina, Ella, Mariah, Frank, Claudia, Niki, Vincent, Kendrick, Myrna, Tom, Wanda, Benjamin, Kiana, Rachel + +:::tip +Simply pass the voice name as a string - LiteLLM automatically handles the internal RunwayML API format conversion. +::: + +## Async Usage + +```python showLineNumbers title="Async Text-to-Speech" +from litellm import aspeech +import os +import asyncio + +os.environ["RUNWAYML_API_KEY"] = "your-api-key" + +async def generate_speech(): + response = await aspeech( + model="runwayml/eleven_multilingual_v2", + input="This is an asynchronous text-to-speech request.", + voice="nova" + ) + + with open("output.mp3", "wb") as f: + f.write(response.content) + + print("Audio generated successfully!") + +asyncio.run(generate_speech()) +``` + +## LiteLLM Proxy Usage + +Add RunwayML to your proxy configuration: + +```yaml showLineNumbers title="config.yaml" +model_list: + - model_name: runway-tts + litellm_params: + model: runwayml/eleven_multilingual_v2 + api_key: os.environ/RUNWAYML_API_KEY +``` + +Start the proxy: + +```bash +litellm --config /path/to/config.yaml +``` + +Generate speech through the proxy: + +```bash showLineNumbers title="Proxy Request" +curl --location 'http://localhost:4000/v1/audio/speech' \ +--header 'Content-Type: application/json' \ +--header 'x-litellm-api-key: sk-1234' \ +--data '{ + "model": "runwayml/eleven_multilingual_v2", + "input": "Hello from the LiteLLM proxy!", + "voice": "alloy" +}' +``` + +With RunwayML-specific voice: + +```bash showLineNumbers title="Proxy Request with RunwayML Voice" +curl --location 'http://localhost:4000/v1/audio/speech' \ +--header 'Content-Type: application/json' \ +--header 'x-litellm-api-key: sk-1234' \ +--data '{ + "model": "runwayml/eleven_multilingual_v2", + "input": "Hello with a custom RunwayML voice!", + "voice": "Bernard" +}' +``` + +## Supported Models + +| Model | Description | +|-------|-------------| +| `runwayml/eleven_multilingual_v2` | High-quality multilingual text-to-speech | + +## Cost Tracking + +LiteLLM automatically tracks RunwayML text-to-speech costs: + +```python showLineNumbers title="Cost Tracking" +from litellm import speech, completion_cost + +response = speech( + model="runwayml/eleven_multilingual_v2", + input="Hello, world!", + voice="alloy" +) + +cost = completion_cost(completion_response=response) +print(f"Text-to-speech cost: ${cost}") +``` + +## Supported Features + +| Feature | Supported | +|---------|-----------| +| Text-to-Speech | ✅ | +| Cost Tracking | ✅ | +| Logging | ✅ | +| Fallbacks | ✅ | +| Load Balancing | ✅ | +| 50+ Voice Presets | ✅ | + +## How It Works + +RunwayML uses an asynchronous task-based API pattern. LiteLLM handles the polling and response transformation automatically. + +### Complete Flow Diagram + +```mermaid +sequenceDiagram + participant Client + box rgb(200, 220, 255) LiteLLM AI Gateway + participant LiteLLM + end + participant RunwayML as RunwayML API + participant Storage as Audio Storage + + Client->>LiteLLM: POST /audio/speech (OpenAI format) + Note over LiteLLM: Transform to RunwayML format
Map voice to preset ID + + LiteLLM->>RunwayML: POST v1/text_to_speech + RunwayML-->>LiteLLM: 200 OK + task ID + + Note over LiteLLM: Automatic Polling + loop Every 2 seconds + LiteLLM->>RunwayML: GET v1/tasks/{task_id} + RunwayML-->>LiteLLM: Status: RUNNING + end + + LiteLLM->>RunwayML: GET v1/tasks/{task_id} + RunwayML-->>LiteLLM: Status: SUCCEEDED + audio URL + + LiteLLM->>Storage: GET audio URL + Storage-->>LiteLLM: Audio data (MP3) + + Note over LiteLLM: Return audio content + LiteLLM-->>Client: Audio Response (binary) +``` + diff --git a/litellm/llms/runwayml/text_to_speech/__init__.py b/litellm/llms/runwayml/text_to_speech/__init__.py new file mode 100644 index 00000000000..491e8449e0a --- /dev/null +++ b/litellm/llms/runwayml/text_to_speech/__init__.py @@ -0,0 +1,5 @@ +"""RunwayML Text-to-Speech implementation.""" +from .transformation import RunwayMLTextToSpeechConfig + +__all__ = ["RunwayMLTextToSpeechConfig"] + diff --git a/litellm/llms/runwayml/text_to_speech/transformation.py b/litellm/llms/runwayml/text_to_speech/transformation.py new file mode 100644 index 00000000000..ac926beb227 --- /dev/null +++ b/litellm/llms/runwayml/text_to_speech/transformation.py @@ -0,0 +1,591 @@ +""" +RunwayML Text-to-Speech transformation + +Maps OpenAI TTS spec to RunwayML Text-to-Speech API +""" +import asyncio +import time +from typing import TYPE_CHECKING, Any, Coroutine, Dict, Optional, Tuple, Union + +import httpx + +import litellm +from litellm._logging import verbose_logger +from litellm.constants import ( + RUNWAYML_DEFAULT_API_VERSION, + RUNWAYML_POLLING_TIMEOUT, +) +from litellm.llms.base_llm.text_to_speech.transformation import ( + BaseTextToSpeechConfig, + TextToSpeechRequestData, +) +from litellm.secret_managers.main import get_secret_str + +if TYPE_CHECKING: + from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj + from litellm.types.llms.openai import HttpxBinaryResponseContent +else: + LiteLLMLoggingObj = Any + HttpxBinaryResponseContent = Any + + +class RunwayMLTextToSpeechConfig(BaseTextToSpeechConfig): + """ + Configuration for RunwayML Text-to-Speech + + Reference: https://api.dev.runwayml.com/v1/text_to_speech + """ + + DEFAULT_BASE_URL: str = "https://api.dev.runwayml.com" + TTS_ENDPOINT_PATH: str = "v1/text_to_speech" + DEFAULT_MODEL: str = "eleven_multilingual_v2" + DEFAULT_VOICE_TYPE: str = "runway-preset" + DEFAULT_VOICE_PRESET_ID: str = "Bernard" + + # Voice mappings from OpenAI voices to RunwayML preset IDs + # OpenAI voices mapped to similar-sounding RunwayML voices + VOICE_MAPPINGS = { + "alloy": "Maya", # Neutral, balanced female voice + "echo": "James", # Male voice + "fable": "Bernard", # Warm, storytelling voice + "onyx": "Vincent", # Deep male voice + "nova": "Serene", # Warm, expressive female voice + "shimmer": "Ella", # Clear, friendly female voice + } + + def dispatch_text_to_speech( + self, + model: str, + input: str, + voice: Optional[Union[str, Dict]], + optional_params: Dict, + litellm_params_dict: Dict, + logging_obj: "LiteLLMLoggingObj", + timeout: Union[float, httpx.Timeout], + extra_headers: Optional[Dict[str, Any]], + base_llm_http_handler: Any, + aspeech: bool, + api_base: Optional[str], + api_key: Optional[str], + **kwargs: Any, + ) -> Union[ + "HttpxBinaryResponseContent", + Coroutine[Any, Any, "HttpxBinaryResponseContent"], + ]: + """ + Dispatch method to handle RunwayML TTS requests + + This method encapsulates RunwayML-specific credential resolution and parameter handling + + Args: + base_llm_http_handler: The BaseLLMHTTPHandler instance from main.py + """ + # Resolve api_base from multiple sources + api_base = ( + api_base + or litellm_params_dict.get("api_base") + or litellm.api_base + or get_secret_str("RUNWAYML_API_BASE") + or self.DEFAULT_BASE_URL + ) + + # Resolve api_key from multiple sources + api_key = ( + api_key + or litellm_params_dict.get("api_key") + or litellm.api_key + or get_secret_str("RUNWAYML_API_SECRET") + or get_secret_str("RUNWAYML_API_KEY") + ) + + # Convert voice to appropriate format + voice_param: Optional[Union[str, Dict]] = voice + if isinstance(voice, str): + # Keep as string, will be processed in map_openai_params + voice_param = voice + elif isinstance(voice, dict): + # Already in dict format, pass through + voice_param = voice + + litellm_params_dict.update({ + "api_key": api_key, + "api_base": api_base, + }) + + # Call the text_to_speech_handler + response = base_llm_http_handler.text_to_speech_handler( + model=model, + input=input, + voice=voice_param, + text_to_speech_provider_config=self, + text_to_speech_optional_params=optional_params, + custom_llm_provider="runwayml", + litellm_params=litellm_params_dict, + logging_obj=logging_obj, + timeout=timeout, + extra_headers=extra_headers, + client=None, + _is_async=aspeech, + ) + + return response + + def get_supported_openai_params(self, model: str) -> list: + """ + RunwayML TTS supports these OpenAI parameters + """ + return ["voice"] + + def map_openai_params( + self, + model: str, + optional_params: Dict, + voice: Optional[Union[str, Dict]] = None, + drop_params: bool = False, + kwargs: Dict = {}, + ) -> Tuple[Optional[str], Dict]: + """ + Map OpenAI parameters to RunwayML TTS parameters + + Returns: + Tuple of (mapped_voice_string, mapped_params) + + Note: Since RunwayML requires voice as a dict, we store it in + mapped_params["runwayml_voice"] and return None for the voice string. + """ + mapped_params = {} + + # Map voice parameter to RunwayML format dict + voice_dict: Optional[Dict] = None + if isinstance(voice, str): + # Check if it's an OpenAI voice name that needs mapping + if voice in self.VOICE_MAPPINGS: + preset_id = self.VOICE_MAPPINGS[voice] + voice_dict = { + "type": self.DEFAULT_VOICE_TYPE, + "presetId": preset_id, + } + else: + # Assume it's a RunwayML preset ID + voice_dict = { + "type": self.DEFAULT_VOICE_TYPE, + "presetId": voice, + } + elif isinstance(voice, dict): + # Already in RunwayML format, use as-is + voice_dict = voice + + # Store the voice dict in optional_params for later use + if voice_dict is not None: + mapped_params["runwayml_voice"] = voice_dict + + # No other OpenAI params are currently supported by RunwayML TTS + # (response_format, speed, etc. are not supported) + + # Return None for voice string since RunwayML uses dict format + return None, mapped_params + + def validate_environment( + self, + headers: dict, + model: str, + api_key: Optional[str] = None, + api_base: Optional[str] = None, + ) -> dict: + """ + Validate RunwayML environment and set up authentication headers + """ + validated_headers = headers.copy() + + final_api_key = ( + api_key + or get_secret_str("RUNWAYML_API_SECRET") + or get_secret_str("RUNWAYML_API_KEY") + ) + + if not final_api_key: + raise ValueError("RUNWAYML_API_SECRET or RUNWAYML_API_KEY is not set") + + validated_headers["Authorization"] = f"Bearer {final_api_key}" + validated_headers["X-Runway-Version"] = RUNWAYML_DEFAULT_API_VERSION + validated_headers["Content-Type"] = "application/json" + + return validated_headers + + def get_complete_url( + self, + model: str, + api_base: Optional[str], + litellm_params: dict, + ) -> str: + """ + Get the complete URL for RunwayML TTS request + """ + complete_url = ( + api_base + or get_secret_str("RUNWAYML_API_BASE") + or self.DEFAULT_BASE_URL + ) + + complete_url = complete_url.rstrip("/") + return f"{complete_url}/{self.TTS_ENDPOINT_PATH}" + + @staticmethod + def _check_timeout(start_time: float, timeout_secs: float) -> None: + """ + Check if operation has timed out. + + Args: + start_time: Start time of the operation + timeout_secs: Timeout duration in seconds + + Raises: + TimeoutError: If operation has exceeded timeout + """ + if time.time() - start_time > timeout_secs: + raise TimeoutError( + f"RunwayML TTS task polling timed out after {timeout_secs} seconds" + ) + + @staticmethod + def _check_task_status(response_data: Dict[str, Any]) -> str: + """ + Check RunwayML task status from response. + + RunwayML statuses: PENDING, RUNNING, SUCCEEDED, FAILED, CANCELLED, THROTTLED + + Args: + response_data: JSON response from RunwayML task endpoint + + Returns: + Normalized status string: "running", "succeeded", or raises on failure + + Raises: + ValueError: If task failed or status is unknown + """ + status = response_data.get("status", "").upper() + + verbose_logger.debug(f"RunwayML TTS task status: {status}") + + if status == "SUCCEEDED": + return "succeeded" + elif status == "FAILED": + failure_reason = response_data.get("failure", "Unknown error") + failure_code = response_data.get("failureCode", "unknown") + raise ValueError( + f"RunwayML TTS failed: {failure_reason} (code: {failure_code})" + ) + elif status == "CANCELLED": + raise ValueError("RunwayML TTS was cancelled") + elif status in ["PENDING", "RUNNING", "THROTTLED"]: + return "running" + else: + raise ValueError(f"Unknown RunwayML task status: {status}") + + def _poll_task_sync( + self, + task_id: str, + api_base: str, + headers: Dict[str, str], + timeout_secs: float = 600, + ) -> httpx.Response: + """ + Poll RunwayML task until completion (sync). + + RunwayML POST returns immediately with a task that has status PENDING/RUNNING. + We need to poll GET /v1/tasks/{task_id} until status is SUCCEEDED or FAILED. + + Args: + task_id: The task ID to poll + api_base: Base URL for RunwayML API + headers: Request headers (including auth) + timeout_secs: Total timeout in seconds (default: 600s = 10 minutes) + + Returns: + Final response with completed task + """ + from litellm.llms.custom_httpx.http_handler import _get_httpx_client + + client = _get_httpx_client() + start_time = time.time() + + # Build task status URL + api_base = api_base.rstrip("/") + task_url = f"{api_base}/v1/tasks/{task_id}" + + verbose_logger.debug(f"Polling RunwayML TTS task: {task_url}") + + while True: + self._check_timeout(start_time=start_time, timeout_secs=timeout_secs) + + # Poll the task status + response = client.get(url=task_url, headers=headers) + response.raise_for_status() + + response_data = response.json() + + # Check task status + status = self._check_task_status(response_data=response_data) + + if status == "succeeded": + return response + elif status == "running": + # Wait before polling again (RunwayML recommends 1-2 second intervals) + time.sleep(2) + + async def _poll_task_async( + self, + task_id: str, + api_base: str, + headers: Dict[str, str], + timeout_secs: float = 600, + ) -> httpx.Response: + """ + Poll RunwayML task until completion (async). + + Args: + task_id: The task ID to poll + api_base: Base URL for RunwayML API + headers: Request headers (including auth) + timeout_secs: Total timeout in seconds (default: 600s = 10 minutes) + + Returns: + Final response with completed task + """ + from litellm.llms.custom_httpx.http_handler import get_async_httpx_client + + client = get_async_httpx_client(llm_provider=litellm.LlmProviders.RUNWAYML) + start_time = time.time() + + # Build task status URL + api_base = api_base.rstrip("/") + task_url = f"{api_base}/v1/tasks/{task_id}" + + verbose_logger.debug(f"Polling RunwayML TTS task (async): {task_url}") + + while True: + self._check_timeout(start_time=start_time, timeout_secs=timeout_secs) + + # Poll the task status + response = await client.get(url=task_url, headers=headers) + response.raise_for_status() + + response_data = response.json() + + # Check task status + status = self._check_task_status(response_data=response_data) + + if status == "succeeded": + return response + elif status == "running": + # Wait before polling again (RunwayML recommends 1-2 second intervals) + await asyncio.sleep(2) + + def transform_text_to_speech_request( + self, + model: str, + input: str, + voice: Optional[Union[str, Dict]], + optional_params: Dict, + litellm_params: Dict, + headers: dict, + ) -> TextToSpeechRequestData: + """ + Transform OpenAI TTS request to RunwayML TTS format + + RunwayML expects: + - model: The model to use (e.g., 'eleven_multilingual_v2') + - promptText: The text to convert to speech + - voice: Voice configuration object + { + "type": "runway-preset", + "presetId": "Bernard" + } + + Returns: + TextToSpeechRequestData: Contains JSON body and headers + """ + # Get voice from optional_params (mapped in map_openai_params) + runwayml_voice = optional_params.get("runwayml_voice") + if runwayml_voice is None: + # Use default voice if not provided + runwayml_voice = { + "type": self.DEFAULT_VOICE_TYPE, + "presetId": self.DEFAULT_VOICE_PRESET_ID, + } + + # Build request body + request_body = { + "model": model or self.DEFAULT_MODEL, + "promptText": input, + "voice": runwayml_voice, + } + + # Add any other optional parameters (except runwayml_voice which we already used) + for k, v in optional_params.items(): + if k not in request_body and k != "runwayml_voice": + request_body[k] = v + + return { + "dict_body": request_body, + "headers": headers, + } + + def transform_text_to_speech_response( + self, + model: str, + raw_response: httpx.Response, + logging_obj: "LiteLLMLoggingObj", + ) -> "HttpxBinaryResponseContent": + """ + Transform RunwayML TTS response to standard format + + RunwayML returns a task immediately with status PENDING/RUNNING. + We need to poll the task until it completes, then download the audio. + + Initial response: + { + "id": "task_123...", + "status": "PENDING" | "RUNNING", + "createdAt": "2025-11-13T..." + } + + After polling: + { + "id": "task_123...", + "status": "SUCCEEDED", + "output": ["https://storage.googleapis.com/.../audio.mp3"], + "completedAt": "2025-11-13T..." + } + """ + from litellm.types.llms.openai import HttpxBinaryResponseContent + + try: + response_data = raw_response.json() + except Exception as e: + raise self.get_error_class( + error_message=f"Error parsing RunwayML TTS response: {e}", + status_code=raw_response.status_code, + headers=dict(raw_response.headers), + ) + + verbose_logger.debug("RunwayML TTS starting polling...") + + # Get task ID + task_id = response_data.get("id") + if not task_id: + raise ValueError("RunwayML TTS response missing task ID") + + # Get headers for polling (need auth) + poll_headers = { + "Authorization": raw_response.request.headers.get("Authorization", ""), + "X-Runway-Version": raw_response.request.headers.get( + "X-Runway-Version", RUNWAYML_DEFAULT_API_VERSION + ), + } + + # Poll until task completes + polled_response = self._poll_task_sync( + task_id=task_id, + api_base=self.DEFAULT_BASE_URL, + headers=poll_headers, + timeout_secs=RUNWAYML_POLLING_TIMEOUT, + ) + + # Get the completed task data + task_data = polled_response.json() + + verbose_logger.debug("RunwayML TTS polling complete, downloading audio") + + # Get audio URL from output + output = task_data.get("output", []) + if not output or not isinstance(output, list) or len(output) == 0: + raise ValueError("RunwayML TTS response missing audio URL in output") + + audio_url = output[0] + if not isinstance(audio_url, str): + raise ValueError(f"RunwayML TTS audio URL is not a string: {audio_url}") + + # Download the audio file + from litellm.llms.custom_httpx.http_handler import _get_httpx_client + + client = _get_httpx_client() + audio_response = client.get(url=audio_url) + audio_response.raise_for_status() + + verbose_logger.debug("RunwayML TTS audio downloaded successfully") + + # Return the audio data wrapped in HttpxBinaryResponseContent + return HttpxBinaryResponseContent(audio_response) + + async def async_transform_text_to_speech_response( + self, + model: str, + raw_response: httpx.Response, + logging_obj: "LiteLLMLoggingObj", + ) -> "HttpxBinaryResponseContent": + """ + Async transform RunwayML TTS response to standard format + + Same as sync version but uses async polling and download + """ + from litellm.types.llms.openai import HttpxBinaryResponseContent + + try: + response_data = raw_response.json() + except Exception as e: + raise self.get_error_class( + error_message=f"Error parsing RunwayML TTS response: {e}", + status_code=raw_response.status_code, + headers=dict(raw_response.headers), + ) + + verbose_logger.debug("RunwayML TTS starting polling (async)...") + + # Get task ID + task_id = response_data.get("id") + if not task_id: + raise ValueError("RunwayML TTS response missing task ID") + + # Get headers for polling (need auth) + poll_headers = { + "Authorization": raw_response.request.headers.get("Authorization", ""), + "X-Runway-Version": raw_response.request.headers.get( + "X-Runway-Version", RUNWAYML_DEFAULT_API_VERSION + ), + } + + # Poll until task completes (async) + polled_response = await self._poll_task_async( + task_id=task_id, + api_base=self.DEFAULT_BASE_URL, + headers=poll_headers, + timeout_secs=RUNWAYML_POLLING_TIMEOUT, + ) + + # Get the completed task data + task_data = polled_response.json() + + verbose_logger.debug("RunwayML TTS polling complete (async), downloading audio") + + # Get audio URL from output + output = task_data.get("output", []) + if not output or not isinstance(output, list) or len(output) == 0: + raise ValueError("RunwayML TTS response missing audio URL in output") + + audio_url = output[0] + if not isinstance(audio_url, str): + raise ValueError(f"RunwayML TTS audio URL is not a string: {audio_url}") + + # Download the audio file (async) + from litellm.llms.custom_httpx.http_handler import get_async_httpx_client + + client = get_async_httpx_client(llm_provider=litellm.LlmProviders.RUNWAYML) + audio_response = await client.get(url=audio_url) + audio_response.raise_for_status() + + verbose_logger.debug("RunwayML TTS audio downloaded successfully (async)") + + # Return the audio data wrapped in HttpxBinaryResponseContent + return HttpxBinaryResponseContent(audio_response) + diff --git a/litellm/llms/runwayml/videos/transformation.py b/litellm/llms/runwayml/videos/transformation.py index c45a1cd60d5..651acff6fc4 100644 --- a/litellm/llms/runwayml/videos/transformation.py +++ b/litellm/llms/runwayml/videos/transformation.py @@ -6,6 +6,8 @@ from httpx._types import RequestFiles import litellm from litellm.constants import RUNWAYML_DEFAULT_API_VERSION +from litellm.llms.base_llm.chat.transformation import BaseLLMException +from litellm.llms.base_llm.videos.transformation import BaseVideoConfig from litellm.llms.custom_httpx.http_handler import ( AsyncHTTPHandler, HTTPHandler, @@ -23,16 +25,9 @@ from litellm.types.videos.utils import ( if TYPE_CHECKING: from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj - from ...base_llm.chat.transformation import BaseLLMException as _BaseLLMException - from ...base_llm.videos.transformation import BaseVideoConfig as _BaseVideoConfig - LiteLLMLoggingObj = _LiteLLMLoggingObj - BaseVideoConfig = _BaseVideoConfig - BaseLLMException = _BaseLLMException else: LiteLLMLoggingObj = Any - BaseVideoConfig = Any - BaseLLMException = Any class RunwayMLVideoConfig(BaseVideoConfig): diff --git a/litellm/main.py b/litellm/main.py index 2ad444a9a20..23ffa90e2db 100644 --- a/litellm/main.py +++ b/litellm/main.py @@ -6006,6 +6006,39 @@ def speech( # noqa: PLR0915 logging_obj=logging_obj, custom_llm_provider=custom_llm_provider, ) + elif custom_llm_provider == "runwayml": + from litellm.llms.runwayml.text_to_speech.transformation import ( + RunwayMLTextToSpeechConfig, + ) + + # RunwayML Text-to-Speech + if text_to_speech_provider_config is None: + raise litellm.BadRequestError( + message="RunwayML Text-to-Speech configuration not found", + model=model, + llm_provider=custom_llm_provider, + ) + + # Cast to specific RunwayML config type to access dispatch method + runwayml_config = cast( + RunwayMLTextToSpeechConfig, text_to_speech_provider_config + ) + + response = runwayml_config.dispatch_text_to_speech( # type: ignore + model=model, + input=input, + voice=voice, + optional_params=optional_params, + litellm_params_dict=litellm_params_dict, + logging_obj=logging_obj, + timeout=timeout, + extra_headers=extra_headers, + base_llm_http_handler=base_llm_http_handler, + aspeech=aspeech or False, + api_base=api_base, + api_key=api_key, + **kwargs, + ) if response is None: raise Exception( diff --git a/litellm/utils.py b/litellm/utils.py index 7f87a800b07..ae3de67374f 100644 --- a/litellm/utils.py +++ b/litellm/utils.py @@ -7811,6 +7811,12 @@ class ProviderConfigManager: ) return AzureAVATextToSpeechConfig() + elif litellm.LlmProviders.RUNWAYML == provider: + from litellm.llms.runwayml.text_to_speech.transformation import ( + RunwayMLTextToSpeechConfig, + ) + + return RunwayMLTextToSpeechConfig() return None @staticmethod diff --git a/provider_endpoints_support.json b/provider_endpoints_support.json index 6c3be897f40..4d9218d6095 100644 --- a/provider_endpoints_support.json +++ b/provider_endpoints_support.json @@ -1368,7 +1368,7 @@ "embeddings": false, "image_generations": true, "audio_transcriptions": false, - "audio_speech": false, + "audio_speech": true, "moderations": false, "batches": false, "rerank": false, diff --git a/tests/audio_tests/runwayml_speech.mp3 b/tests/audio_tests/runwayml_speech.mp3 new file mode 100644 index 00000000000..22aee2084ba Binary files /dev/null and b/tests/audio_tests/runwayml_speech.mp3 differ diff --git a/tests/audio_tests/test_audio_speech.py b/tests/audio_tests/test_audio_speech.py index bf886a8d5cd..409e3544fcf 100644 --- a/tests/audio_tests/test_audio_speech.py +++ b/tests/audio_tests/test_audio_speech.py @@ -382,6 +382,60 @@ async def test_azure_ava_tts_async(): pytest.fail(f"Test failed with exception: {str(e)}") +@pytest.mark.asyncio +@pytest.mark.flaky(retries=3, delay=1) +async def test_runwayml_tts_async(): + """ + Test RunwayML Text-to-Speech with real API request. + """ + litellm._turn_on_debug() + api_key = os.getenv("RUNWAYML_API_KEY") + api_base = os.getenv("RUNWAYML_API_BASE") + + + speech_file_path = Path(__file__).parent / "runwayml_speech.mp3" + + try: + response = await litellm.aspeech( + model="runwayml/eleven_multilingual_v2", + voice="Rachel", + input="Yuneng is gone, we miss him so much I hope he has a good coffee", + api_base=api_base, + api_key=api_key, + response_format="mp3", + speed=1.0, + ) + + # Assert the response is HttpxBinaryResponseContent + from litellm.types.llms.openai import HttpxBinaryResponseContent + + assert isinstance(response, HttpxBinaryResponseContent) + + # Get the binary content + binary_content = response.content + assert len(binary_content) > 0 + + # MP3 files start with these magic bytes + # ID3 tag or MPEG sync word + assert binary_content[:3] == b"ID3" or binary_content[:2] == b"\xff\xfb" or binary_content[:2] == b"\xff\xf3" + + # Write to file + response.stream_to_file(speech_file_path) + + # Verify file was created and has content + assert speech_file_path.exists() + assert speech_file_path.stat().st_size > 0 + + print(f"Azure TTS audio saved to: {speech_file_path}") + + # assert response cost is greater than 0 + print("Response cost: ", response._hidden_params["response_cost"]) + assert response._hidden_params["response_cost"] > 0 + + except Exception as e: + pytest.fail(f"Test failed with exception: {str(e)}") + + @pytest.mark.asyncio async def test_azure_ava_tts_with_custom_voice(): """ diff --git a/tests/test_litellm/llms/runwayml/test_text_to_speech_transformation.py b/tests/test_litellm/llms/runwayml/test_text_to_speech_transformation.py new file mode 100644 index 00000000000..277a47a03bc --- /dev/null +++ b/tests/test_litellm/llms/runwayml/test_text_to_speech_transformation.py @@ -0,0 +1,67 @@ +""" +Test RunwayML text-to-speech transformation +""" +import os +import sys + +sys.path.insert(0, os.path.abspath("../../../..")) + +from litellm.llms.runwayml.text_to_speech.transformation import ( + RunwayMLTextToSpeechConfig, +) + + +def test_openai_voice_mapping_to_runwayml(): + """ + Test that OpenAI voice names are correctly mapped to RunwayML preset IDs + """ + config = RunwayMLTextToSpeechConfig() + + # Test OpenAI voice mappings + openai_to_runway = { + "alloy": "Maya", + "echo": "James", + "fable": "Bernard", + "onyx": "Vincent", + "nova": "Serene", + "shimmer": "Ella", + } + + for openai_voice, expected_runway_voice in openai_to_runway.items(): + mapped_voice, mapped_params = config.map_openai_params( + model="eleven_multilingual_v2", + optional_params={}, + voice=openai_voice, + drop_params=False, + kwargs={}, + ) + + assert mapped_voice is None + assert "runwayml_voice" in mapped_params + assert mapped_params["runwayml_voice"]["type"] == "runway-preset" + assert mapped_params["runwayml_voice"]["presetId"] == expected_runway_voice + + +def test_runwayml_native_voice_passthrough(): + """ + Test that RunwayML native voice names are passed through correctly as-is + """ + config = RunwayMLTextToSpeechConfig() + + # Test various RunwayML native voices + runway_voices = ["Bernard", "Maya", "Arjun", "Serene", "Chad"] + + for runway_voice in runway_voices: + mapped_voice, mapped_params = config.map_openai_params( + model="eleven_multilingual_v2", + optional_params={}, + voice=runway_voice, + drop_params=False, + kwargs={}, + ) + + assert mapped_voice is None + assert "runwayml_voice" in mapped_params + assert mapped_params["runwayml_voice"]["type"] == "runway-preset" + assert mapped_params["runwayml_voice"]["presetId"] == runway_voice +