mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
[Feat] RunwayML - Add support for /audio/speech eleven_multilingual_v2 endpoint (#16604)
* init RunwayMLTextToSpeechConfig * add RunwayMLTextToSpeechConfig * add RunwayMLTextToSpeechConfig * test_runwayml_tts_async * runway ml speech * fix voices * fix test * docs runway lm * add runwayml here * fix RunwayMLTextToSpeechConfig * test_openai_voice_mapping_to_runwayml
This commit is contained in:
Ishaan Jaff
GitHub
parent
4be372eb48
commit
124ba463f8
10 changed files with 1003 additions and 8 deletions
244
docs/my-website/docs/providers/runwayml/text-to-speech.md
Normal file
244
docs/my-website/docs/providers/runwayml/text-to-speech.md
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
# RunwayML - Text-to-Speech
|
||||
|
||||
## Overview
|
||||
|
||||
| Property | Details |
|
||||
|-------|-------|
|
||||
| Description | RunwayML provides high-quality AI-powered text-to-speech with natural-sounding voices |
|
||||
| Provider Route on LiteLLM | `runwayml/` |
|
||||
| Supported Operations | [`/audio/speech`](#quick-start) |
|
||||
| Link to Provider Doc | [RunwayML API ↗](https://docs.dev.runwayml.com/) |
|
||||
|
||||
LiteLLM supports RunwayML's text-to-speech API with automatic task polling, allowing you to generate natural-sounding audio from text.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python showLineNumbers title="Basic Text-to-Speech"
|
||||
from litellm import speech
|
||||
import os
|
||||
|
||||
os.environ["RUNWAYML_API_KEY"] = "your-api-key"
|
||||
|
||||
response = speech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
input="Step right up, ladies and gentlemen! Have you ever wished for a toaster that's not just a toaster but a marvel of modern ingenuity?",
|
||||
voice="alloy"
|
||||
)
|
||||
|
||||
# Save the audio
|
||||
with open("output.mp3", "wb") as f:
|
||||
f.write(response.content)
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
Set your RunwayML API key:
|
||||
|
||||
```python showLineNumbers title="Set API Key"
|
||||
import os
|
||||
|
||||
os.environ["RUNWAYML_API_KEY"] = "your-api-key"
|
||||
```
|
||||
|
||||
## Supported Parameters
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `model` | string | Yes | Model to use (e.g., `runwayml/eleven_multilingual_v2`) |
|
||||
| `input` | string | Yes | Text to convert to speech |
|
||||
| `voice` | string or dict | Yes | Voice to use (OpenAI name, RunwayML preset, or voice config) |
|
||||
|
||||
## Voice Options
|
||||
|
||||
### Using OpenAI Voice Names
|
||||
|
||||
OpenAI voice names are automatically mapped to appropriate RunwayML voices:
|
||||
|
||||
```python showLineNumbers title="OpenAI Voice Names"
|
||||
from litellm import speech
|
||||
|
||||
# These OpenAI voice names work automatically
|
||||
response = speech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
input="Hello, world!",
|
||||
voice="alloy" # Maya - neutral, balanced female voice
|
||||
)
|
||||
```
|
||||
|
||||
**Voice Mappings:**
|
||||
- `alloy` → Maya (neutral, balanced female voice)
|
||||
- `echo` → James (male voice)
|
||||
- `fable` → Bernard (warm, storytelling voice)
|
||||
- `onyx` → Vincent (deep male voice)
|
||||
- `nova` → Serene (warm, expressive female voice)
|
||||
- `shimmer` → Ella (clear, friendly female voice)
|
||||
|
||||
### Using RunwayML Preset Voices
|
||||
|
||||
You can directly specify any RunwayML preset voice by passing the preset name as a string:
|
||||
|
||||
```python showLineNumbers title="RunwayML Preset Names"
|
||||
from litellm import speech
|
||||
|
||||
# Pass the RunwayML voice name as a string
|
||||
response = speech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
input="Hello, world!",
|
||||
voice="Maya" # LiteLLM automatically formats this for RunwayML
|
||||
)
|
||||
|
||||
# Try different RunwayML voices
|
||||
response = speech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
input="Step right up, ladies and gentlemen!",
|
||||
voice="Bernard" # Great for storytelling
|
||||
)
|
||||
```
|
||||
|
||||
**Available RunwayML Voices:**
|
||||
|
||||
Maya, Arjun, Serene, Bernard, Billy, Mark, Clint, Mabel, Chad, Leslie, Eleanor, Elias, Elliot, Grungle, Brodie, Sandra, Kirk, Kylie, Lara, Lisa, Malachi, Marlene, Martin, Miriam, Monster, Paula, Pip, Rusty, Ragnar, Xylar, Maggie, Jack, Katie, Noah, James, Rina, Ella, Mariah, Frank, Claudia, Niki, Vincent, Kendrick, Myrna, Tom, Wanda, Benjamin, Kiana, Rachel
|
||||
|
||||
:::tip
|
||||
Simply pass the voice name as a string - LiteLLM automatically handles the internal RunwayML API format conversion.
|
||||
:::
|
||||
|
||||
## Async Usage
|
||||
|
||||
```python showLineNumbers title="Async Text-to-Speech"
|
||||
from litellm import aspeech
|
||||
import os
|
||||
import asyncio
|
||||
|
||||
os.environ["RUNWAYML_API_KEY"] = "your-api-key"
|
||||
|
||||
async def generate_speech():
|
||||
response = await aspeech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
input="This is an asynchronous text-to-speech request.",
|
||||
voice="nova"
|
||||
)
|
||||
|
||||
with open("output.mp3", "wb") as f:
|
||||
f.write(response.content)
|
||||
|
||||
print("Audio generated successfully!")
|
||||
|
||||
asyncio.run(generate_speech())
|
||||
```
|
||||
|
||||
## LiteLLM Proxy Usage
|
||||
|
||||
Add RunwayML to your proxy configuration:
|
||||
|
||||
```yaml showLineNumbers title="config.yaml"
|
||||
model_list:
|
||||
- model_name: runway-tts
|
||||
litellm_params:
|
||||
model: runwayml/eleven_multilingual_v2
|
||||
api_key: os.environ/RUNWAYML_API_KEY
|
||||
```
|
||||
|
||||
Start the proxy:
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
Generate speech through the proxy:
|
||||
|
||||
```bash showLineNumbers title="Proxy Request"
|
||||
curl --location 'http://localhost:4000/v1/audio/speech' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'x-litellm-api-key: sk-1234' \
|
||||
--data '{
|
||||
"model": "runwayml/eleven_multilingual_v2",
|
||||
"input": "Hello from the LiteLLM proxy!",
|
||||
"voice": "alloy"
|
||||
}'
|
||||
```
|
||||
|
||||
With RunwayML-specific voice:
|
||||
|
||||
```bash showLineNumbers title="Proxy Request with RunwayML Voice"
|
||||
curl --location 'http://localhost:4000/v1/audio/speech' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'x-litellm-api-key: sk-1234' \
|
||||
--data '{
|
||||
"model": "runwayml/eleven_multilingual_v2",
|
||||
"input": "Hello with a custom RunwayML voice!",
|
||||
"voice": "Bernard"
|
||||
}'
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
| Model | Description |
|
||||
|-------|-------------|
|
||||
| `runwayml/eleven_multilingual_v2` | High-quality multilingual text-to-speech |
|
||||
|
||||
## Cost Tracking
|
||||
|
||||
LiteLLM automatically tracks RunwayML text-to-speech costs:
|
||||
|
||||
```python showLineNumbers title="Cost Tracking"
|
||||
from litellm import speech, completion_cost
|
||||
|
||||
response = speech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
input="Hello, world!",
|
||||
voice="alloy"
|
||||
)
|
||||
|
||||
cost = completion_cost(completion_response=response)
|
||||
print(f"Text-to-speech cost: ${cost}")
|
||||
```
|
||||
|
||||
## Supported Features
|
||||
|
||||
| Feature | Supported |
|
||||
|---------|-----------|
|
||||
| Text-to-Speech | ✅ |
|
||||
| Cost Tracking | ✅ |
|
||||
| Logging | ✅ |
|
||||
| Fallbacks | ✅ |
|
||||
| Load Balancing | ✅ |
|
||||
| 50+ Voice Presets | ✅ |
|
||||
|
||||
## How It Works
|
||||
|
||||
RunwayML uses an asynchronous task-based API pattern. LiteLLM handles the polling and response transformation automatically.
|
||||
|
||||
### Complete Flow Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
box rgb(200, 220, 255) LiteLLM AI Gateway
|
||||
participant LiteLLM
|
||||
end
|
||||
participant RunwayML as RunwayML API
|
||||
participant Storage as Audio Storage
|
||||
|
||||
Client->>LiteLLM: POST /audio/speech (OpenAI format)
|
||||
Note over LiteLLM: Transform to RunwayML format<br/>Map voice to preset ID
|
||||
|
||||
LiteLLM->>RunwayML: POST v1/text_to_speech
|
||||
RunwayML-->>LiteLLM: 200 OK + task ID
|
||||
|
||||
Note over LiteLLM: Automatic Polling
|
||||
loop Every 2 seconds
|
||||
LiteLLM->>RunwayML: GET v1/tasks/{task_id}
|
||||
RunwayML-->>LiteLLM: Status: RUNNING
|
||||
end
|
||||
|
||||
LiteLLM->>RunwayML: GET v1/tasks/{task_id}
|
||||
RunwayML-->>LiteLLM: Status: SUCCEEDED + audio URL
|
||||
|
||||
LiteLLM->>Storage: GET audio URL
|
||||
Storage-->>LiteLLM: Audio data (MP3)
|
||||
|
||||
Note over LiteLLM: Return audio content
|
||||
LiteLLM-->>Client: Audio Response (binary)
|
||||
```
|
||||
|
||||
5
litellm/llms/runwayml/text_to_speech/__init__.py
Normal file
5
litellm/llms/runwayml/text_to_speech/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""RunwayML Text-to-Speech implementation."""
|
||||
from .transformation import RunwayMLTextToSpeechConfig
|
||||
|
||||
__all__ = ["RunwayMLTextToSpeechConfig"]
|
||||
|
||||
591
litellm/llms/runwayml/text_to_speech/transformation.py
Normal file
591
litellm/llms/runwayml/text_to_speech/transformation.py
Normal file
|
|
@ -0,0 +1,591 @@
|
|||
"""
|
||||
RunwayML Text-to-Speech transformation
|
||||
|
||||
Maps OpenAI TTS spec to RunwayML Text-to-Speech API
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any, Coroutine, Dict, Optional, Tuple, Union
|
||||
|
||||
import httpx
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.constants import (
|
||||
RUNWAYML_DEFAULT_API_VERSION,
|
||||
RUNWAYML_POLLING_TIMEOUT,
|
||||
)
|
||||
from litellm.llms.base_llm.text_to_speech.transformation import (
|
||||
BaseTextToSpeechConfig,
|
||||
TextToSpeechRequestData,
|
||||
)
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
|
||||
from litellm.types.llms.openai import HttpxBinaryResponseContent
|
||||
else:
|
||||
LiteLLMLoggingObj = Any
|
||||
HttpxBinaryResponseContent = Any
|
||||
|
||||
|
||||
class RunwayMLTextToSpeechConfig(BaseTextToSpeechConfig):
|
||||
"""
|
||||
Configuration for RunwayML Text-to-Speech
|
||||
|
||||
Reference: https://api.dev.runwayml.com/v1/text_to_speech
|
||||
"""
|
||||
|
||||
DEFAULT_BASE_URL: str = "https://api.dev.runwayml.com"
|
||||
TTS_ENDPOINT_PATH: str = "v1/text_to_speech"
|
||||
DEFAULT_MODEL: str = "eleven_multilingual_v2"
|
||||
DEFAULT_VOICE_TYPE: str = "runway-preset"
|
||||
DEFAULT_VOICE_PRESET_ID: str = "Bernard"
|
||||
|
||||
# Voice mappings from OpenAI voices to RunwayML preset IDs
|
||||
# OpenAI voices mapped to similar-sounding RunwayML voices
|
||||
VOICE_MAPPINGS = {
|
||||
"alloy": "Maya", # Neutral, balanced female voice
|
||||
"echo": "James", # Male voice
|
||||
"fable": "Bernard", # Warm, storytelling voice
|
||||
"onyx": "Vincent", # Deep male voice
|
||||
"nova": "Serene", # Warm, expressive female voice
|
||||
"shimmer": "Ella", # Clear, friendly female voice
|
||||
}
|
||||
|
||||
def dispatch_text_to_speech(
|
||||
self,
|
||||
model: str,
|
||||
input: str,
|
||||
voice: Optional[Union[str, Dict]],
|
||||
optional_params: Dict,
|
||||
litellm_params_dict: Dict,
|
||||
logging_obj: "LiteLLMLoggingObj",
|
||||
timeout: Union[float, httpx.Timeout],
|
||||
extra_headers: Optional[Dict[str, Any]],
|
||||
base_llm_http_handler: Any,
|
||||
aspeech: bool,
|
||||
api_base: Optional[str],
|
||||
api_key: Optional[str],
|
||||
**kwargs: Any,
|
||||
) -> Union[
|
||||
"HttpxBinaryResponseContent",
|
||||
Coroutine[Any, Any, "HttpxBinaryResponseContent"],
|
||||
]:
|
||||
"""
|
||||
Dispatch method to handle RunwayML TTS requests
|
||||
|
||||
This method encapsulates RunwayML-specific credential resolution and parameter handling
|
||||
|
||||
Args:
|
||||
base_llm_http_handler: The BaseLLMHTTPHandler instance from main.py
|
||||
"""
|
||||
# Resolve api_base from multiple sources
|
||||
api_base = (
|
||||
api_base
|
||||
or litellm_params_dict.get("api_base")
|
||||
or litellm.api_base
|
||||
or get_secret_str("RUNWAYML_API_BASE")
|
||||
or self.DEFAULT_BASE_URL
|
||||
)
|
||||
|
||||
# Resolve api_key from multiple sources
|
||||
api_key = (
|
||||
api_key
|
||||
or litellm_params_dict.get("api_key")
|
||||
or litellm.api_key
|
||||
or get_secret_str("RUNWAYML_API_SECRET")
|
||||
or get_secret_str("RUNWAYML_API_KEY")
|
||||
)
|
||||
|
||||
# Convert voice to appropriate format
|
||||
voice_param: Optional[Union[str, Dict]] = voice
|
||||
if isinstance(voice, str):
|
||||
# Keep as string, will be processed in map_openai_params
|
||||
voice_param = voice
|
||||
elif isinstance(voice, dict):
|
||||
# Already in dict format, pass through
|
||||
voice_param = voice
|
||||
|
||||
litellm_params_dict.update({
|
||||
"api_key": api_key,
|
||||
"api_base": api_base,
|
||||
})
|
||||
|
||||
# Call the text_to_speech_handler
|
||||
response = base_llm_http_handler.text_to_speech_handler(
|
||||
model=model,
|
||||
input=input,
|
||||
voice=voice_param,
|
||||
text_to_speech_provider_config=self,
|
||||
text_to_speech_optional_params=optional_params,
|
||||
custom_llm_provider="runwayml",
|
||||
litellm_params=litellm_params_dict,
|
||||
logging_obj=logging_obj,
|
||||
timeout=timeout,
|
||||
extra_headers=extra_headers,
|
||||
client=None,
|
||||
_is_async=aspeech,
|
||||
)
|
||||
|
||||
return response
|
||||
|
||||
def get_supported_openai_params(self, model: str) -> list:
|
||||
"""
|
||||
RunwayML TTS supports these OpenAI parameters
|
||||
"""
|
||||
return ["voice"]
|
||||
|
||||
def map_openai_params(
|
||||
self,
|
||||
model: str,
|
||||
optional_params: Dict,
|
||||
voice: Optional[Union[str, Dict]] = None,
|
||||
drop_params: bool = False,
|
||||
kwargs: Dict = {},
|
||||
) -> Tuple[Optional[str], Dict]:
|
||||
"""
|
||||
Map OpenAI parameters to RunwayML TTS parameters
|
||||
|
||||
Returns:
|
||||
Tuple of (mapped_voice_string, mapped_params)
|
||||
|
||||
Note: Since RunwayML requires voice as a dict, we store it in
|
||||
mapped_params["runwayml_voice"] and return None for the voice string.
|
||||
"""
|
||||
mapped_params = {}
|
||||
|
||||
# Map voice parameter to RunwayML format dict
|
||||
voice_dict: Optional[Dict] = None
|
||||
if isinstance(voice, str):
|
||||
# Check if it's an OpenAI voice name that needs mapping
|
||||
if voice in self.VOICE_MAPPINGS:
|
||||
preset_id = self.VOICE_MAPPINGS[voice]
|
||||
voice_dict = {
|
||||
"type": self.DEFAULT_VOICE_TYPE,
|
||||
"presetId": preset_id,
|
||||
}
|
||||
else:
|
||||
# Assume it's a RunwayML preset ID
|
||||
voice_dict = {
|
||||
"type": self.DEFAULT_VOICE_TYPE,
|
||||
"presetId": voice,
|
||||
}
|
||||
elif isinstance(voice, dict):
|
||||
# Already in RunwayML format, use as-is
|
||||
voice_dict = voice
|
||||
|
||||
# Store the voice dict in optional_params for later use
|
||||
if voice_dict is not None:
|
||||
mapped_params["runwayml_voice"] = voice_dict
|
||||
|
||||
# No other OpenAI params are currently supported by RunwayML TTS
|
||||
# (response_format, speed, etc. are not supported)
|
||||
|
||||
# Return None for voice string since RunwayML uses dict format
|
||||
return None, mapped_params
|
||||
|
||||
def validate_environment(
|
||||
self,
|
||||
headers: dict,
|
||||
model: str,
|
||||
api_key: Optional[str] = None,
|
||||
api_base: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""
|
||||
Validate RunwayML environment and set up authentication headers
|
||||
"""
|
||||
validated_headers = headers.copy()
|
||||
|
||||
final_api_key = (
|
||||
api_key
|
||||
or get_secret_str("RUNWAYML_API_SECRET")
|
||||
or get_secret_str("RUNWAYML_API_KEY")
|
||||
)
|
||||
|
||||
if not final_api_key:
|
||||
raise ValueError("RUNWAYML_API_SECRET or RUNWAYML_API_KEY is not set")
|
||||
|
||||
validated_headers["Authorization"] = f"Bearer {final_api_key}"
|
||||
validated_headers["X-Runway-Version"] = RUNWAYML_DEFAULT_API_VERSION
|
||||
validated_headers["Content-Type"] = "application/json"
|
||||
|
||||
return validated_headers
|
||||
|
||||
def get_complete_url(
|
||||
self,
|
||||
model: str,
|
||||
api_base: Optional[str],
|
||||
litellm_params: dict,
|
||||
) -> str:
|
||||
"""
|
||||
Get the complete URL for RunwayML TTS request
|
||||
"""
|
||||
complete_url = (
|
||||
api_base
|
||||
or get_secret_str("RUNWAYML_API_BASE")
|
||||
or self.DEFAULT_BASE_URL
|
||||
)
|
||||
|
||||
complete_url = complete_url.rstrip("/")
|
||||
return f"{complete_url}/{self.TTS_ENDPOINT_PATH}"
|
||||
|
||||
@staticmethod
|
||||
def _check_timeout(start_time: float, timeout_secs: float) -> None:
|
||||
"""
|
||||
Check if operation has timed out.
|
||||
|
||||
Args:
|
||||
start_time: Start time of the operation
|
||||
timeout_secs: Timeout duration in seconds
|
||||
|
||||
Raises:
|
||||
TimeoutError: If operation has exceeded timeout
|
||||
"""
|
||||
if time.time() - start_time > timeout_secs:
|
||||
raise TimeoutError(
|
||||
f"RunwayML TTS task polling timed out after {timeout_secs} seconds"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _check_task_status(response_data: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Check RunwayML task status from response.
|
||||
|
||||
RunwayML statuses: PENDING, RUNNING, SUCCEEDED, FAILED, CANCELLED, THROTTLED
|
||||
|
||||
Args:
|
||||
response_data: JSON response from RunwayML task endpoint
|
||||
|
||||
Returns:
|
||||
Normalized status string: "running", "succeeded", or raises on failure
|
||||
|
||||
Raises:
|
||||
ValueError: If task failed or status is unknown
|
||||
"""
|
||||
status = response_data.get("status", "").upper()
|
||||
|
||||
verbose_logger.debug(f"RunwayML TTS task status: {status}")
|
||||
|
||||
if status == "SUCCEEDED":
|
||||
return "succeeded"
|
||||
elif status == "FAILED":
|
||||
failure_reason = response_data.get("failure", "Unknown error")
|
||||
failure_code = response_data.get("failureCode", "unknown")
|
||||
raise ValueError(
|
||||
f"RunwayML TTS failed: {failure_reason} (code: {failure_code})"
|
||||
)
|
||||
elif status == "CANCELLED":
|
||||
raise ValueError("RunwayML TTS was cancelled")
|
||||
elif status in ["PENDING", "RUNNING", "THROTTLED"]:
|
||||
return "running"
|
||||
else:
|
||||
raise ValueError(f"Unknown RunwayML task status: {status}")
|
||||
|
||||
def _poll_task_sync(
|
||||
self,
|
||||
task_id: str,
|
||||
api_base: str,
|
||||
headers: Dict[str, str],
|
||||
timeout_secs: float = 600,
|
||||
) -> httpx.Response:
|
||||
"""
|
||||
Poll RunwayML task until completion (sync).
|
||||
|
||||
RunwayML POST returns immediately with a task that has status PENDING/RUNNING.
|
||||
We need to poll GET /v1/tasks/{task_id} until status is SUCCEEDED or FAILED.
|
||||
|
||||
Args:
|
||||
task_id: The task ID to poll
|
||||
api_base: Base URL for RunwayML API
|
||||
headers: Request headers (including auth)
|
||||
timeout_secs: Total timeout in seconds (default: 600s = 10 minutes)
|
||||
|
||||
Returns:
|
||||
Final response with completed task
|
||||
"""
|
||||
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
|
||||
|
||||
client = _get_httpx_client()
|
||||
start_time = time.time()
|
||||
|
||||
# Build task status URL
|
||||
api_base = api_base.rstrip("/")
|
||||
task_url = f"{api_base}/v1/tasks/{task_id}"
|
||||
|
||||
verbose_logger.debug(f"Polling RunwayML TTS task: {task_url}")
|
||||
|
||||
while True:
|
||||
self._check_timeout(start_time=start_time, timeout_secs=timeout_secs)
|
||||
|
||||
# Poll the task status
|
||||
response = client.get(url=task_url, headers=headers)
|
||||
response.raise_for_status()
|
||||
|
||||
response_data = response.json()
|
||||
|
||||
# Check task status
|
||||
status = self._check_task_status(response_data=response_data)
|
||||
|
||||
if status == "succeeded":
|
||||
return response
|
||||
elif status == "running":
|
||||
# Wait before polling again (RunwayML recommends 1-2 second intervals)
|
||||
time.sleep(2)
|
||||
|
||||
async def _poll_task_async(
|
||||
self,
|
||||
task_id: str,
|
||||
api_base: str,
|
||||
headers: Dict[str, str],
|
||||
timeout_secs: float = 600,
|
||||
) -> httpx.Response:
|
||||
"""
|
||||
Poll RunwayML task until completion (async).
|
||||
|
||||
Args:
|
||||
task_id: The task ID to poll
|
||||
api_base: Base URL for RunwayML API
|
||||
headers: Request headers (including auth)
|
||||
timeout_secs: Total timeout in seconds (default: 600s = 10 minutes)
|
||||
|
||||
Returns:
|
||||
Final response with completed task
|
||||
"""
|
||||
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
|
||||
|
||||
client = get_async_httpx_client(llm_provider=litellm.LlmProviders.RUNWAYML)
|
||||
start_time = time.time()
|
||||
|
||||
# Build task status URL
|
||||
api_base = api_base.rstrip("/")
|
||||
task_url = f"{api_base}/v1/tasks/{task_id}"
|
||||
|
||||
verbose_logger.debug(f"Polling RunwayML TTS task (async): {task_url}")
|
||||
|
||||
while True:
|
||||
self._check_timeout(start_time=start_time, timeout_secs=timeout_secs)
|
||||
|
||||
# Poll the task status
|
||||
response = await client.get(url=task_url, headers=headers)
|
||||
response.raise_for_status()
|
||||
|
||||
response_data = response.json()
|
||||
|
||||
# Check task status
|
||||
status = self._check_task_status(response_data=response_data)
|
||||
|
||||
if status == "succeeded":
|
||||
return response
|
||||
elif status == "running":
|
||||
# Wait before polling again (RunwayML recommends 1-2 second intervals)
|
||||
await asyncio.sleep(2)
|
||||
|
||||
def transform_text_to_speech_request(
|
||||
self,
|
||||
model: str,
|
||||
input: str,
|
||||
voice: Optional[Union[str, Dict]],
|
||||
optional_params: Dict,
|
||||
litellm_params: Dict,
|
||||
headers: dict,
|
||||
) -> TextToSpeechRequestData:
|
||||
"""
|
||||
Transform OpenAI TTS request to RunwayML TTS format
|
||||
|
||||
RunwayML expects:
|
||||
- model: The model to use (e.g., 'eleven_multilingual_v2')
|
||||
- promptText: The text to convert to speech
|
||||
- voice: Voice configuration object
|
||||
{
|
||||
"type": "runway-preset",
|
||||
"presetId": "Bernard"
|
||||
}
|
||||
|
||||
Returns:
|
||||
TextToSpeechRequestData: Contains JSON body and headers
|
||||
"""
|
||||
# Get voice from optional_params (mapped in map_openai_params)
|
||||
runwayml_voice = optional_params.get("runwayml_voice")
|
||||
if runwayml_voice is None:
|
||||
# Use default voice if not provided
|
||||
runwayml_voice = {
|
||||
"type": self.DEFAULT_VOICE_TYPE,
|
||||
"presetId": self.DEFAULT_VOICE_PRESET_ID,
|
||||
}
|
||||
|
||||
# Build request body
|
||||
request_body = {
|
||||
"model": model or self.DEFAULT_MODEL,
|
||||
"promptText": input,
|
||||
"voice": runwayml_voice,
|
||||
}
|
||||
|
||||
# Add any other optional parameters (except runwayml_voice which we already used)
|
||||
for k, v in optional_params.items():
|
||||
if k not in request_body and k != "runwayml_voice":
|
||||
request_body[k] = v
|
||||
|
||||
return {
|
||||
"dict_body": request_body,
|
||||
"headers": headers,
|
||||
}
|
||||
|
||||
def transform_text_to_speech_response(
|
||||
self,
|
||||
model: str,
|
||||
raw_response: httpx.Response,
|
||||
logging_obj: "LiteLLMLoggingObj",
|
||||
) -> "HttpxBinaryResponseContent":
|
||||
"""
|
||||
Transform RunwayML TTS response to standard format
|
||||
|
||||
RunwayML returns a task immediately with status PENDING/RUNNING.
|
||||
We need to poll the task until it completes, then download the audio.
|
||||
|
||||
Initial response:
|
||||
{
|
||||
"id": "task_123...",
|
||||
"status": "PENDING" | "RUNNING",
|
||||
"createdAt": "2025-11-13T..."
|
||||
}
|
||||
|
||||
After polling:
|
||||
{
|
||||
"id": "task_123...",
|
||||
"status": "SUCCEEDED",
|
||||
"output": ["https://storage.googleapis.com/.../audio.mp3"],
|
||||
"completedAt": "2025-11-13T..."
|
||||
}
|
||||
"""
|
||||
from litellm.types.llms.openai import HttpxBinaryResponseContent
|
||||
|
||||
try:
|
||||
response_data = raw_response.json()
|
||||
except Exception as e:
|
||||
raise self.get_error_class(
|
||||
error_message=f"Error parsing RunwayML TTS response: {e}",
|
||||
status_code=raw_response.status_code,
|
||||
headers=dict(raw_response.headers),
|
||||
)
|
||||
|
||||
verbose_logger.debug("RunwayML TTS starting polling...")
|
||||
|
||||
# Get task ID
|
||||
task_id = response_data.get("id")
|
||||
if not task_id:
|
||||
raise ValueError("RunwayML TTS response missing task ID")
|
||||
|
||||
# Get headers for polling (need auth)
|
||||
poll_headers = {
|
||||
"Authorization": raw_response.request.headers.get("Authorization", ""),
|
||||
"X-Runway-Version": raw_response.request.headers.get(
|
||||
"X-Runway-Version", RUNWAYML_DEFAULT_API_VERSION
|
||||
),
|
||||
}
|
||||
|
||||
# Poll until task completes
|
||||
polled_response = self._poll_task_sync(
|
||||
task_id=task_id,
|
||||
api_base=self.DEFAULT_BASE_URL,
|
||||
headers=poll_headers,
|
||||
timeout_secs=RUNWAYML_POLLING_TIMEOUT,
|
||||
)
|
||||
|
||||
# Get the completed task data
|
||||
task_data = polled_response.json()
|
||||
|
||||
verbose_logger.debug("RunwayML TTS polling complete, downloading audio")
|
||||
|
||||
# Get audio URL from output
|
||||
output = task_data.get("output", [])
|
||||
if not output or not isinstance(output, list) or len(output) == 0:
|
||||
raise ValueError("RunwayML TTS response missing audio URL in output")
|
||||
|
||||
audio_url = output[0]
|
||||
if not isinstance(audio_url, str):
|
||||
raise ValueError(f"RunwayML TTS audio URL is not a string: {audio_url}")
|
||||
|
||||
# Download the audio file
|
||||
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
|
||||
|
||||
client = _get_httpx_client()
|
||||
audio_response = client.get(url=audio_url)
|
||||
audio_response.raise_for_status()
|
||||
|
||||
verbose_logger.debug("RunwayML TTS audio downloaded successfully")
|
||||
|
||||
# Return the audio data wrapped in HttpxBinaryResponseContent
|
||||
return HttpxBinaryResponseContent(audio_response)
|
||||
|
||||
async def async_transform_text_to_speech_response(
|
||||
self,
|
||||
model: str,
|
||||
raw_response: httpx.Response,
|
||||
logging_obj: "LiteLLMLoggingObj",
|
||||
) -> "HttpxBinaryResponseContent":
|
||||
"""
|
||||
Async transform RunwayML TTS response to standard format
|
||||
|
||||
Same as sync version but uses async polling and download
|
||||
"""
|
||||
from litellm.types.llms.openai import HttpxBinaryResponseContent
|
||||
|
||||
try:
|
||||
response_data = raw_response.json()
|
||||
except Exception as e:
|
||||
raise self.get_error_class(
|
||||
error_message=f"Error parsing RunwayML TTS response: {e}",
|
||||
status_code=raw_response.status_code,
|
||||
headers=dict(raw_response.headers),
|
||||
)
|
||||
|
||||
verbose_logger.debug("RunwayML TTS starting polling (async)...")
|
||||
|
||||
# Get task ID
|
||||
task_id = response_data.get("id")
|
||||
if not task_id:
|
||||
raise ValueError("RunwayML TTS response missing task ID")
|
||||
|
||||
# Get headers for polling (need auth)
|
||||
poll_headers = {
|
||||
"Authorization": raw_response.request.headers.get("Authorization", ""),
|
||||
"X-Runway-Version": raw_response.request.headers.get(
|
||||
"X-Runway-Version", RUNWAYML_DEFAULT_API_VERSION
|
||||
),
|
||||
}
|
||||
|
||||
# Poll until task completes (async)
|
||||
polled_response = await self._poll_task_async(
|
||||
task_id=task_id,
|
||||
api_base=self.DEFAULT_BASE_URL,
|
||||
headers=poll_headers,
|
||||
timeout_secs=RUNWAYML_POLLING_TIMEOUT,
|
||||
)
|
||||
|
||||
# Get the completed task data
|
||||
task_data = polled_response.json()
|
||||
|
||||
verbose_logger.debug("RunwayML TTS polling complete (async), downloading audio")
|
||||
|
||||
# Get audio URL from output
|
||||
output = task_data.get("output", [])
|
||||
if not output or not isinstance(output, list) or len(output) == 0:
|
||||
raise ValueError("RunwayML TTS response missing audio URL in output")
|
||||
|
||||
audio_url = output[0]
|
||||
if not isinstance(audio_url, str):
|
||||
raise ValueError(f"RunwayML TTS audio URL is not a string: {audio_url}")
|
||||
|
||||
# Download the audio file (async)
|
||||
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
|
||||
|
||||
client = get_async_httpx_client(llm_provider=litellm.LlmProviders.RUNWAYML)
|
||||
audio_response = await client.get(url=audio_url)
|
||||
audio_response.raise_for_status()
|
||||
|
||||
verbose_logger.debug("RunwayML TTS audio downloaded successfully (async)")
|
||||
|
||||
# Return the audio data wrapped in HttpxBinaryResponseContent
|
||||
return HttpxBinaryResponseContent(audio_response)
|
||||
|
||||
|
|
@ -6,6 +6,8 @@ from httpx._types import RequestFiles
|
|||
|
||||
import litellm
|
||||
from litellm.constants import RUNWAYML_DEFAULT_API_VERSION
|
||||
from litellm.llms.base_llm.chat.transformation import BaseLLMException
|
||||
from litellm.llms.base_llm.videos.transformation import BaseVideoConfig
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
AsyncHTTPHandler,
|
||||
HTTPHandler,
|
||||
|
|
@ -23,16 +25,9 @@ from litellm.types.videos.utils import (
|
|||
if TYPE_CHECKING:
|
||||
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
|
||||
|
||||
from ...base_llm.chat.transformation import BaseLLMException as _BaseLLMException
|
||||
from ...base_llm.videos.transformation import BaseVideoConfig as _BaseVideoConfig
|
||||
|
||||
LiteLLMLoggingObj = _LiteLLMLoggingObj
|
||||
BaseVideoConfig = _BaseVideoConfig
|
||||
BaseLLMException = _BaseLLMException
|
||||
else:
|
||||
LiteLLMLoggingObj = Any
|
||||
BaseVideoConfig = Any
|
||||
BaseLLMException = Any
|
||||
|
||||
|
||||
class RunwayMLVideoConfig(BaseVideoConfig):
|
||||
|
|
|
|||
|
|
@ -6006,6 +6006,39 @@ def speech( # noqa: PLR0915
|
|||
logging_obj=logging_obj,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
)
|
||||
elif custom_llm_provider == "runwayml":
|
||||
from litellm.llms.runwayml.text_to_speech.transformation import (
|
||||
RunwayMLTextToSpeechConfig,
|
||||
)
|
||||
|
||||
# RunwayML Text-to-Speech
|
||||
if text_to_speech_provider_config is None:
|
||||
raise litellm.BadRequestError(
|
||||
message="RunwayML Text-to-Speech configuration not found",
|
||||
model=model,
|
||||
llm_provider=custom_llm_provider,
|
||||
)
|
||||
|
||||
# Cast to specific RunwayML config type to access dispatch method
|
||||
runwayml_config = cast(
|
||||
RunwayMLTextToSpeechConfig, text_to_speech_provider_config
|
||||
)
|
||||
|
||||
response = runwayml_config.dispatch_text_to_speech( # type: ignore
|
||||
model=model,
|
||||
input=input,
|
||||
voice=voice,
|
||||
optional_params=optional_params,
|
||||
litellm_params_dict=litellm_params_dict,
|
||||
logging_obj=logging_obj,
|
||||
timeout=timeout,
|
||||
extra_headers=extra_headers,
|
||||
base_llm_http_handler=base_llm_http_handler,
|
||||
aspeech=aspeech or False,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
if response is None:
|
||||
raise Exception(
|
||||
|
|
|
|||
|
|
@ -7811,6 +7811,12 @@ class ProviderConfigManager:
|
|||
)
|
||||
|
||||
return AzureAVATextToSpeechConfig()
|
||||
elif litellm.LlmProviders.RUNWAYML == provider:
|
||||
from litellm.llms.runwayml.text_to_speech.transformation import (
|
||||
RunwayMLTextToSpeechConfig,
|
||||
)
|
||||
|
||||
return RunwayMLTextToSpeechConfig()
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
|
|
|
|||
|
|
@ -1368,7 +1368,7 @@
|
|||
"embeddings": false,
|
||||
"image_generations": true,
|
||||
"audio_transcriptions": false,
|
||||
"audio_speech": false,
|
||||
"audio_speech": true,
|
||||
"moderations": false,
|
||||
"batches": false,
|
||||
"rerank": false,
|
||||
|
|
|
|||
BIN
tests/audio_tests/runwayml_speech.mp3
Normal file
BIN
tests/audio_tests/runwayml_speech.mp3
Normal file
Binary file not shown.
|
|
@ -382,6 +382,60 @@ async def test_azure_ava_tts_async():
|
|||
pytest.fail(f"Test failed with exception: {str(e)}")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.flaky(retries=3, delay=1)
|
||||
async def test_runwayml_tts_async():
|
||||
"""
|
||||
Test RunwayML Text-to-Speech with real API request.
|
||||
"""
|
||||
litellm._turn_on_debug()
|
||||
api_key = os.getenv("RUNWAYML_API_KEY")
|
||||
api_base = os.getenv("RUNWAYML_API_BASE")
|
||||
|
||||
|
||||
speech_file_path = Path(__file__).parent / "runwayml_speech.mp3"
|
||||
|
||||
try:
|
||||
response = await litellm.aspeech(
|
||||
model="runwayml/eleven_multilingual_v2",
|
||||
voice="Rachel",
|
||||
input="Yuneng is gone, we miss him so much I hope he has a good coffee",
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
response_format="mp3",
|
||||
speed=1.0,
|
||||
)
|
||||
|
||||
# Assert the response is HttpxBinaryResponseContent
|
||||
from litellm.types.llms.openai import HttpxBinaryResponseContent
|
||||
|
||||
assert isinstance(response, HttpxBinaryResponseContent)
|
||||
|
||||
# Get the binary content
|
||||
binary_content = response.content
|
||||
assert len(binary_content) > 0
|
||||
|
||||
# MP3 files start with these magic bytes
|
||||
# ID3 tag or MPEG sync word
|
||||
assert binary_content[:3] == b"ID3" or binary_content[:2] == b"\xff\xfb" or binary_content[:2] == b"\xff\xf3"
|
||||
|
||||
# Write to file
|
||||
response.stream_to_file(speech_file_path)
|
||||
|
||||
# Verify file was created and has content
|
||||
assert speech_file_path.exists()
|
||||
assert speech_file_path.stat().st_size > 0
|
||||
|
||||
print(f"Azure TTS audio saved to: {speech_file_path}")
|
||||
|
||||
# assert response cost is greater than 0
|
||||
print("Response cost: ", response._hidden_params["response_cost"])
|
||||
assert response._hidden_params["response_cost"] > 0
|
||||
|
||||
except Exception as e:
|
||||
pytest.fail(f"Test failed with exception: {str(e)}")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_azure_ava_tts_with_custom_voice():
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -0,0 +1,67 @@
|
|||
"""
|
||||
Test RunwayML text-to-speech transformation
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.abspath("../../../.."))
|
||||
|
||||
from litellm.llms.runwayml.text_to_speech.transformation import (
|
||||
RunwayMLTextToSpeechConfig,
|
||||
)
|
||||
|
||||
|
||||
def test_openai_voice_mapping_to_runwayml():
|
||||
"""
|
||||
Test that OpenAI voice names are correctly mapped to RunwayML preset IDs
|
||||
"""
|
||||
config = RunwayMLTextToSpeechConfig()
|
||||
|
||||
# Test OpenAI voice mappings
|
||||
openai_to_runway = {
|
||||
"alloy": "Maya",
|
||||
"echo": "James",
|
||||
"fable": "Bernard",
|
||||
"onyx": "Vincent",
|
||||
"nova": "Serene",
|
||||
"shimmer": "Ella",
|
||||
}
|
||||
|
||||
for openai_voice, expected_runway_voice in openai_to_runway.items():
|
||||
mapped_voice, mapped_params = config.map_openai_params(
|
||||
model="eleven_multilingual_v2",
|
||||
optional_params={},
|
||||
voice=openai_voice,
|
||||
drop_params=False,
|
||||
kwargs={},
|
||||
)
|
||||
|
||||
assert mapped_voice is None
|
||||
assert "runwayml_voice" in mapped_params
|
||||
assert mapped_params["runwayml_voice"]["type"] == "runway-preset"
|
||||
assert mapped_params["runwayml_voice"]["presetId"] == expected_runway_voice
|
||||
|
||||
|
||||
def test_runwayml_native_voice_passthrough():
|
||||
"""
|
||||
Test that RunwayML native voice names are passed through correctly as-is
|
||||
"""
|
||||
config = RunwayMLTextToSpeechConfig()
|
||||
|
||||
# Test various RunwayML native voices
|
||||
runway_voices = ["Bernard", "Maya", "Arjun", "Serene", "Chad"]
|
||||
|
||||
for runway_voice in runway_voices:
|
||||
mapped_voice, mapped_params = config.map_openai_params(
|
||||
model="eleven_multilingual_v2",
|
||||
optional_params={},
|
||||
voice=runway_voice,
|
||||
drop_params=False,
|
||||
kwargs={},
|
||||
)
|
||||
|
||||
assert mapped_voice is None
|
||||
assert "runwayml_voice" in mapped_params
|
||||
assert mapped_params["runwayml_voice"]["type"] == "runway-preset"
|
||||
assert mapped_params["runwayml_voice"]["presetId"] == runway_voice
|
||||
|
||||
Loading…
Add table
Reference in a new issue