mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-18 09:45:23 +00:00
* feat(tts): stream audio from the provider as it arrives on /v1/audio/speech The proxy read the whole audio body from the provider before sending the first byte to the client, so time-to-first-byte was the full synthesis time. Pass stream=True to get the bytes forwarded as they arrive. The response type does not change: it is still HttpxBinaryResponseContent, only unread, so every existing caller, logging path and spend hook works the same. Providers opt in with supports_streaming on their config, the others ignore the flag and buffer as before. * fix(tts): keep streamed speech inside its concurrency and cleanup guarantees Streaming the audio out moved two things that used to be covered by the buffered response, and the review on #46222 caught both. A deployment's max_parallel_requests counted syntheses that had started rather than syntheses in flight, because aspeech now returns at the response headers with the audio still arriving. The slot is held until the response is closed, the way the chat streaming path already does it. A client that walks away mid-clip leaves the body unread, and httpx only releases the connection when it is read to the end. The body iterator closes it on the way out, and the route arms the same close as a background task for a response whose body is never read at all. These are two different paths on purpose: Starlette raises ClientDisconnect out of the response before background tasks run, so neither one covers both cases. Also pulls the speech body cast into one helper beside speech_request_body, which drops a banned typing.cast import. * test(tts): cover the slot-holding helper directly router_code_coverage.py wants every function in router.py called by name from a unit test, and _hold_slot_until_closed was only reached through router.aspeech(). Test it on its own instead of adding it to the ignore list: which responses keep their slot is the part worth pinning, so the cases are buffered body, non-binary response and unread body. * fix(tts): stream ElevenLabs speech from its /stream endpoint ElevenLabs' convert endpoint synthesizes the whole clip before it sends the first byte, so stream: true still waited for full synthesis. Requests with stream: true now call /v1/text-to-speech/{voice_id}/stream. * fix(tts): log streamed speech like buffered speech A deployment-hook sync copied stream=True onto the logging object for speech, so the success handler waited for an assembled chat stream that never comes and streamed speech never reached spend logs or callbacks. * fix(tts): forward streamed speech audio as it arrives * fix(tts): log streamed speech when the audio ends Streamed speech now logs success when the caller reads the audio to the end or closes it, and logs a failure when the upstream breaks mid-clip. Key max_parallel_requests slots stay held for the whole stream, the logged duration and bytes match what was sent, and the router slot is no longer pinned by a pending logging task. * fix(tts): bill and map streamed speech failures, defer logging only when streaming A streamed clip that breaks mid-way is now logged as a failure at the clip's cost, since speech is priced per input character and the provider bills the whole request. Transport errors map to litellm.Timeout or litellm.APIConnectionError, and the proxy runs post_call_failure_hook with the clip's cost. Success logging waits for the stream end only when the caller asked for stream=True. * fix(tts): stream only on a literal true, mark hang-ups, and cut off without a fake DB outage * fix(tts): put the 499 hang-up marker on the spend row * fix(tts): keep the caller's metadata and positional base_url intact, add speech stream integration cells * fix(speech): keep stream keyword-only and buffer the speech health check Passing stream positionally skipped the logging wrapper, and a deployment with stream: true left the health check's provider stream open. Also gives each scripted chat reply its own id so the non-boolean stream cases do not collide on the spend log key, and adds the audit's round-2 cells * test(speech): check the health check calls the buffered path * test(integration): read the speech chaos spend rows from the worker database Main now gives each owned proxy a private database unless the test file opts into the worker's state. The chaos tests read billed rows from the worker database, so they opt in with the shares_worker_state marker like the other providers chaos files * fix(tts): release the router slot when a streamed read fails The deployment slot was only released through Response.aclose, which httpx does not call when the body read raises. A provider cutting a clip off mid-read left the slot held, so later Router.aspeech calls got 429. Release the slot on any stream end, and close the upstream stream when a read fails. Co-Authored-By: Claude <noreply@anthropic.com> * test(tts): drive speech stream tests with events instead of sleeps Co-Authored-By: Claude <noreply@anthropic.com> * test(utils): capture streamed sync speech usage when the audio ends, not at the headers * fix(proxy): close streamed speech when the route fails before responding A response headers hook, or any other step between aspeech returning an unread audio stream and the route returning its StreamingResponse, could raise and leave that stream open. The upstream connection and the deployment's max_parallel_requests slot then stayed held, so later clips got 429s. The route now closes the audio on any error before it returns * fix(proxy): end streamed speech as failed when the route errors before responding Closing the unread audio on a pre-response error went through aclose, which always ends as StreamClosedEarly: the clip was success-logged with a 499 ClientDisconnected marker while the route returned 500. The route now ends the audio with the error, through every OnEndAsyncByteStream wrapper, so it is logged as a failure. A cancellation still closes it as a hang-up. Co-Authored-By: Claude <noreply@anthropic.com> * fix(proxy): close unread speech audio when a deployment guardrail rejects it A guardrail's post-call deployment hook runs before the audio stream's end logger is attached, so a rejection left the unread provider response open and held its connection. The failure path of the async wrapper now closes it, and a close that fails never replaces the rejection Also annotates the new test locals in test_routes_audio.py with Final * fix(proxy): free the key's parallel slot before a streamed speech clip ends A streamed clip released its key's max_parallel_requests slot only from the success logging task, which defers the Redis write until after the caller has seen the end of the audio. A next clip sent to another instance at that moment could still be counted against the key and get 429. The speech route now releases the slot when the audio finishes, before the response ends, the same way chat streams do. * fix(speech): log a failed streamed clip once per failure hook A logger with both failure hooks got both for one failed aspeech stream. The stream's failure path now marks its sync run as part of an async request, as the non-stream path does. Also type the route test helpers and check that the speech health check leaves no provider connection open. * test(speech): expect one async failure event from a cut-off aspeech stream * fix(proxy): shield the speech upstream close and tag cut-off rows as aspeech A hang-up while the provider is silent no longer lets the anyio cancel abort the upstream close on servers that cancel the stream on disconnect. Mid-clip cut-off failure rows now store the aspeech call type instead of an empty one. * refactor(logging): record a speech client disconnect through the logging object * test(speech): hold the provider mid-clip in the guardrail-rejection wire test The test sent the whole short clip at once on a keep-alive connection, so when the clip reached the HTTP client before the guardrail rejected it, closing the response handed the finished connection back to the pool and the test timed out waiting for a close. The peer now holds after the first audio chunk until the rejection is raised, as a real provider still sending a long clip would * fix(speech): close the unread audio when a sync speech call fails after its headers The async wrapper already closed it. The sync wrapper left the provider connection open until garbage collection when post-call processing raised. * fix(router): keep hung-up speech clips out of latency samples A streamed clip the caller hangs up on ends early, so its short duration made the deployment look fast to latency-based routing. The clip still counts as a request. * test(proxy): type the speech request helper in the audio route tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| _model_cost_sdk_child.py | ||
| conftest.py | ||
| test_aiohttp_session_rebuild_wire.py | ||
| test_anthropic_common_utils_real_io.py | ||
| test_anthropic_json_tool_cut_stream_sync_sdk.py | ||
| test_anthropic_messages_bridge_mid_stream_failure_sdk.py | ||
| test_azure_common_utils_real_io.py | ||
| test_azure_prompt_shield_tuple_messages.py | ||
| test_bedrock_converse_redacted_thinking_router_sdk.py | ||
| test_bedrock_converse_stream_sync_decoder_wire.py | ||
| test_bedrock_model_discovery_sdk.py | ||
| test_bedrock_native_guardrail_shared_metadata_sdk.py | ||
| test_bedrock_native_output_fallback_sdk.py | ||
| test_bedrock_native_output_register_model_sdk.py | ||
| test_cache_control_census_pydantic_messages_sdk.py | ||
| test_caching_redis_sdk.py | ||
| test_callback_logging_cache.py | ||
| test_configuration_real_io.py | ||
| test_context_real_io.py | ||
| test_customer_model_policy_fallback_wire.py | ||
| test_databricks_json_schema_refs_sdk.py | ||
| test_device_code_login_guard_sdk.py | ||
| test_docker_real_io.py | ||
| test_drop_params_env_var_real_io.py | ||
| test_dual_cache_redis.py | ||
| test_eager_tiktoken_load_real_io.py | ||
| test_exa_search_envelope_sdk.py | ||
| test_gemini_long_decode_sync_sdk.py | ||
| test_gemini_tts_speech_logging_sdk.py | ||
| test_generic_api_callback.py | ||
| test_google_genai_adapter_stream_usage_sdk.py | ||
| test_http2_wire.py | ||
| test_http_handler_real_io.py | ||
| test_huggingface_tokenizer_downloads_sdk.py | ||
| test_idle_upstream_reconnect_sdk.py | ||
| test_interception_converted_stream_options_sdk.py | ||
| test_jwt_signing_real_io.py | ||
| test_langfuse_otel_trace_fields_sdk.py | ||
| test_langfuse_redaction_sdk.py | ||
| test_lazy_imports_real_io.py | ||
| test_litellm_cli_real_io.py | ||
| test_local_sandbox_sdk.py | ||
| test_long_decode_real_io.py | ||
| test_model_cost_fetch_sdk.py | ||
| test_model_cost_sdk.py | ||
| test_ocr_matrix_sdk.py | ||
| test_ollama_embedding_wire.py | ||
| test_openai_realtime_http_ca_bundle_wire.py | ||
| test_openai_sdk_client_sessions_wire.py | ||
| test_owned_redis_ports.py | ||
| test_prometheus_copilot_session_auth_failure_sdk.py | ||
| test_provider_budget_redis.py | ||
| test_provider_error_and_extra_body_families_wire.py | ||
| test_redis_breaker_metrics_multiprocess_sdk.py | ||
| test_redis_cluster_iam_auth.py | ||
| test_redis_service_metrics.py | ||
| test_response_cache_sync_sdk.py | ||
| test_responses_bridge_stream_errors_sdk.py | ||
| test_responses_bridge_sync_reasoning_sdk.py | ||
| test_responses_bridge_tool_search_sdk.py | ||
| test_router_behavior_wire.py | ||
| test_router_budget_limiter_wire.py | ||
| test_router_concurrent_deployment_delete_sdk.py | ||
| test_router_cooldown_fallback_sdk.py | ||
| test_router_fuzz_regressions_rust_only_real_io.py | ||
| test_router_host_real_io.py | ||
| test_router_include_fallback_errors_wire.py | ||
| test_router_lowest_latency_wire.py | ||
| test_router_redis_tls_url.py | ||
| test_router_session_pins_redis.py | ||
| test_router_silent_model_shadow_sdk.py | ||
| test_router_sync_stream_fallback_wire.py | ||
| test_router_tool_call_stream_fallback_budget_wire.py | ||
| test_router_unsupported_params_fallback_wire.py | ||
| test_sagemaker_embedding_sdk.py | ||
| test_slack_daily_report_redis.py | ||
| test_speech_stream_sdk_wire.py | ||
| test_stream_error_api_base_sdk.py | ||
| test_stream_open_error_provider_headers_sdk.py | ||
| test_streaming_handler_real_io.py | ||
| test_sync_stream_connection_release_wire.py | ||
| test_text_completion_stream_rebuild_sdk.py | ||
| test_token_counter_real_io.py | ||
| test_tokenizer_real_io.py | ||
| test_tool_loop_responses_history_replay_sdk.py | ||
| test_tool_loop_server_tool_replay_sdk.py | ||
| test_topaz_image_variation_trailing_slash_sdk.py | ||
| test_unset_effort_tool_bridge_wire.py | ||
| test_usage_based_routing_sdk_redis_reads.py | ||
| test_usage_routing_counter_ttl.py | ||
| test_vertex_batch_row_line_errors_sdk.py | ||
| test_vertex_gemini_image_batch_cost_sdk.py | ||
| test_vertex_gemini_stream_timeout_wire.py | ||
| test_vertex_partner_count_tokens_sdk.py | ||