litellm/tests/integration/sdk
Gabriel Creti 60d05972ae
feat(tts)!: stream audio from the provider as it arrives on /v1/audio/speech (#46222)
* feat(tts): stream audio from the provider as it arrives on /v1/audio/speech

The proxy read the whole audio body from the provider before sending the
first byte to the client, so time-to-first-byte was the full synthesis
time. Pass stream=True to get the bytes forwarded as they arrive.

The response type does not change: it is still HttpxBinaryResponseContent,
only unread, so every existing caller, logging path and spend hook works
the same. Providers opt in with supports_streaming on their config, the
others ignore the flag and buffer as before.

* fix(tts): keep streamed speech inside its concurrency and cleanup guarantees

Streaming the audio out moved two things that used to be covered by the
buffered response, and the review on #46222 caught both.

A deployment's max_parallel_requests counted syntheses that had started
rather than syntheses in flight, because aspeech now returns at the
response headers with the audio still arriving. The slot is held until
the response is closed, the way the chat streaming path already does it.

A client that walks away mid-clip leaves the body unread, and httpx only
releases the connection when it is read to the end. The body iterator
closes it on the way out, and the route arms the same close as a
background task for a response whose body is never read at all. These
are two different paths on purpose: Starlette raises ClientDisconnect
out of the response before background tasks run, so neither one covers
both cases.

Also pulls the speech body cast into one helper beside
speech_request_body, which drops a banned typing.cast import.

* test(tts): cover the slot-holding helper directly

router_code_coverage.py wants every function in router.py called by name
from a unit test, and _hold_slot_until_closed was only reached through
router.aspeech(). Test it on its own instead of adding it to the ignore
list: which responses keep their slot is the part worth pinning, so the
cases are buffered body, non-binary response and unread body.

* fix(tts): stream ElevenLabs speech from its /stream endpoint

ElevenLabs' convert endpoint synthesizes the whole clip before it sends the
first byte, so stream: true still waited for full synthesis. Requests with
stream: true now call /v1/text-to-speech/{voice_id}/stream.

* fix(tts): log streamed speech like buffered speech

A deployment-hook sync copied stream=True onto the logging object for
speech, so the success handler waited for an assembled chat stream that
never comes and streamed speech never reached spend logs or callbacks.

* fix(tts): forward streamed speech audio as it arrives

* fix(tts): log streamed speech when the audio ends

Streamed speech now logs success when the caller reads the audio to the end or closes it, and logs a failure when the upstream breaks mid-clip. Key max_parallel_requests slots stay held for the whole stream, the logged duration and bytes match what was sent, and the router slot is no longer pinned by a pending logging task.

* fix(tts): bill and map streamed speech failures, defer logging only when streaming

A streamed clip that breaks mid-way is now logged as a failure at the clip's cost, since speech is priced per input character and the provider bills the whole request. Transport errors map to litellm.Timeout or litellm.APIConnectionError, and the proxy runs post_call_failure_hook with the clip's cost. Success logging waits for the stream end only when the caller asked for stream=True.

* fix(tts): stream only on a literal true, mark hang-ups, and cut off without a fake DB outage

* fix(tts): put the 499 hang-up marker on the spend row

* fix(tts): keep the caller's metadata and positional base_url intact, add speech stream integration cells

* fix(speech): keep stream keyword-only and buffer the speech health check

Passing stream positionally skipped the logging wrapper, and a deployment
with stream: true left the health check's provider stream open. Also gives
each scripted chat reply its own id so the non-boolean stream cases do not
collide on the spend log key, and adds the audit's round-2 cells

* test(speech): check the health check calls the buffered path

* test(integration): read the speech chaos spend rows from the worker database

Main now gives each owned proxy a private database unless the test file opts
into the worker's state. The chaos tests read billed rows from the worker
database, so they opt in with the shares_worker_state marker like the other
providers chaos files

* fix(tts): release the router slot when a streamed read fails

The deployment slot was only released through Response.aclose, which httpx
does not call when the body read raises. A provider cutting a clip off
mid-read left the slot held, so later Router.aspeech calls got 429.

Release the slot on any stream end, and close the upstream stream when a
read fails.

Co-Authored-By: Claude <noreply@anthropic.com>

* test(tts): drive speech stream tests with events instead of sleeps

Co-Authored-By: Claude <noreply@anthropic.com>

* test(utils): capture streamed sync speech usage when the audio ends, not at the headers

* fix(proxy): close streamed speech when the route fails before responding

A response headers hook, or any other step between aspeech returning an unread
audio stream and the route returning its StreamingResponse, could raise and leave
that stream open. The upstream connection and the deployment's
max_parallel_requests slot then stayed held, so later clips got 429s. The route
now closes the audio on any error before it returns

* fix(proxy): end streamed speech as failed when the route errors before responding

Closing the unread audio on a pre-response error went through aclose, which always
ends as StreamClosedEarly: the clip was success-logged with a 499 ClientDisconnected
marker while the route returned 500. The route now ends the audio with the error,
through every OnEndAsyncByteStream wrapper, so it is logged as a failure. A
cancellation still closes it as a hang-up.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(proxy): close unread speech audio when a deployment guardrail rejects it

A guardrail's post-call deployment hook runs before the audio stream's end logger is attached, so a rejection left the unread provider response open and held its connection. The failure path of the async wrapper now closes it, and a close that fails never replaces the rejection

Also annotates the new test locals in test_routes_audio.py with Final

* fix(proxy): free the key's parallel slot before a streamed speech clip ends

A streamed clip released its key's max_parallel_requests slot only from the
success logging task, which defers the Redis write until after the caller has
seen the end of the audio. A next clip sent to another instance at that moment
could still be counted against the key and get 429. The speech route now
releases the slot when the audio finishes, before the response ends, the same
way chat streams do.

* fix(speech): log a failed streamed clip once per failure hook

A logger with both failure hooks got both for one failed aspeech stream. The
stream's failure path now marks its sync run as part of an async request, as the
non-stream path does. Also type the route test helpers and check that the speech
health check leaves no provider connection open.

* test(speech): expect one async failure event from a cut-off aspeech stream

* fix(proxy): shield the speech upstream close and tag cut-off rows as aspeech

A hang-up while the provider is silent no longer lets the anyio cancel abort the upstream close
on servers that cancel the stream on disconnect. Mid-clip cut-off failure rows now store the
aspeech call type instead of an empty one.

* refactor(logging): record a speech client disconnect through the logging object

* test(speech): hold the provider mid-clip in the guardrail-rejection wire test

The test sent the whole short clip at once on a keep-alive connection, so when the clip reached the HTTP client before the guardrail rejected it, closing the response handed the finished connection back to the pool and the test timed out waiting for a close. The peer now holds after the first audio chunk until the rejection is raised, as a real provider still sending a long clip would

* fix(speech): close the unread audio when a sync speech call fails after its headers

The async wrapper already closed it. The sync wrapper left the provider
connection open until garbage collection when post-call processing raised.

* fix(router): keep hung-up speech clips out of latency samples

A streamed clip the caller hangs up on ends early, so its short duration made
the deployment look fast to latency-based routing. The clip still counts as a
request.

* test(proxy): type the speech request helper in the audio route tests

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-17 15:03:12 +00:00
..
_model_cost_sdk_child.py refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
conftest.py fix(tests): drain SDK logging before restoring callbacks (#46924) 2026-10-16 12:39:26 -07:00
test_aiohttp_session_rebuild_wire.py fix(aiohttp_transport): close an owned aiohttp session when its transport is collected (#48650) 2026-10-27 20:33:03 +00:00
test_anthropic_common_utils_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_anthropic_json_tool_cut_stream_sync_sdk.py fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
test_anthropic_messages_bridge_mid_stream_failure_sdk.py test: extend runtime regression contracts for Messages and SDK (#46641) 2026-10-15 17:51:15 +00:00
test_azure_common_utils_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_azure_prompt_shield_tuple_messages.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_bedrock_converse_redacted_thinking_router_sdk.py test(anthropic, bedrock): audit #47489 (#48782) 2026-10-28 04:24:50 -07:00
test_bedrock_converse_stream_sync_decoder_wire.py fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) 2026-10-18 00:06:07 +00:00
test_bedrock_model_discovery_sdk.py fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on (#43981) 2026-10-09 14:14:17 -07:00
test_bedrock_native_guardrail_shared_metadata_sdk.py fix(logging): bill each SDK call its own native Bedrock guardrail trace when calls share one metadata dict (LIT-10239) (#48923) 2026-10-28 15:11:26 -07:00
test_bedrock_native_output_fallback_sdk.py fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) 2026-10-18 00:06:07 +00:00
test_bedrock_native_output_register_model_sdk.py refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) 2026-11-11 01:57:54 +00:00
test_cache_control_census_pydantic_messages_sdk.py fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it (#46591) 2026-10-15 14:46:43 -07:00
test_caching_redis_sdk.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_callback_logging_cache.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_configuration_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_context_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_customer_model_policy_fallback_wire.py feat(customer-portal): complete hierarchy management (#51521) 2026-11-13 17:58:34 +00:00
test_databricks_json_schema_refs_sdk.py fix(databricks): keep #/$defs refs in json_schema response_format (#45659) 2026-10-09 18:51:36 -07:00
test_device_code_login_guard_sdk.py test(copilot): align login guards with direct access tokens (#50285) 2026-11-05 15:38:33 -08:00
test_docker_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_drop_params_env_var_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_dual_cache_redis.py feat(redis): serve the router's session pins from a native store (#50870) 2026-11-13 03:22:28 +00:00
test_eager_tiktoken_load_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_exa_search_envelope_sdk.py fix(exa): return highlights, highlight scores, score and the response envelope from Exa search (#47423) 2026-10-19 15:45:56 -07:00
test_gemini_long_decode_sync_sdk.py feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240) 2026-10-15 22:54:23 +00:00
test_gemini_tts_speech_logging_sdk.py fix(speech)!: correctly propagate spend logging and usage for Gemini TTS bridge (#44551) 2026-10-19 23:02:11 +00:00
test_generic_api_callback.py perf(ci): isolate and shard integration suites (#44146) 2026-10-10 11:15:19 -07:00
test_google_genai_adapter_stream_usage_sdk.py fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836) 2026-10-15 08:06:49 -07:00
test_http2_wire.py test(http2): drain client traffic after sending GOAWAY (#50972) 2026-11-10 18:31:57 +00:00
test_http_handler_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_huggingface_tokenizer_downloads_sdk.py fix(token_counter): count DeepSeek V4 text with DeepSeek's own tokenizer (#47353) 2026-10-20 03:43:35 +00:00
test_idle_upstream_reconnect_sdk.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
test_interception_converted_stream_options_sdk.py fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream (#45433) 2026-10-09 22:16:02 -07:00
test_jwt_signing_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_langfuse_otel_trace_fields_sdk.py fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705) 2026-10-27 10:34:55 -07:00
test_langfuse_redaction_sdk.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_lazy_imports_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_litellm_cli_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_local_sandbox_sdk.py test(harness): audit #47530 (#48751) 2026-10-28 01:57:41 -07:00
test_long_decode_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_model_cost_fetch_sdk.py fix(cost-map): restore backward compatibility after the Rust loader move (#51545) 2026-11-14 08:12:17 +09:00
test_model_cost_sdk.py refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
test_ocr_matrix_sdk.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_ollama_embedding_wire.py fix(ollama): forward embedding credentials, headers and timeout (#48707) 2026-10-28 09:32:47 -07:00
test_openai_realtime_http_ca_bundle_wire.py feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
test_openai_sdk_client_sessions_wire.py fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
test_owned_redis_ports.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_prometheus_copilot_session_auth_failure_sdk.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_provider_budget_redis.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_provider_error_and_extra_body_families_wire.py feat(providers): add Inference APIs as OpenAI-compatible provider (#41669) 2026-10-15 08:17:36 -07:00
test_redis_breaker_metrics_multiprocess_sdk.py fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger (#46484) 2026-10-14 14:29:54 -07:00
test_redis_cluster_iam_auth.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_redis_service_metrics.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_response_cache_sync_sdk.py fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104) 2026-10-14 14:42:49 -07:00
test_responses_bridge_stream_errors_sdk.py test(integration): update stale CircleCI integration expectations (#46570) 2026-10-14 18:21:59 -07:00
test_responses_bridge_sync_reasoning_sdk.py fix(bedrock)!: keep reasoning, thinking blocks and the json_mode finish_reason on fake streams (#46863) 2026-10-16 17:27:57 +00:00
test_responses_bridge_tool_search_sdk.py fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444) 2026-10-14 21:40:14 -07:00
test_router_behavior_wire.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_budget_limiter_wire.py fix(router): stop queueing budget increments for Redis when no Redis is configured (#46831) 2026-10-16 15:52:42 +00:00
test_router_concurrent_deployment_delete_sdk.py test(router): hold the concurrent upsert where its build reads the list back (#51878) 2026-11-14 23:57:30 +00:00
test_router_cooldown_fallback_sdk.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_router_fuzz_regressions_rust_only_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_router_host_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_router_include_fallback_errors_wire.py test(router): expect no fallback errors header for non-bool include_fallback_errors (#50259) 2026-11-05 15:34:08 -08:00
test_router_lowest_latency_wire.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_redis_tls_url.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_session_pins_redis.py feat(redis): serve the router's session pins from a native store (#50870) 2026-11-13 03:22:28 +00:00
test_router_silent_model_shadow_sdk.py fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs (#48888) 2026-10-29 05:12:50 -07:00
test_router_sync_stream_fallback_wire.py fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493) 2026-10-14 14:19:25 -07:00
test_router_tool_call_stream_fallback_budget_wire.py fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493) 2026-10-14 14:19:25 -07:00
test_router_unsupported_params_fallback_wire.py fix(router): honor fallback opt-out for 413 and 422 bad requests (#52195) 2026-11-16 21:51:23 -08:00
test_sagemaker_embedding_sdk.py feat(sagemaker)!: inference components and OpenAI-compatible embeddings (#44985) 2026-10-12 22:25:52 +00:00
test_slack_daily_report_redis.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_speech_stream_sdk_wire.py feat(tts)!: stream audio from the provider as it arrives on /v1/audio/speech (#46222) 2026-11-17 15:03:12 +00:00
test_stream_error_api_base_sdk.py fix(streaming): report the deployment's api_base when a stream fails mid-way (#47045) 2026-10-17 15:41:24 +00:00
test_stream_open_error_provider_headers_sdk.py fix(streaming): keep provider response headers on SDK streaming errors (#41528) 2026-10-15 08:11:54 -07:00
test_streaming_handler_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_sync_stream_connection_release_wire.py test(streaming): pin that a stream stuck on one chunk closes its provider stream and logs one failure (#38753) 2026-10-26 18:03:37 +00:00
test_text_completion_stream_rebuild_sdk.py fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions (#44349) 2026-10-20 05:31:39 +00:00
test_token_counter_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_tokenizer_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_tool_loop_responses_history_replay_sdk.py fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912) 2026-10-29 00:56:02 -07:00
test_tool_loop_server_tool_replay_sdk.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_topaz_image_variation_trailing_slash_sdk.py fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
test_unset_effort_tool_bridge_wire.py fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions (#50968) 2026-11-10 18:33:27 +00:00
test_usage_based_routing_sdk_redis_reads.py test(integration): repair tests that merged changes left stale (#50782) 2026-11-10 01:08:35 +00:00
test_usage_routing_counter_ttl.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_vertex_batch_row_line_errors_sdk.py fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097) 2026-10-15 03:21:48 -07:00
test_vertex_gemini_image_batch_cost_sdk.py refactor: expose public hidden_params accessors (#44668) 2026-10-07 00:01:33 -07:00
test_vertex_gemini_stream_timeout_wire.py fix(vertex_ai): pass the per-request timeout to Gemini streaming calls (#46349) 2026-10-14 10:51:18 -07:00
test_vertex_partner_count_tokens_sdk.py fix(responses)!: count input tokens the way inference sends the request (#47363) 2026-10-20 15:13:59 +00:00