mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
9445634193
* fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307) * test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Backport of #46307 to rc/1.107.0 Cherry-picked from |
||
|---|---|---|
| .. | ||
| _azure_storage_support.py | ||
| _logging_only_scope_support.py | ||
| _openinference_support.py | ||
| _s3_v2_support.py | ||
| _ternary_support.py | ||
| azure_dispatch_support.py | ||
| conftest.py | ||
| logging_worker_drain_canary.py | ||
| test_akto_logging_only.py | ||
| test_apply_guardrail_endpoint.py | ||
| test_arize_otel_v2_openinference_chaos.py | ||
| test_arize_otel_v2_openinference_family.py | ||
| test_arize_otel_v2_openinference_modes.py | ||
| test_arize_otel_v2_openinference_sad_edge.py | ||
| test_arize_otel_v2_openinference_spans.py | ||
| test_azure_content_safety_audit.py | ||
| test_azure_content_safety_dispatch.py | ||
| test_azure_content_safety_dispatch_resilience.py | ||
| test_azure_content_safety_endpoints.py | ||
| test_azure_storage_chaos.py | ||
| test_azure_storage_client_ttl.py | ||
| test_azure_storage_file_names.py | ||
| test_batch_line_items_cold_storage.py | ||
| test_bedrock_converse_stream_post_call_guardrail.py | ||
| test_bedrock_error_request_id.py | ||
| test_bedrock_guardrail_attachments.py | ||
| test_bedrock_invoke_stream_post_call_guardrail.py | ||
| test_cache_hit_guardrail_metrics.py | ||
| test_cache_hit_guardrail_metrics_chaos.py | ||
| test_callback_delivery.py | ||
| test_content_filter_logging_only.py | ||
| test_custom_logger_redaction.py | ||
| test_datadog_logging_delivery.py | ||
| test_file_usage_counter_span.py | ||
| test_grayswan_wire.py | ||
| test_grayswan_wire_chaos.py | ||
| test_guardrail_attachment.py | ||
| test_guardrail_effects.py | ||
| test_guardrail_load_balancing.py | ||
| test_guardrail_stream_scope.py | ||
| test_guardrail_stream_scope_chaos.py | ||
| test_guardrail_stream_scope_matrix.py | ||
| test_guardrail_timeout_all_providers.py | ||
| test_langfuse_delivery.py | ||
| test_langsmith_logging_delivery.py | ||
| test_langtrace_delivery.py | ||
| test_logging_only_scope_chaos.py | ||
| test_logging_only_scope_config.py | ||
| test_logging_only_scope_runtime.py | ||
| test_logging_worker_drain.py | ||
| test_otel_conversation_id.py | ||
| test_otel_excluded_services.py | ||
| test_otel_excluded_services_matrix.py | ||
| test_otel_team_capture_message_content.py | ||
| test_otel_text_completion_choices.py | ||
| test_otel_trace_response_header.py | ||
| test_otel_v1_request_trace.py | ||
| test_passthrough_upstream_error_chaos.py | ||
| test_passthrough_upstream_error_visibility.py | ||
| test_presidio_entity_masking.py | ||
| test_presidio_streaming_output.py | ||
| test_prometheus_project_alias_label.py | ||
| test_prometheus_request_metrics.py | ||
| test_prometheus_series_cap.py | ||
| test_prometheus_series_cap_chaos.py | ||
| test_prometheus_shared_custom_labels.py | ||
| test_realtime_guardrail_translation.py | ||
| test_s3_v2_flush_surfaces.py | ||
| test_s3_v2_partition_granularity.py | ||
| test_s3_v2_streaming_failure_dedupe.py | ||
| test_s3_v2_upload_fanout.py | ||
| test_signoz_delivery.py | ||
| test_straiker_v3_platform.py | ||
| test_ternary_focus_export.py | ||
| test_wingback_guardrail.py | ||
| test_xecguard_wire.py | ||