mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-31 06:56:59 +00:00
06cdee91da
* feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines Gives OpenAITextCompletionHandler a process_output_streaming_response that assembles the buffered TextCompletionResponse chunks per choice index at end of stream, runs the guardrail over the assembled text, and writes text rewrites back into the first text-carrying chunk of each choice. Tests and live proof still pending. * fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs. * test(guardrails): cover streamed /v1/completions under post_call guardrails * fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through * fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter * fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent The content filter's streaming hook held back the end of a choice until its finish_reason, which litellm sends only for the first choice of an n>1 stream, so choice 1 lost its last characters on text and chat streams. A closing chunk now releases every unfinished choice when the stream ends. A blocked text stream's closing chunk reported 0/0 usage when the upstream sent none; it now carries usage only when the upstream reported it. * fix(guardrails): send no usage on the content filter's end-of-stream chunk * fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions * fix(guardrails): stream Responses API events through the content filter as they arrive * fix(guardrails): treat a null text completion choice index as choice zero when assembling streams --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| faked_responses_case.py | ||
| test_assistants_evals_stream.py | ||
| test_azure_streaming_function_calls.py | ||
| test_chat_stream_guardrail_error_events.py | ||
| test_decision_model_buffering.py | ||
| test_decision_model_pipeline_buffering.py | ||
| test_fake_stream_reasoning_wire.py | ||
| test_faked_responses_stream_bridges.py | ||
| test_faked_responses_stream_cache_replay.py | ||
| test_faked_responses_stream_chaos.py | ||
| test_faked_responses_stream_guardrail_blocks.py | ||
| test_faked_responses_stream_items.py | ||
| test_file_content_streaming.py | ||
| test_google_genai_stream_usage_settings_wire.py | ||
| test_interception_converted_stream_options.py | ||
| test_mid_stream_fallback_continuation.py | ||
| test_noma_v2_streaming_settings.py | ||
| test_openai_like_stream_usage_wire.py | ||
| test_openai_sdk_plain_chunk_fields.py | ||
| test_openai_stream_passthrough_fields.py | ||
| test_responses_bridge_stream_chaos.py | ||
| test_responses_bridge_stream_errors.py | ||
| test_responses_top_level_error_events.py | ||
| test_stream_contracts.py | ||
| test_stream_open_error_provider_headers_proxy.py | ||
| test_stream_parallel_slot_release.py | ||
| test_text_completion_stream_guardrail_rebuild.py | ||
| test_ttft_keepalive.py | ||