mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-26 06:03:19 +00:00
68a8b12ab3
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out. * fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE] A guardrail block before the first chunk is raised again, so its response stays the 400 event-stream body it is on main. A mid-stream error event now carries the failure hook's replacement error when one is returned, is followed by data: [DONE] on OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details JSON cannot encode are written as strings instead of breaking the event. * fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400 * fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band * fix(proxy): encode non-JSON guardrail details in early stream error frames An early guardrail block whose detail holds a value JSON cannot encode sent 400 headers and then cut the body. The error frame now writes such a value as its string form, the same as the mid-stream event. The keepalive test now runs on a virtual clock instead of real sleeps. * fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload. * fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client * fix(proxy): print CLI stream errors without Rich highlighting or emoji codes * test(proxy): cover the CLI's plain-text 400 error print * test(proxy): annotate the new client stream test locals as Final * fix(proxy): send the failure hook replacement on Responses streams and before the first chunk * fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception * fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it A non-HTTP failure whose hook returns an HTTP error was raised before the first chunk, which create_response turned into the hook's status but which a bare StreamingResponse route such as /queue/chat/completions answered as a 200 with zero bytes. Only the original HTTP errors raise now. The replacement goes out as the stream's error event, which create_response still answers with the hook's status as a JSON error. Adds audit round 2's integration cells for failure hook replacements on chat and Responses streams, built-in exceptions on Responses streams, and the Python client. * test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent
47 lines
1.5 KiB
Python
47 lines
1.5 KiB
Python
import io
|
|
from typing import Final
|
|
|
|
import pytest
|
|
import responses
|
|
from pydantic import JsonValue
|
|
from rich.console import Console
|
|
|
|
from litellm.proxy.client.chat import ChatClient
|
|
from litellm.proxy.client.cli.commands.chat import _stream_response
|
|
|
|
BASE_URL: Final = "http://localhost:8000"
|
|
MESSAGE: Final = "blocked [/pii] :warning: content in [prompt_injection] after 42 tokens"
|
|
PARTIAL_THEN_ERROR: Final = (
|
|
b'data: {"choices": [{"delta": {"content": "Hel"}}]}\n\n'
|
|
+ f'data: {{"error": {{"message": "{MESSAGE}"}}}}\n\n'.encode()
|
|
+ b"data: [DONE]\n\n"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"status, body, json_body",
|
|
[
|
|
(200, PARTIAL_THEN_ERROR, None),
|
|
(400, None, {"error": {"message": MESSAGE}}),
|
|
(400, MESSAGE.encode(), None),
|
|
],
|
|
ids=["error frame mid-stream", "json error before the stream", "text error before the stream"],
|
|
)
|
|
@responses.activate
|
|
def test_stream_response_prints_the_error_message_as_plain_text(
|
|
status: int, body: bytes | None, json_body: JsonValue
|
|
) -> None:
|
|
responses.add(responses.POST, f"{BASE_URL}/chat/completions", body=body, json=json_body, status=status)
|
|
output: Final = io.StringIO()
|
|
|
|
content: Final = _stream_response(
|
|
Console(file=output, width=200, force_terminal=True, color_system="truecolor"),
|
|
ChatClient(base_url=BASE_URL),
|
|
"gpt-4",
|
|
[{"role": "user", "content": "hi"}],
|
|
0.7,
|
|
None,
|
|
)
|
|
|
|
assert content is None
|
|
assert MESSAGE in output.getvalue()
|