mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-24 05:45:11 +00:00
68a8b12ab3
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out. * fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE] A guardrail block before the first chunk is raised again, so its response stays the 400 event-stream body it is on main. A mid-stream error event now carries the failure hook's replacement error when one is returned, is followed by data: [DONE] on OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details JSON cannot encode are written as strings instead of breaking the event. * fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400 * fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band * fix(proxy): encode non-JSON guardrail details in early stream error frames An early guardrail block whose detail holds a value JSON cannot encode sent 400 headers and then cut the body. The error frame now writes such a value as its string form, the same as the mid-stream event. The keepalive test now runs on a virtual clock instead of real sleeps. * fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload. * fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client * fix(proxy): print CLI stream errors without Rich highlighting or emoji codes * test(proxy): cover the CLI's plain-text 400 error print * test(proxy): annotate the new client stream test locals as Final * fix(proxy): send the failure hook replacement on Responses streams and before the first chunk * fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception * fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it A non-HTTP failure whose hook returns an HTTP error was raised before the first chunk, which create_response turned into the hook's status but which a bare StreamingResponse route such as /queue/chat/completions answered as a 200 with zero bytes. Only the original HTTP errors raise now. The replacement goes out as the stream's error event, which create_response still answers with the hook's status as a JSON error. Adds audit round 2's integration cells for failure hook replacements on chat and Responses streams, built-in exceptions on Responses streams, and the Python client. * test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent
40 lines
1.2 KiB
Python
40 lines
1.2 KiB
Python
import asyncio
|
|
import selectors
|
|
from collections.abc import Callable
|
|
from typing import Final
|
|
|
|
|
|
class ManualClockLoop(asyncio.SelectorEventLoop):
|
|
"""An event loop whose clock moves only when the test advances it, so timeouts fire on test-controlled conditions"""
|
|
|
|
def __init__(self, selector: selectors.BaseSelector | None = None) -> None:
|
|
super().__init__(selector)
|
|
self._now = 0.0
|
|
|
|
def time(self) -> float:
|
|
return self._now
|
|
|
|
def advance(self, seconds: float) -> None:
|
|
self._now += seconds
|
|
|
|
|
|
class _AutojumpSelector(selectors.DefaultSelector):
|
|
def __init__(self, advance: Callable[[float], None]) -> None:
|
|
super().__init__()
|
|
self._advance = advance
|
|
|
|
def select(self, timeout: float | None = None) -> list[tuple[selectors.SelectorKey, int]]:
|
|
ready: Final = super().select(0)
|
|
if ready or timeout == 0:
|
|
return ready
|
|
if timeout is None:
|
|
return super().select(None)
|
|
self._advance(timeout)
|
|
return []
|
|
|
|
|
|
class AutojumpClockLoop(ManualClockLoop):
|
|
"""A manual-clock loop that jumps to the next timer only once no callback or I/O event is left to run"""
|
|
|
|
def __init__(self) -> None:
|
|
super().__init__(_AutojumpSelector(self.advance))
|