mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
490d7dd983
Streaming requests through the Rust bridge previously buffered the full provider response and faked the SSE stream on the Python side, so the first chunk only arrived after the last. Add a native streaming path: - core: SseFrameStream reassembles upstream bytes into complete SSE frames (blank-line delimited, keep-alive frames dropped, trailing partial frame preserved); messages_stream_frames exposes it and forces stream:true upstream so a host cannot misuse the entrypoint. - bridge: amessages_stream returns a MessagesStream async iterator yielding one complete SSE frame as bytes per __anext__ (recv_text pattern: per-call future_into_py, pull-based backpressure). Cancelling __anext__ drops only the frame fetch; dropping the iterator aborts the upstream request; mid-stream failures map to exceptions then terminate iteration. trace=true is rejected (no terminal response). - python: rust_bridge.messages_stream wrapper + aclose-aware adapter; streaming in llm_http_handler now routes through the native stream gate when eligible (same rust gates + agentic-hook fallback) and the fake-stream path is removed. - delete the orphaned duplicate routes/runtime.rs (never declared). |
||
|---|---|---|
| .. | ||
| ai-gateway | ||
| CODING_STANDARDS | ||
| core | ||
| python-bridge | ||
| python-interop | ||