mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
The AWS event stream framer now yields aws_smithy_types Message directly instead of copying headers and payload into a second struct, which halves its allocations (16 to 8 per frame) and cuts wall time by about 30%. It also parses from a slice and advances the buffer rather than splitting and freezing it. SSE parsing no longer goes through sse-stream. The new parser follows the WHATWG event stream grammar (unknown fields ignored, colonless lines are empty fields, last event line wins, non-digit retry and NUL ids ignored, events without data are not dispatched), caps an event at MAX_FRAME_BYTES so an unterminated upstream cannot grow memory without bound, and reports upstream body failures as Error::Body like the AWS path does. Allocations stay at two per event and wall time is within a few percent. The Framer trait had no generic consumer, so both parsers are plain functions now. MissingStreamData in litellm-core is gone because a dispatched SSE event always carries data. |
||
|---|---|---|
| .. | ||
| crates | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||