mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
The AWS event stream framer now yields aws_smithy_types Message directly instead of copying headers and payload into a second struct, which halves its allocations (16 to 8 per frame) and cuts wall time by about 30%. It also parses from a slice and advances the buffer rather than splitting and freezing it. SSE parsing no longer goes through sse-stream. The new parser follows the WHATWG event stream grammar (unknown fields ignored, colonless lines are empty fields, last event line wins, non-digit retry and NUL ids ignored, events without data are not dispatched), caps an event at MAX_FRAME_BYTES so an unterminated upstream cannot grow memory without bound, and reports upstream body failures as Error::Body like the AWS path does. Allocations stay at two per event and wall time is within a few percent. The Framer trait had no generic consumer, so both parsers are plain functions now. MissingStreamData in litellm-core is gone because a dispatched SSE event always carries data. |
||
|---|---|---|
| .. | ||
| auth | ||
| auth-aws | ||
| auth-azure | ||
| auth-gcp | ||
| cache | ||
| cache-memory | ||
| cache-redis | ||
| core | ||
| framer | ||
| python-bridge | ||
| python-interop | ||
| token-counter | ||