mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-16 09:22:22 +00:00
* fix(proxy): close streams before rebuilding the served output for logging The streaming iterator hook awaited _record_served_stream_output before it returned, so every streamed response stayed open after its last event while the proxy re-parsed all served chunks. That held the connection about 11 ms plus 25 ms per 1,000 output tokens, and SDK clients wait for the close. The parked deferred logging is now claimed at stream end and run in a rooted background task after the served output is recorded, so logging still sees the served text and the client sees the stream end right away. * fix(proxy): cap background served-output rebuilds and fall back to waiting past it Each background rebuild holds its served chunks until it runs, and the stream's concurrency slot is already released. Past 64 pending rebuilds a stream waits for its own rebuild as it did before, so overload keeps its backpressure and no spend logging is dropped. |
||
|---|---|---|
| .. | ||
| helpers | ||
| prisma_and_spend | ||
| proxy_logging | ||
| __init__.py | ||