litellm/tests/unit/proxy/utils
maciej-berri 8366e15be0
fix(proxy): close streams before rebuilding the served output for logging (#51487)
* fix(proxy): close streams before rebuilding the served output for logging

The streaming iterator hook awaited _record_served_stream_output before it
returned, so every streamed response stayed open after its last event while
the proxy re-parsed all served chunks. That held the connection about 11 ms
plus 25 ms per 1,000 output tokens, and SDK clients wait for the close.

The parked deferred logging is now claimed at stream end and run in a rooted
background task after the served output is recorded, so logging still sees
the served text and the client sees the stream end right away.

* fix(proxy): cap background served-output rebuilds and fall back to waiting past it

Each background rebuild holds its served chunks until it runs, and the stream's
concurrency slot is already released. Past 64 pending rebuilds a stream waits
for its own rebuild as it did before, so overload keeps its backpressure and
no spend logging is dropped.
2026-11-14 22:32:48 +00:00
..
helpers fix(responses): run the guardrails a Responses WebSocket frame selects (#48725) 2026-10-30 14:12:51 -07:00
prisma_and_spend fix(proxy): retain unwritten rollups after database disconnects (#51706) 2026-11-13 13:36:12 -08:00
proxy_logging fix(proxy): close streams before rebuilding the served output for logging (#51487) 2026-11-14 22:32:48 +00:00
__init__.py test(proxy): move utils, agent_endpoints and endpoint tests into tests/unit/proxy (#44006) 2026-10-01 11:06:42 -07:00