mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
b2e2a38bc0
* fix(passthrough): stream non-sse passthrough responses instead of buffering in memory Non-SSE passthrough responses were fully read into proxy memory (content = await response.aread()) before the first byte reached the client. For large non-JSON bodies such as Anthropic batch results jsonl files this ballooned proxy RSS to a multiple of the file size and produced near-total TTFB dead air, letting intermediaries kill the silent connection and truncate the download. The upstream request is now sent with httpx stream semantics and the buffering decision is made from the response headers: application/json (and +json) bodies plus upstream errors keep the buffered behavior since spend logging, guardrails and managed-id rewriting inspect them, while every other 2xx body is relayed as a StreamingResponse that iterates upstream bytes without accumulating them, preserving status code and headers (including x-litellm-*) and firing the success-handler logging with response_body=None once the stream completes. * fix(passthrough): log client disconnects mid-stream and derive test client cache key from production code * test(passthrough): intercept AsyncClient.send in legacy passthrough tests and assert final wire params * test(passthrough): fail with a clear assert when the passthrough client cache scan misses |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_async_streaming_error_propagation.py | ||
| test_passthrough_main.py | ||
| test_streaming_interrupt_spend_tracking.py | ||