mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-17 04:34:25 +00:00
b025681bef
* fix(proxy): gzip responses over 16 KiB off the event loop GZipBufferedResponseMiddleware compressed every body under 1 MiB inline. gzip.compress holds the event loop for its whole run, and its cost follows how well the body compresses: an 824 KB base64 embeddings response (100 inputs, the OpenAI SDK default encoding) takes about 165 ms, so every other request on the worker waits that long, and with several in flight the compressions run back to back. Lower OFF_LOOP_SIZE_BYTES to 16 KiB so only bodies that compress in well under a millisecond stay on the loop. zlib releases the GIL, so the loop keeps serving while a worker thread compresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): pin the gzip off-loop boundary at exactly OFF_LOOP_SIZE_BYTES A body of exactly OFF_LOOP_SIZE_BYTES compresses in a worker thread, as the `<` in _compress implies. Suggested in review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): pin an embeddings-sized gzip body off the event loop and drop the source comment The off-loop test was relative to OFF_LOOP_SIZE_BYTES, so it still passed with the old 1 MiB line. An 824 KB body, the size of the 100-input base64 embeddings response in the issue, now has to compress in a worker thread. The recorder no longer mutates a list, and the threshold rationale stays in the commit history instead of a source comment. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| admission_control_middleware.py | ||
| billable_request_metrics_middleware.py | ||
| budget_reservation_release_middleware.py | ||
| gzip_middleware.py | ||
| in_flight_requests_middleware.py | ||
| per_request_root_path_middleware.py | ||
| prometheus_auth_middleware.py | ||
| redis_request_batch_middleware.py | ||
| request_size_limit_middleware.py | ||
| security_headers_middleware.py | ||