scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-11 22:51:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 123

add: comment above optimization

Browse source 
For anybody that would change this value for whatever reason, the comment makes the tradeoff clear.
This commit is contained in:
AlexsanderHamir 2025-11-17 12:59:48 -08:00
parent f1895265e6
commit 7241b4e9b5
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
litellm/proxy/proxy_server.py
View file

@ -5236,6 +5236,10 @@ async def moderations(
async def _audio_speech_chunk_generator(
_response: HttpxBinaryResponseContent,
) -> AsyncGenerator[bytes, None]:
# chunk_size has a big impact on latency, it can't be too small or too large
# too small: latency is high
# too large: latency is low, but memory usage is high
# 8192 is a good compromise
_generator = await _response.aiter_bytes(chunk_size=8192)
async for chunk in _generator:
yield chunk