mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-23 00:41:40 +00:00
e2e7f5f879
Vertex AI file content retrieval downloaded the whole GCS object into memory before responding, which made large batch output files (hundreds of MB, image generation JSONL past 4 GiB) impractical to fetch through the proxy. Add BaseLLMHTTPHandler.async_retrieve_file_content_streaming, an httpx stream=True path that hands the byte iterator to the provider config through the new BaseFilesConfig.transform_file_content_stream hook and closes the response on completion, early close, and HTTP error. VertexAIFilesConfig peeks at the first JSONL row: Generate Content batch output is converted to OpenAI batch format one row at a time (content-length dropped since it changes), embeddings output stays buffered so fanned-out rows can be regrouped, and anything else passes through with the upstream content-type and content-length. vertex_ai joins FILE_CONTENT_STREAMING_PROVIDERS, so the proxy returns a StreamingResponse for it while OpenAI-compatible providers and the buffered Vertex path are unchanged. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| main.py | ||
| streaming.py | ||
| types.py | ||
| utils.py | ||