mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
Vertex AI file content retrieval downloaded the whole GCS object into memory before responding, which made large batch output files (hundreds of MB, image generation JSONL past 4 GiB) impractical to fetch through the proxy. Add BaseLLMHTTPHandler.async_retrieve_file_content_streaming, an httpx stream=True path that hands the byte iterator to the provider config through the new BaseFilesConfig.transform_file_content_stream hook and closes the response on completion, early close, and HTTP error. VertexAIFilesConfig peeks at the first JSONL row: Generate Content batch output is converted to OpenAI batch format one row at a time (content-length dropped since it changes), embeddings output stays buffered so fanned-out rows can be regrouped, and anything else passes through with the upstream content-type and content-length. vertex_ai joins FILE_CONTENT_STREAMING_PROVIDERS, so the proxy returns a StreamingResponse for it while OpenAI-compatible providers and the buffered Vertex path are unchanged. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| test_batch_guardrails.py | ||
| test_files_batch_file_validation.py | ||
| test_files_common_utils.py | ||
| test_files_endpoint.py | ||
| test_general_upload_validation.py | ||
| test_storage_backend_service.py | ||