mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
f1c4a40fab
POST /v1/batches could hang indefinitely. BatchRateLimiter.async_pre_call_hook runs inline in the request path and, for keys with applicable rpm/tpm limits, read the input file to count tokens with no deadline. With none set the OpenAI SDK default applied (600s, max_retries=2), so a slow or stalled Files API held the request open far past any client timeout; 63.6s was observed on stage against a 60s client read timeout. The read does double duty: it counts tokens for rate limiting, and it validates every body.model in the JSONL against the caller's allowlist. Those have opposite safe defaults, so the timeout policy splits on whether the key needs that check. A key restricted to a subset of models is rejected, because admitting it unchecked grants exactly the bypass _should_skip_batch_input_file_processing refuses to allow via operator config. A key with unrestricted access is admitted unmetered, matching the existing fail-open, so a degraded Files API does not become an outage. The deadline is passed to afile_content as well as to wait_for. afile_content runs the sync client via run_in_executor, and cancelling that await does not interrupt a thread already in the pool, so bounding only the await would leak the worker until the SDK's own timeout fired. Also unskips the e2e test that guards the LIT-3266 unattributed-spend-row regression, which was blocked on this hang. Defaults to 10s; override with general_settings.batch_input_file_read_timeout. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||