mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-10 08:28:25 +00:00
150f4b706b
* fix(router): count embedding batches per input in pre-call checks Embedding input lists went through the Responses input transform, which calls .get() on each item, so every list-input /embeddings call logged "failed to count tokens" and skipped the context window check. Lists of texts, token IDs or token ID lists now count as their longest input, since an embedding model's context window applies to each input on its own * fix(router): keep empty and boolean input lists on the Responses count path An empty list carries no embedding evidence, and counting it as a zero token batch skipped the instructions and tools of an empty Responses input. Boolean lists are not token IDs * fix(router): honor disable_token_counter for token ID embedding input Text inputs already counted as zero through token_counter when token counting is disabled, while token ID inputs were still measured by length and could be rejected. The router reads the flag through its dynamic config like its other process settings * refactor(router): type the embedding count helper with a read-only sequence and final locals * fix(router): count embedding batches per input on the media-free text projection The per-input count now runs on the text projection from the embedding media split, so data URIs a deployment embeds as media keep the media path and only text items are checked against the context window. A caption next to an image is checked on its own, and a text-only deployment still counts an image data URI as text * fix(router): keep embedding images out of the pre-call count only on image backends, and skip tokenizing inputs that fit by bytes Media leaves the embedding count only when every deployment in the group declares image embedding input, so a text-only backend still counts a data URI as text and an image backend outside the TPM media list no longer gets a false 400. A text no longer than the smallest window in UTF-8 bytes counts as its byte length without the tokenizer, since a token is never shorter than a byte. * fix(router): keep embedding images out of the pre-call count when any backend in the group embeds images Mark Cohere embed-english-light-v3.0 as embedding images, as its v3 siblings are. * fix(router): check embedding text batches only with an exact OpenAI tokenizer litellm's default tokenizer counts non-English text several times higher than non-OpenAI embedding providers do, so counting batches with it refused inputs the provider accepts. Text batches on those groups now stay unchecked as at the merge base, without the error log. Main's per-provider media split and the cost map are restored, so strings and embedding media keep main's path. * fix(router): count long embedding batches in slices so chat counts do not wait behind them A text batch is now tokenized in slices of 65,536 characters, each its own offloaded call, so the FIFO count limiter can serve a chat count between slices. The no-OpenAI-tokenizer rule for text lists moved into the pre-call check, so RoutingContext.count_pre_call_check_tokens counts a text list by its longest text. * fix(router): read embedding media off the event loop and wait on the count limiter in the fairness test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| core | ||
| __init__.py | ||
| test_enforce_model_rate_limits.py | ||
| test_io_token_rate_limits.py | ||
| test_router.py | ||
| test_router_callback_hook_sequence.py | ||
| test_router_endpoints.py | ||
| test_router_entry_points.py | ||
| test_router_facade.py | ||
| test_router_fallbacks.py | ||
| test_router_fuzz_regressions_rust_only.py | ||
| test_router_get_deployments.py | ||
| test_router_helper_utils.py | ||
| test_router_host.py | ||
| test_router_max_parallel_requests.py | ||
| test_router_provider_endpoints.py | ||
| test_router_retries.py | ||
| test_router_surface_check.py | ||
| test_router_timeout.py | ||
| test_router_vector_store_endpoints.py | ||