mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
41890bdf64
* fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(tokenizer): isolate HuggingFace tokenizer caches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): clear the traceback before re-raising a cached tokenizer load error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): retry failed HuggingFace tokenizer loads after the cache window Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): start tokenizer retry TTL after a failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): lock every HuggingFace tokenizer load and keep load failures as values Concurrent llama-2, llama-3 and cohere counts each started their own failed download because only the bundled tokenizers took a load lock. Every tokenizer kind now has one, so a burst during an outage makes a single attempt and two counts can no longer read the same expiring entry at once. A failed load is now remembered as a small value with the error message instead of the error object, so re-raising it no longer keeps the first failing request's frames alive. Adds tests for the concurrent outage and for fork-guard errors staying raised and unremembered. * fix(utils): keep Hugging Face tokenizer downloads unlocked and warn once per failed download Reverts the per-kind load lock so concurrent first downloads still run in parallel, as #47353 intends. A short lock around the failure cache read and write stops two counts racing on an expiring entry, and the tiktoken fallback warning now fires only when a download fails, not on every count that reuses the remembered failure. * fix(caching): read an entry another reader just expired as a cache miss Two threads reading the same expiring key in InMemoryCache could both pass the membership check, and the second then raised KeyError after the first removed it. get_cache now treats that key as a miss. This replaces the extra lock the tokenizer failure cache took, and the regression test forces the interleaving without timing. * fix(caching): treat a TTL another reader just removed as not expired InMemoryCache's expiry check read the TTL after a separate membership check, so a reader racing another reader's eviction raised KeyError. It now reads the TTL once and treats a missing one as not expired, and get_cache already reads the vanished entry as a miss. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_huggingface_chat_transformation.py | ||