mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-27 06:13:07 +00:00
41890bdf64
* fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(tokenizer): isolate HuggingFace tokenizer caches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): clear the traceback before re-raising a cached tokenizer load error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): retry failed HuggingFace tokenizer loads after the cache window Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): start tokenizer retry TTL after a failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): lock every HuggingFace tokenizer load and keep load failures as values Concurrent llama-2, llama-3 and cohere counts each started their own failed download because only the bundled tokenizers took a load lock. Every tokenizer kind now has one, so a burst during an outage makes a single attempt and two counts can no longer read the same expiring entry at once. A failed load is now remembered as a small value with the error message instead of the error object, so re-raising it no longer keeps the first failing request's frames alive. Adds tests for the concurrent outage and for fork-guard errors staying raised and unremembered. * fix(utils): keep Hugging Face tokenizer downloads unlocked and warn once per failed download Reverts the per-kind load lock so concurrent first downloads still run in parallel, as #47353 intends. A short lock around the failure cache read and write stops two counts racing on an expiring entry, and the tiktoken fallback warning now fires only when a download fails, not on every count that reuses the remembered failure. * fix(caching): read an entry another reader just expired as a cache miss Two threads reading the same expiring key in InMemoryCache could both pass the membership check, and the second then raised KeyError after the first removed it. get_cache now treats that key as a miss. This replaces the extra lock the tokenizer failure cache took, and the regression test forces the interleaving without timing. * fix(caching): treat a TTL another reader just removed as not expired InMemoryCache's expiry check read the TTL after a separate membership check, so a reader racing another reader's eviction raised KeyError. It now reads the TTL once and treats a missing one as not expired, and get_cache already reads the vanished entry as a miss. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_azure_blob_cache.py | ||
| test_cache_preset_key.py | ||
| test_caching.py | ||
| test_caching_handler.py | ||
| test_check_and_fix_namespace_none_guard.py | ||
| test_disk_cache.py | ||
| test_dual_cache.py | ||
| test_embedding_router.py | ||
| test_evicted_client_closer.py | ||
| test_gcs_cache.py | ||
| test_in_memory_cache.py | ||
| test_llm_caching_handler.py | ||
| test_llm_client_cache_e2e.py | ||
| test_qdrant_semantic_cache.py | ||
| test_redis_batch.py | ||
| test_redis_cache.py | ||
| test_redis_cluster_cache.py | ||
| test_redis_cluster_node_isolation.py | ||
| test_redis_connection_pool.py | ||
| test_redis_semantic_cache.py | ||
| test_request_redis_batch_post_call.py | ||
| test_request_redis_batch_pre_call.py | ||
| test_responses_stream_cache_keys.py | ||
| test_s3_cache.py | ||
| test_unit_test_caching.py | ||
| test_valkey_semantic_cache.py | ||