mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
quietjules
|
d037d86a94
|
fix(token_counter)!: keep a loaded Hugging Face tokenizer when a slower download fails (#48916)
* fix(utils): keep a loaded Hugging Face tokenizer over a racing failure and retry once per window A download that fails after another count loaded the same tokenizer no longer hides it, the first count after the retry window claims the retry while the others keep using tiktoken, and HUGGINGFACE_TOKENIZER_LOAD_RETRY_SECONDS defaults to 60s to match the proxy's tokenizer reuse window * fix(utils): keep Hugging Face tokenizer outcomes until their retry time instead of a TTL The outcome store no longer expires entries after twice the retry window, so after a long idle stretch the first count still claims the one retry and the other counts keep using tiktoken instead of each downloading again. The store is a small lock-guarded holder of an immutable mapping keyed by tokenizer kind and backend * fix(token_counter): keep the loaded tokenizer in its outcome and run one retry at a time The loaded outcome now holds the tokenizer, so an LRU eviction can no longer send every count back to an unguarded download. A retry blocks further retries until it ends, even when it runs past the window, and a download that fails after another count loaded the tokenizer counts with that tokenizer * fix(token_counter): retry a failed Hugging Face tokenizer download in the background After the retry window, the count that finds the stored failure starts one background download and answers with tiktoken, so no live count waits on a retry. A failure from a download that started before the first failure no longer resets a running retry, and a failed retry never replaces a tokenizer another download loaded. The outcome store takes its clock and retry starter as dependencies, so the tests inject them instead of patching time.monotonic. * fix(token_counter): keep retrying a failed Hugging Face tokenizer download when its retry thread cannot start If starting the background retry thread raises, the count answers with tiktoken and the failure is stored with a fresh retry time, so the next window retries again instead of the tokenizer staying on tiktoken until restart. * fix(token_counter): let a forked process or a crashed retry thread retry a failed Hugging Face tokenizer download again A failed download's in-flight retry mark is now the pid of the process running it, so a process forked during a retry starts its own instead of waiting on a thread it does not have. The retry thread clears its mark in a finally, so a native panic or fork guard error escaping it leaves the next window free to retry. The fork guard error is no longer swallowed in the retry thread and reaches the thread's exception hook, as on the first load * fix(token_counter): let a process that exits during a background tokenizer retry exit cleanly instead of aborting --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
41890bdf64
|
fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download (#47586)
* fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(tokenizer): isolate HuggingFace tokenizer caches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): clear the traceback before re-raising a cached tokenizer load error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): retry failed HuggingFace tokenizer loads after the cache window Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): start tokenizer retry TTL after a failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): lock every HuggingFace tokenizer load and keep load failures as values Concurrent llama-2, llama-3 and cohere counts each started their own failed download because only the bundled tokenizers took a load lock. Every tokenizer kind now has one, so a burst during an outage makes a single attempt and two counts can no longer read the same expiring entry at once. A failed load is now remembered as a small value with the error message instead of the error object, so re-raising it no longer keeps the first failing request's frames alive. Adds tests for the concurrent outage and for fork-guard errors staying raised and unremembered. * fix(utils): keep Hugging Face tokenizer downloads unlocked and warn once per failed download Reverts the per-kind load lock so concurrent first downloads still run in parallel, as #47353 intends. A short lock around the failure cache read and write stops two counts racing on an expiring entry, and the tiktoken fallback warning now fires only when a download fails, not on every count that reuses the remembered failure. * fix(caching): read an entry another reader just expired as a cache miss Two threads reading the same expiring key in InMemoryCache could both pass the membership check, and the second then raised KeyError after the first removed it. get_cache now treats that key as a miss. This replaces the extra lock the tokenizer failure cache took, and the regression test forces the interleaving without timing. * fix(caching): treat a TTL another reader just removed as not expired InMemoryCache's expiry check read the TTL after a separate membership check, so a reader racing another reader's eviction raised KeyError. It now reads the TTL once and treats a missing one as not expired, and get_cache already reads the vanished entry as a miss. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e0398455d0
|
test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)
* test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
fa2c8984ba
|
test: move the unit half of 126 mixed legacy files into tests/unit (#45090)
* test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai> |