mirror of
https://github.com/zotero/zotero.git
synced 2026-09-07 08:26:14 +00:00
fbb0495f05
Using tokenizer to determine exact token size of a chunk puts a lot of work on the main thread during indexing only to find if a chunk is over the min limit and is under the max. There is not much value in precision so we estimate the token value per chunk with a language table (e.g. english ~4 chars/token, CJK ~1 char/token, etc.) If needed precision ends up being necessary, we can run a tokenizer on a small subset of the text in document and derive chars/token ration from that instead of actually tokenizing the entire document. |
||
|---|---|---|
| .. | ||
| components | ||
| content | ||
| resource | ||
| tests | ||
| chrome.manifest | ||
| runtests.sh | ||