mirror of
https://github.com/zotero/zotero.git
synced 2026-10-11 03:38:25 +00:00
836e4b7cbc
- added environment to run embedding models locally (transformers.js, ONNX Runtime WASM binary, etc.). The actual inference execution happens in a separate worker environment (worker.js) - added local itemEmbeddings table to store embeddings locally - in advanced preferences, one can select two options for semantic search model: english and multilingual. English model (bge-small-en-v1.5) is better for english-only corpus but multilingual (multilingual-e5-small) is necessary to handle abstracts with any other language than english. We can add more language-specific models as needed. - when the model is selected, Zotero.Embeddings.download will download the model (quantized ~100mb) and store it locally. - Zotero.Embeddings.Indexing will start a process to index all regular items with title+abstract. It happens in batches and takes some time. The progress will appear in the advanced preferences pane. Embeddings are inserted into itemEmbeddings SQL table. For now, the table is local only, no syncing is involved. - when embedding model pref is set to "Disabled", the model is deleted and embeddings table is cleared. - when an embedding model is selected, quick search dropdown has a new "Similarity" mode, which will run semantic search on the current scope of items. - semantic search does not clearly define what counts as "relevant" and what is "not relevant". In addition, it will change depending on the library and query. So we cannot semantically filter out items the way it is done via SQL. Semantic search returns the ranking but items cannot be sorted because it is done by the itemTree based on column selection. So in "similarity" quicksearch mode, there is also a dropdown to select how many top relevant items to keep (top 5 - top 100). It allows the user to keep the most relevant items depending on the context, without conflicting with itemTree sorting. - semantic search happens in-memory. On a large 5K library it's fast, but we could consider sqlite-vec extension if needed.
6 lines
326 B
Text
6 lines
326 B
Text
*.sql text eol=lf
|
|
app/win/zotero.exe.tar.xz filter=lfs diff=lfs merge=lfs -text
|
|
app/mac/updater.tar.xz filter=lfs diff=lfs merge=lfs -text
|
|
app/win/updater.exe.tar.xz filter=lfs diff=lfs merge=lfs -text
|
|
app/linux/updater.tar.xz filter=lfs diff=lfs merge=lfs -text
|
|
resource/embeddings/*.wasm filter=lfs diff=lfs merge=lfs -text
|