Using tokenizer to determine exact token size of a chunk
puts a lot of work on the main thread during indexing
only to find if a chunk is over the min limit and is
under the max. There is not much value in precision
so we estimate the token value per chunk with a language
table (e.g. english ~4 chars/token, CJK ~1 char/token, etc.)
If needed precision ends up being necessary, we can
run a tokenizer on a small subset of the text in document
and derive chars/token ration from that instead of
actually tokenizing the entire document.
BestMatch session extracts only a small portion of
previews during scoring to show immediately. The rest
is derived when the browser is idle to not freeze the UI
and sent to the itemTree via onPreviewsFilled callback.
Similar approach to earlier placeholder rows - but
better performing.
Lexical matches from a longer natural language query add a good
amount of noise to search results, so hybrid mode is
sometimes even worse than just semantic. However, lexical
matches are necessary to find particular keywords -
if you search for X, the item with title X or the only
item stating what X is better show up. To reconcile
these two conflicting requirements, fuse scores
in the following way:
- if there are > 2 search terms,
the user is likely asking for a conceptual query that should
be able to produce an informative vector to find
decent matches. Run lexical search on ONLY title + abstract
to make sure that if a rare term is mentioned in metadata,
the item can surface. No fulltext lexical to not add
noise.
- if there are <= 2 search terms, it's likely that
particular terms are being looked up, so run semantic
with fulltext lexical searches
Instead of by global frequency. So if you search "size of a dinosaur"
chunks returned from a paper about dinosaurs use the weights
of "size" and "dinosaur" from within the paper, where "size"
is much rarer and more useful. Otherwise, chunks that
meantion "dinosaur" a lot are returned.
Firstly, it did not always perform well with itemTree scroll.
More importantly, semantic search does take some time -
can be 10+ seconds on a large library. A few extra seconds
to fetch all snippets should not be a big problem. And if
the tail of search results is so long that it becomes -
the fix is to trim the number of search results by
stricter relevance criteria.
Refactor of best match module to contain all best match-related
logic from collectionViewItemTree, so itemTree just
calls relevant methods when needed.
One drawback is that we can't pick the best sentence from
semantic chunk to use as a snippet because that would
mean re-embedding every chunk's sentences on search. So
instead just show the first sentence if no lexical chunks
are available.
Drop the wildcard from the last search term.
It was added for search-as-you-type and probably not
needed now.
Disabled context menu options on search result rows
fix irrelevant annotations not getting hidden
Added search results pane that is shown when a search
result row is selected. It displays the entire chunk
that the snippet is based on. When an attachment is
selected, its search results section lists all search
results
Make the chunk the single unit of a search match. Both
engines now score the same passages instead of returning
two incompatible things that had to be merged: lexical
windows cut around a matched word, and semantic chunks.
A window cut around a word doesn't know where it sits in
the document, so it could not carry a section or page, and
overlap between the two kinds had to be guessed at with a
substring probe. A chunk knows its location, which is what
a snippet header and navigating to the match both need.
Chunks come from the best source an item has: the ones the
semantic index already holds, ranked by the model or not,
and for an unindexed item the same chunking applied to its
structured text or, failing that, its flat text. Everything
above the ladder is indifferent to which rung ran.
An item returns at most three matches, blending the model's
score with how much of the query the chunk's own words
carry. Each entry keeps the whole chunk plus the extent of
the one line worth quoting, so the tree can show a line and
the item pane can read the passage without deriving twice.
Choosing that line is the expensive half -- a chunk that
only means the query has to be read by the model -- so it
happens after ranking and only for the three that survive,
asking about an item's lines in a single call.
Structured text is only used when already extracted.
Generating it costs seconds, which is not a price a preview
can charge.
Fix scrolling while matches fill in. A row asked for its
matches when painted, but a paint pass only touches rows
newly in view and each request replaces the last, so rows
still waiting were dropped and re-requested; the request now
states everything still on screen. Restoring scroll position
also snapped to a row boundary, discarding how far into the
top row the view was scrolled -- unnoticeable at 24px rows,
a visible jump backwards at the height of a match row.
Move the chunking from Zotero.Embedding.Chunking to
Zotero.Utilities.Internal.Chunking so that it can be
shared by Zotero.Embeddings and Zotero.Lexical.
Lexical would need to chunk a document to fetch search
snippets on a per-chunk basis.
Also, helper lexical functions to score how much of a
query a piece of text carries, bm25 style.
Render search snippets in itemTree lazily, as the user
scrolls to them. Fulltext table is contentless, so we cannot
fetch snippet() for each search match. For embeddings,
we need to fetch the structured-text to locate the right
block. Both of these operations can take a long time
when done to a lot of items in _refresh before rendering,
which is why search snippets are extracted on demand.
BestMatch.Session is a new object to wrap the interaction
between the item tree and the search engines. BestMatch.Session.score
returns the search results with an indication which
of them should have search snippets. Not all search
results do - purely semantic matches on abstracts or
notes, as well as all matches on annotations get a snippet.
ItemTree renders a placeholder child row for items that will
have snippets.
Based on the matches flag above, the itemTree renders
placeholder rows. When the placeholder row is rendered,
onSearchMatchRendered is called to tell BestMatch.Session
which attachment's snippets need to be shown. BestMatch.Session
maintains a queue and handles extracting of snippets
when the browser is free to avoid freezing the main thread.
When the snippets are extracted, the placeholder row is
replaced with rows of search matches.
BestMatch.Session maintains the state of what snippets were already
extracted.
Drop search result itemPane componenets, on a new search
scroll the itemTree to the top to see the most relevant results.
Do not store the text of each embedded chunk, just store
the block start/end and character offset into the block
as indications where the chunk is in structured text.
When we need to show that snippet, re-derive it from
structured text. This way, we don't store another copy
of fulltext and save on database space. In addition,
better indication of chunk offset allows us to navigate
to matching piece of text from the snippet more accurately.
Use token budget instead of character budget to enforce
the limit on the memory usage during indexing. Tokens are
the proper measure of embedding work, and different
languages use different amount of tokens. e.g. English
is ~4 characters per token and Chinese is ~1-2, which
means that character budget can mean significantly
more indexing work depending on user's library.
With semantic search disabled, Best Match runs a purely lexical ranking:
query terms are weighted by their rarity in the user's own library, any term
can match (OR semantics), and items are scored by how much of the query they
cover and where — titles count most, then abstracts, then fulltext, notes, and annotations.
Consecutive query words also count as a unit: an item containing "special education"
as a phrase ranks above one containing the words scattered, with the phrase's
importance again measured by its rarity. Items whose evidence is a single matched
unit that is common are not included (e.g. "fall of communism" -> do no list all
items that just have "fall").
With semantic search enabled, both engines score every candidate and the two rankings
are fused with Reciprocal Rank Fusion: an item can match by its words, by its meaning,
or — ranking highest — by both. Results are the union of the two engines' matches, so
a literal match the model doesn't understand and a paraphrase the words don't catch both surface.
The relevance bar shows the item's strongest single piece of evidence, whichever engine found it.
While the semantic index is still building or switching models, searches degrade
to lexical ranking instead of showing nothing; the "not ready" error state is gone.
Previews (the Search Results section) now explain the actual ranking for any matched item,
not just attachments: excerpts of the item's own text around the literal matches,
with the matched words highlighted. When semantic chunks are available they're shown too — with
literal matches highlighted inside them — and merged with the lexical excerpts by evidence strength,
deduplicating excerpts that show the same passage.
Add the query-to-score flow to Zotero.Lexical, built on the content and
item-text indexes:
- Term statistics span both corpora: document frequency sums MATCH
counts over fulltextContent and fulltextItemText, corpus size sums
the state tables, so a term's rarity is a property of the library --
and libraries with few attachments still get real weights
- Units too common to matter are cut relative to the query's best unit
(INFORMATIVE_WEIGHT_FRACTION), so "of" is dropped next to "communism"
while an all-common query keeps its best word
- Matchers are index probes: matchContent() against the content index,
matchFields()/matchNotes()/matchAnnotations() against the item-text
columns. Notes fetch text only for probe matches plus stale/unindexed
notes (getStaleOrUnindexedNoteIDs); quoted phrases verify literally
against stored text everywhere (whitespace/hyphen runs interchange,
other punctuation must match)
- scoreItemIDs() assembles the score (see below), with a floor and
cancellation
Ranking algorithm, per query:
1. Parse into units (words, quoted phrases, CJK runs); trailing
mid-word token matches as a prefix
2. Weigh each unit by smoothed BM25 IDF from the combined corpora;
keep the informative ones
3. Match: presence (1) in titles, abstracts, annotations; saturated,
length-normalized term frequency for notes (computed from text)
and documents (recovered as rank ratios per unit -- for a one-unit
query, ranks compare documents exactly, and the strongest match
anchors 1)
4. Score = sum over units of weight x best boosted evidence across
sources (title x2, abstract x1.3; max, so one word never counts
twice), normalized against the query's ceiling: 1 = full-strength
match on everything asked; below SCORE_FLOOR is no match
Add ftindex.fulltextItemText, an FTS5 table (plus CJK 2-gram twin and
state table) holding each item's searchable text in per-type columns:
title and abstract for regular items, note text for notes, passage and
comment for annotations. One row per itemID; text stored normalized.
- Regular items and annotations index inline in their save transaction;
notes are written by the existing stale-flag queue in the same step
that feeds the trigram tables. Erase clears the entries.
- Backfill queue (processItemTextIndexQueue) covers pre-existing items
after the index rebuild, wired into the startup and background
drains; feed libraries are excluded. _indexDBVersion bumped to 3.
- Advanced prefs: "Items and annotations indexed" line in Index
Statistics; the item-text queue joins the progress/up-to-date logic
and drains while the pane is open.
Why an index: ranked search needs two things per query term that no
existing structure can answer -- whole-word membership ("fall" must not
match "rainfall") and per-word document counts, which drive term
weighting ("communism" outranks "fall" by rarity). The trigram note
index answers substrings, not words, and inflates counts; scanning
item text in JS costs a pass over the whole library per query and
can't prefilter without dropping diacritic matches. Word-level FTS
answers both with index probes, normalized at write time.
Add optional pref to index and search fulltext of attachments.
When enabled, attachment IDs are enqueue after regular items,
notes, and annotations.
Added helpers to extract outline and sections from structured
text module. During indexing, the sections of the attachment
are extracted, large sections are broken into chunks, and
small sections are combined to fit into the context window
of the model. Then, each chunk is embedded with its outline
path as the prefix and added to embeddings table.
Each row now contains full text of the chunk
and it's path - a good amount of duplication needed to
ensure that we can reliably connect the embedding of
the chunk to its text for a preview.
On search in Best Match mode, attachment rows get the score
of the highest ranking chunk, so if there is a very relevant
chunk in an attachment, the regular item with a non-relevant
abstract will still rank highly.
When the attachment row is selected, top 5 matching chunks
appear in the new search results collapsible-section of
the item pane, so one can examine matching chunks
without opening the actual reader.
The pruneModels test set the model pref without stubbing startIndexing,
so its switch chains outlived the test: a delayed disabled-model prune
deleted the shared test calibration, and a later indexing run re-measured
the real model under the fake test-model key, breaking centering in
scoreItemIDs. Only reproducible with a window loaded (full suite / CI),
since ModelHub is inert without one. Stub startIndexing and wait out
switches in the pruneModels test, and stub ensureCalibration wherever
tests run a real indexing pass.
Very short notes one or two non-descriptive words (e.g. "test")
tend to rank highly for very many irrelevant queries,
so raise the bar for indexing to have 2 words for
reguar items' title and abstract, and 3 words for
everything else.
Firefox ML engine self terminates when idle. Its status
needs to be verified in _getEngine and the old engine needs
to be shut down. Then, a new engine is created. Otherwise,
all subsequent inference calls will fail.
Move the generation of mean vectors into its own
Zotero.Embeddings.Calibration object from tests.
Now a test run is not required to get mean vector
for a newly added model. Instead, the corpus on which
we generate embeddings to calculate mean vector, as well
as the logic for actual mean vector calculation, lives
in Zotero.Embeddings.Calibration so it can be done on
demand - at the start of the first indexing pass for a
model that hasn't been measured yet.
Each passage from Zotero.Embeddings.Calibration corpus
also has a matching query, to which the passage is
an expected search result. Those queries are also embedded
during calibration, which is used to calculate the appropriate
min score cutoff and max display score for relevance bar.
Score floor is computed by embedding all queries and passages,
finding the similarity between all pairs and locating the cutoff
that only 1% of non-matching query/passage pairs exceed.
Max display score is the median of the matching pairs, so
half of genuine matches fill the bar completely.
Calculated score cutoff, max display value and mean vector
are stored in the embeddings database to be reused later.
During calibration of language-specific models, passages/queries
for languages that the model cannot handle are excluded
(e.g drop russian/chinese/spanish that english models are not meant to handle).
For this, model config expects a new optional language field.
A model that omits it is measured against the whole corpus.
Chunk size is now bounded by CHUNK_MAX_TOKENS rather than by
the model's context window, which still caps it. A model with
a large window would otherwise stop notes being chunked at all,
putting a whole note into a single vector and defeating scoring
an item by its best chunk.
With the mean vector and both score bounds measured rather than
configured, each model's hardcoded meanVector, minScore and
maxDisplayScore are gone from MODELS, along with an unused files
array. A model entry is now only facts that can be read off its
model card.
Added bge-small-zh-v1.5 as a model specifically for chinese,
as well as a number of multilingual and english models
for testing.
Notes are indexed on their text and annotations on the passage they mark
together with their comment, so both match on what they actually say
rather than on their parent's title and abstract.
A note can hold more text than a model's context window, so
Zotero.Embeddings.Chunking splits long text using the selected model's
tokenizer. Paragraphs are the topic units: two never share a chunk
unless one is too small to embed on its own, in which case small
paragraphs are combined, and a paragraph over the window is split at
sentence boundaries into even pieces. Stored embeddings are keyed by item
and chunk, so the index is rebuilt on upgrade.
An item scores as its best chunk, so a long note
that addresses a query in one paragraph isn't diluted by the rest, and
its rank reflects the best match anywhere beneath it: a strongly matching
annotation lifts its attachment and its paper. The relevance bar reports
only the row's own score, so a paper ranked by its annotation shows a
high rank over an empty bar rather than claiming to be a match it isn't.
List order and bar fill deliberately disagree in that case.
Annotation rows render the relevance cell, and the
Relevance column moves to the far right while a best-match search is
active so the bars line up across item, note, attachment and annotation
rows.
A query naming a word an item's title or abstract uses shouldn't come
back empty because the model scored the item below the floor. Text
matches are used only when the model matched nothing, ranked by their
own scores.
A one-character title carries no signal but still scores as a moderate
match against any query. A single ideograph can be a whole word, so
those are kept.
Every embedding a model produces shares a large common direction that
says nothing about the text, so items with little content scored as
moderate matches against everything: for the query "sun", an item titled
"C" outscored a paper about a coronal mass ejection. Removing that
direction spreads the scores out, so no relevance reads as no score.
The mean is a constant per model, computed over titles and abstracts
across fields and languages, and applies to stored vectors as they're
compared, so the index doesn't change. Display ranges are refit to the
scores that result.
Indexing re-enqueues every eligible item on each start to find what
changed, and checked each one's stored hash with its own query, so a
large library ran thousands of queries at startup.
Replace the bundled transformers.js/ONNX Runtime worker with a Zotero.ML
engine on the native ONNX backend, which embeds a batch of abstracts
several times faster and runs the model outside the main process. The
runtime downloads the model files and caches them in the profile
directory, so drop the model directory in the data directory along with
the code that filled it.
Size batches by the amount of text they hold rather than by a fixed item
count, since a batch of long abstracts needs far more memory than the
same number of short ones, and shrink the budget further while the
system is under memory pressure.
A best-match rerank re-ran the underlying search on every embeddings
update, even though only the ranking depends on the embeddings. Reuse
the cached search results and recompute just the ranking, except when a
top-K cutoff makes membership depend on the scores.
The item tree reran the active best-match search on every 'refresh' item
event, so unrelated bursts (e.g., full-text indexing) triggered a full
re-search and re-score. Flag the embeddings indexer's own notifications
and rerank only for those.
Changing the mode -- including disabling -- silently dropped the
stored embeddings and any other downloaded model, costing a full
reindex. Prompt first, since the menu click gives no hint of the cost.
While a best-match search runs against a partially built embeddings
index, results cover only the indexed items and can look arbitrary.
Show the indexing progress in a banner above the items list, updating
as the index fills, so incomplete results aren't mistaken for a
complete ranking.
The query is trimmed and a single pair of wrapping quotes is stripped
-- they carry no phrase semantics, since the whole query embeds as one
string. A query that normalizes to nothing (e.g., just quotes) is
treated as no search at all rather than being scored against noise.
Covers saved-search ranking and quick-search precedence, mixed
top-K/collection selections, rank-only behavior with an unavailable
index, reranking on indexer refresh events, scoped 'Match any' saves
keeping the marker at the root, numeric-operator round trips, and
concurrent query embeds sharing one worker call.
A root-level 'bestMatch' condition -- serialized as a marker like
joinMode and resultLevel -- ranks results with the Relevance column via
a "Sort results by best match for" field below the root group's
conditions, composing semantic ranking with Boolean filtering. A
best-match quick search converts to it via the Advanced Search button,
and a selected saved search's own marker activates ranking too,
overridden by an active best-match quick search.
Without a cutoff the condition is rank-only and membership is untouched
-- including while the index is unavailable -- so a saved search acts
identically as a source, and unscoreable items just sort last. With the
optional "keeping top" cutoff, carried in the marker's operator,
membership becomes the K most similar results, applied in search() so
scopes, counts, and the API see the same set. A transient search's
cutoff, which applies uniformly to every selected row, is reapplied
over the merged results so a multi-collection selection returns K
members total; a saved search's cutoff is part of its own membership
and never trims other selected rows. The query embedding is cached
in-flight, so the per-row membership passes and the merged ranking
share one worker embed.
A search could embed the query with one model's prefix on a worker
initialized for another and compare it against vectors from a third.
Tag the worker with the model version it was initialized for, make
scoring wait out an in-progress model switch, refuse to score an index
that wasn't stamped by the active model, and discard results if the
model changes mid-scoring. The items view treats a not-ready index as
an empty result rather than showing an unranked scope.
Semantic similarity has no natural relevance threshold, so instead of
asking the user to pick an arbitrary result count, show every scored
item and surface the ranking directly: a Relevance column appears and
becomes the sort while a best-match search is active, and the previous
sort and columns return when it clears. The merged results are scored
in a single pass in the row provider, so ranks are global across a
multi-collection selection, child items (attachments, notes,
annotations) rank via their top-level item, and equal scores get equal
ranks that order deterministically via the secondary sort fields. Items
without a stored embedding are filtered out.
Each cell renders the score's position within the model's display range
as a bar, so relevant results read as full and the irrelevant tail
reads as empty. The ranges are provisional per-model display constants.
Sorting uses the ranks, which are also exposed to assistive technology
and as the cell tooltip. On a focused selected row the bar
switches to white so the fill doesn't vanish into the accent selection
background.
The embeddings are a local, rebuildable, model-specific index, so they
don't belong in the main database or its backups. Follow the full-text
content index pattern: a lazily attached embeddings.sqlite versioned via
PRAGMA user_version, tied to the main database by localUserKey, with
corruption recovery and idle-maintenance vacuuming via the DBConnection
hooks. Since a cross-database foreign key isn't possible, item deletions
now clear embeddings via the notifier, and the indexed-model identity
moves from a pref into the database's meta table.
Binding is meaningful for a condition that matches at every level -- a
tag bound to an attachment means the tag is on the attachment -- but a
group carrying one lost the binding as soon as the search was
serialized, so "items with an attachment tagged foo" couldn't be built.
An exact-match condition compared with SQLite's case-sensitive '=' and
skipped the normalized shadow columns, so 'publication is "review of
finance"' missed "Review of Finance" while every other kind of search
matched it.
