Versioned backups (zotero.sqlite.<version>.bak) were kept until the
next userdata upgrade, potentially bloating the data directory by
gigabytes, while minor upgrades made no backup at all. Now any userdata
upgrade, integrity check, or global schema update forces a rotation
backup. Since versioned backups ignored backup.numBackups, setting that
to 0 now fully disables backups, including before upgrades.
Show indexing progress as a bar (items in the search index out of those
that will be) that resolves to "up to date", alongside
Indexed/Partial/Notes counts and a "File or full-text content not
available" count for attachments with no local file or full-text
content. While the pane is open, trigger indexing directly so it
advances as the user watches.
Remove the Rebuild Index and Clear Index buttons. Rebuild Index marked
all content unsynced and re-uploaded it, triggering a server reindex and
re-download on other devices, and the per-item reindex paths now cover
what needed it -- raising a length limit re-extracts only the affected
items, and unindexed items are indexed automatically. Clear Index wiped
the local index and extracted-text cache files, a destructive action
that shouldn't be exposed.
Note content is indexed into fulltext.sqlite, making note searches
accent- and case-insensitive and matching the note's plain text rather
than its HTML markup. To avoid re-indexing on every auto-save, a save
flags the note for background indexing, and searches match a flagged
note from its normalized text in memory until it's indexed.
Closes#378
The condition `required` flag was removed in #5962, but the column was
kept so older clients could still read the database. The full-text
search changes bump the userdata compatibility version, locking out
those clients, so the column can now be dropped.
Index attachment content into a contentless trigram FTS5 table in a
separate, attached fulltext.sqlite, normalized so matching is accent-
and case-insensitive. For content containing CJK characters, a companion
'ascii'-tokenized table holds bigrams so 1-2 character CJK queries, which
the trigram tokenizer can't match, still work. The extracted text still
lives in the .zotero-ft-cache files, so the index is fully derived and
rebuildable.
Use the FTS index for the fulltextContent condition, falling back to the
cached-text scan for queries too short to index, and point quick
search's content matching at the FTS index in place of the now-removed
word index. (One side effect: quick search now matches attachment
content by substring rather than by word.)
Already-extracted content is migrated into the index at startup, slowing
down on active usage. A background queue then extracts not-yet-indexed
attachments gradually when Zotero is idle. Attachments with no local
file or full-text content are recorded as missing. Content downloaded
via sync is processed into the index immediately when the sync finishes,
rather than waiting for idle like it did before, so it's searchable
immediately in on-demand file-download mode.
The index DB is tied to the main DB via the local user key and rebuilt
if they don't match (e.g., after a delete-and-resync). We compact it by
running FTS5's 'optimize' command once the indexing queue drains, and we
vacuum the attached database when necessary to reclaim disk space.
Closes#2038, #2044
Addresses #1595
Code that attaches its own database to the main connection needs to hook
into that connection's lifecycle. Add onIdle() to run maintenance (e.g.,
vacuuming) during the main database's idle maintenance, and
addCorruptionHandler() to handle a corruption error for the attached
database (after confirming that the main database is actually corrupt,
so a corrupt attached database doesn't trigger main-database recovery).
Load a bundled SQLite extension (e.g., FTS5) by name. mozStorage
disables generic extension loading but allows specific bundled ones.
Extensions are registered per connection, so a loaded extension is
remembered and re-loaded automatically after a reconnect, before
onConnect() callbacks run.
Search now ignores accents, so "seance" matches "séance" and vice versa.
Text is normalized with Unicode NFKD compatibility decomposition (which
also handles typographic ligatures, superscripts, full-width forms,
etc.) plus a small map for letters NFKD leaves alone (ø, œ, æ, ß, ...)
and the fraction slash, via Z.Utilities.Internal.normalizeForSearch().
The HTML tags we support in item fields are stripped, so markup isn't
matched (#81). Typographic quotes (#29, #1876) and dashes are folded to
ASCII.
Each searchable column gets a normalized shadow column --
itemDataValues.valueNormalized, tags.nameNormalized,
creators.firstNameNormalized/lastNameNormalized, and
itemAnnotations.textNormalized/commentNormalized -- populated at write
time and matched via COALESCE(normalized, raw) LIKE. NULL is stored when
normalizing only changes case, so plain-ASCII values are only stored
once. This covers the contains/doesNotContain/beginsWith operators in
both quick search and Advanced Search.
The new columns are local-only derived data and aren't synced. Older
clients will ignore them, so this doesn't break DB compatibility.
Existing rows are backfilled after the startup sync by
Zotero.Schema.populateNormalizedSearchColumns(), which should only take
a few seconds on most databases.
Closes#29, #81, #1300, #1876
Rename the sync-delivered full-text content processor's methods and
state to "sync content" (registerSyncContentProcessor,
processSyncedContent, indexSyncedContent, getSyncedContentCacheFile,
etc.), so it reads distinctly from the local index-building queues added
for content search. No behavior change.
The document-worker bump raised SDT_PROCESSOR_VERSIONS.pdf from 2 to 3,
so the test's "valid" pack fixture (embedding version 2) was rejected as
stale-processor during generation, failing "should generate the pack when
missing."
viewAttachment() fired the 'open' notification before launchFile() saved
a sniffed content type, so AttachmentReadObserver's concurrent lastRead
save could reload the item and discard the pending change, leaving the
attachment with a blank or incorrect content type.
The #focus() Tab/Shift-Tab tests wait on focus/blur events, which fire
only while the window is active. It intermittently loses activation in
CI's Xvfb, so call win.focus() in beforeEach when the window isn't
active.
Temporary diagnostic for the intermittent shift-tab focus-test timeout.
A central afterEach in runtests.js tracks when
Services.focus.activeWindow stops being the main window; the shift-tab
test logs the last-active and first-inactive test in its timeout
message, to identify what leaves the pane inactive in CI.
This reverts commit 03a610e696.
activate() foregrounds a window via the window manager, which CI's Xvfb
doesn't have, so activateZoteroPane() in beforeEach just hangs there
instead of fixing anything. Restore the temporary open-window logging to
try to catch what leaves the pane window inactive.
An image pasted into a note is stored as a hidden attachment, so a
matching search condition (e.g., "# of Tags" is 0) would pull the
image's parent item into the results with no visible match.
Notes and attachments are counted on regular items, and annotations on an
attachment or across a regular item's attachments; other rows are excluded
rather than always matching with a count of 0. Trashed children aren't
counted.
The front-most tab's item appeared under Selected Items instead of Open
Documents, and the library tab's actual selection wasn't shown. Selected
Items now always reflects the item tree selection, and an item that's both
selected and open stays under Open Documents (sorted after the front-most
tab) so that suggestions specific to open documents (e.g., the current page
number, planned in #6001) can apply to it. Notes open in tabs now appear
under Open Documents when adding a note.
Opening the Advanced Search from the quick search cleared the search
text only on the first selected row, while the quick search had applied
it to all of them, so with a multi-collection selection the other
collections stayed filtered by the now-invisible text -- both while the
advanced search was open and after closing it.
With the saved-search editor open, Cmd/Ctrl-Shift-F replaced it with the
temporary Advanced Search pane and the search field's close button
closed it, in both cases discarding the editor's working copy with no
way to recover the edits. Move the collection-switch save prompt into a
shared helper and show it before replacing or closing the editor.
Choosing Save on a collection switch now continues the switch directly
instead of relying on the save's modify notification to re-sync the view.
onConditionSelected() populates the Author value menu asynchronously,
and until it finished, the row's default textbox was still the visible
value control, so serializing the row (running or saving the search)
replaced the loaded author with an empty value. Serialize the stored
value while the menu is pending, and invalidate the population if a
newer condition selection takes over meanwhile.
removeRow() remembered the removed row's index within its own group,
but when the removal emptied the group and pruned it, the index was
applied to the ancestor's children, focusing an unrelated row. Track the
pruned group's slot in its parent instead.
The pane-level Enter handler ran the default action (save in the saved
editor, submit in the temporary pane) regardless of the event target, so
Enter with focus on the Cancel button saved and closed the editor, and
Enter on Clear ran the search. Click the focused button instead, which
the platform otherwise only does for Space.
The "in the same annotation" binding menu was hidden as soon as a group
no longer had two conditions at the bound level, but the group's stored
binding was still saved and applied. After switching one of two bound
annotation conditions to another type, the group looked like a plain
"Match all of the following" group but still required a single
annotation to satisfy all of its conditions, and no control was left to
see or clear the binding.
Show the menu whenever the group is bound, drop the binding only once no
condition at its level remains, and refresh the menus before
collectGroup() in updateSearch() so the first save after such an edit
doesn't serialize stale state.
setAdvancedSearch() invalidated the row unconditionally, unlike
setSearch()/setTags(), so merely opening or closing the Advanced Search
pane (which applies a null filter) re-queried and rebuilt the items
list. A passed search is still always applied, since its conditions may
have been edited in place, and setFilter() still waits for a refresh
already in flight so that callers see the settled view.
_rollUpAnyToLevel() only followed an annotation's parent when the
attachment itself had a parent, so in a search for top-level items, a
tag on a standalone attachment's annotation matched nothing.
The condition seeded by _loadConditions() used `mode: undefined`, but
toJSON() only omits the "/mode" suffix when the mode is exactly false
(what parseCondition() returns), so a saved search migrated from the
obsolete childNote condition serialized -- and synced -- the condition
as "resultLevel/undefined".
_buildQuery() already uses the first top-level joinMode for the
full-text post-filter; with multiple top-level joinMode conditions (only
creatable via the API), combineConditions() used the last.
Extends existing handling for tag selection and quicksearch to handle
advanced search as well.
Fixes#5996 (not just for zotero://select but for all selectItems()
callers)
Instead of just prefixing the labels with "-", indent the whole row,
including the icon. This looks better and fixes FAYT on subcollection
names.
https://forums.zotero.org/discussion/132561/
The #focus() Tab/Shift-Tab tests dispatch synthetic keydowns and wait on
a focusin event to sync with the collections search field's async
(setTimeout) focus. Gecko only fires focus/blur events while the window
is active, so when the pane window wasn't active -- it opens in the
background on macOS/Windows and can lose activation on Linux/CI -- the
focusin never fired and the test hung until Mocha's timeout. Activate
the window in beforeEach, as the item pane and tags box focus tests
already do.
Also removes the temporary open-window logging added in e1f19f34f5.
