If queued items can't be indexed and removed for any reason, the
startup and background drains would otherwise run forever. Stop after
three passes without the queues shrinking; the items are picked up
again on the next trigger.
If a queued attachment was missing its cache file and re-extraction
produced no text, nothing was recorded in the index state, so the
content-index queue selected the same items forever and the startup
drain never finished.
https://forums.zotero.org/discussion/132808/
Say "Attachments indexed"/"Partially indexed" instead of
"Indexed"/"Partial", reverse "File or full-text content not
available", and list notes last so the attachment rows are contiguous.
The progress popup was opened with alwaysontop, which is system-wide,
so a long-running popup (e.g., full-text indexing) stayed on top of
other apps' windows after switching away from Zotero. Open it as a
dependent window instead, which keeps it above only the Zotero window.
A term mixing CJK with non-ASCII words (e.g., Cyrillic plus Japanese)
was routed to the CJK index with only its CJK characters, matching
every document that contained those. Treat any non-CJK letter or digit
as making a term mixed-script, which falls back to a literal scan of
the cached text.
Replace the trigram FTS5 index for attachment content with a unicode61
word index, so terms match whole words with the final token as a prefix
("archive" matches "archives", but "ion" doesn't match "condition"), as
in the pre-FTS5 word index. A multi-word phrase gets adjacent-token
candidates from the index and is then verified against the cached text
of just those items, since FTS5 ignores what separates adjacent tokens;
the verification treats whitespace and hyphen runs as equivalent
(they're frequently extraction layout or styling) but requires other
punctuation to match literally. Notes keep the trigram index and CJK
matching is unchanged; the index database version is bumped so the
index is rebuilt.
Follow-up to #5979
JNA 5.x ships per-architecture macOS native libraries instead of a
single com/sun/jna/darwin/ directory, so the notarization re-signing
step failed with "filename not matched".
This step can be removed entirely once the plugin ships a jna.jar
without the macOS natives, which are never loaded.
SQLite replays a leftover -wal file (e.g., from a force-quit) into
whatever file next occupies the database path, so copying a backup
over zotero.sqlite produced a corrupted-database error, and the
automatic restore recreated the same mismatch and failed every time.
When corruption is detected, check if the database file is valid without
its journal files, and if so, save a verified copy and restart, swapping
the copy in at the next startup before the database is reopened. (The
copy is made before shutdown because SQLite automatically checkpoints
the WAL into the database file when the last connection closes, which
would write the stale WAL data into the file.) Otherwise, move journal
files along with the .damaged file to clear them from the main path
before restoring from the automatic backup or creating a new database.
Since a mismatched WAL can cause subtle data damage without errors, also
run a full integrity check at startup after an unclean shutdown. Skip
the explicit close-time WAL checkpoint once corruption has been flagged,
and truncate the WAL during idle maintenance to limit stale WAL data.
showZoteroPaneProgressMeter() now returns a token capturing the
previous message, meter mode and position, and owner, and
restoreZoteroPaneProgressMeter() restores that state if no other
operation has changed the display since, with nested tokens restoring
in reverse order.
The CSL locales repository replaced locales-sr-RS.xml with script
variants, so citing with a stored or default 'sr-RS' locale loaded no
terms and failed with an et-al error. Resolve unavailable locales to
the closest available CSL locale (sr-Cyrl-RS for sr-RS) when rendering
and in locale selectors.
https://forums.zotero.org/discussion/comment/515598/#Comment_515598
Backup copies preserve the database file's mtime, and the backup
interval is measured from the backup file's mtime, so a forced
pre-update backup of a long-idle database could be rotated out a day
early. Regular backups still keep the database mtime, which the
unchanged-database check relies on.
The backup before repairing integrity-check errors from the Advanced
pane has thrown a TypeError since backUpDatabase() started requiring an
options object in 8.0, aborting the repair. Also force the backup so a
recent automatic backup can't cause it to be skipped.
If a schema update committed the userdata upgrade but kept failing in a
later step, each restart forced another rotation backup of the
now-upgraded database, deleting the pre-upgrade backup after numBackups
restarts. Record the pending target versions in the database when
making a forced backup and skip the backup while they're unchanged, so
the pre-update backup survives the normal rotation period.
Replace the manually maintained minorUpdateFrom flag, which was easy to
forget to update when adding a slow upgrade step and couldn't account
for database size or hardware. The message is now shown for any upgrade
still running after 500 ms.
Versioned backups (zotero.sqlite.<version>.bak) were kept until the
next userdata upgrade, potentially bloating the data directory by
gigabytes, while minor upgrades made no backup at all. Now any userdata
upgrade, integrity check, or global schema update forces a rotation
backup. Since versioned backups ignored backup.numBackups, setting that
to 0 now fully disables backups, including before upgrades.
Show indexing progress as a bar (items in the search index out of those
that will be) that resolves to "up to date", alongside
Indexed/Partial/Notes counts and a "File or full-text content not
available" count for attachments with no local file or full-text
content. While the pane is open, trigger indexing directly so it
advances as the user watches.
Remove the Rebuild Index and Clear Index buttons. Rebuild Index marked
all content unsynced and re-uploaded it, triggering a server reindex and
re-download on other devices, and the per-item reindex paths now cover
what needed it -- raising a length limit re-extracts only the affected
items, and unindexed items are indexed automatically. Clear Index wiped
the local index and extracted-text cache files, a destructive action
that shouldn't be exposed.
Note content is indexed into fulltext.sqlite, making note searches
accent- and case-insensitive and matching the note's plain text rather
than its HTML markup. To avoid re-indexing on every auto-save, a save
flags the note for background indexing, and searches match a flagged
note from its normalized text in memory until it's indexed.
Closes#378
The condition `required` flag was removed in #5962, but the column was
kept so older clients could still read the database. The full-text
search changes bump the userdata compatibility version, locking out
those clients, so the column can now be dropped.
Index attachment content into a contentless trigram FTS5 table in a
separate, attached fulltext.sqlite, normalized so matching is accent-
and case-insensitive. For content containing CJK characters, a companion
'ascii'-tokenized table holds bigrams so 1-2 character CJK queries, which
the trigram tokenizer can't match, still work. The extracted text still
lives in the .zotero-ft-cache files, so the index is fully derived and
rebuildable.
Use the FTS index for the fulltextContent condition, falling back to the
cached-text scan for queries too short to index, and point quick
search's content matching at the FTS index in place of the now-removed
word index. (One side effect: quick search now matches attachment
content by substring rather than by word.)
Already-extracted content is migrated into the index at startup, slowing
down on active usage. A background queue then extracts not-yet-indexed
attachments gradually when Zotero is idle. Attachments with no local
file or full-text content are recorded as missing. Content downloaded
via sync is processed into the index immediately when the sync finishes,
rather than waiting for idle like it did before, so it's searchable
immediately in on-demand file-download mode.
The index DB is tied to the main DB via the local user key and rebuilt
if they don't match (e.g., after a delete-and-resync). We compact it by
running FTS5's 'optimize' command once the indexing queue drains, and we
vacuum the attached database when necessary to reclaim disk space.
Closes#2038, #2044
Addresses #1595
Code that attaches its own database to the main connection needs to hook
into that connection's lifecycle. Add onIdle() to run maintenance (e.g.,
vacuuming) during the main database's idle maintenance, and
addCorruptionHandler() to handle a corruption error for the attached
database (after confirming that the main database is actually corrupt,
so a corrupt attached database doesn't trigger main-database recovery).
Load a bundled SQLite extension (e.g., FTS5) by name. mozStorage
disables generic extension loading but allows specific bundled ones.
Extensions are registered per connection, so a loaded extension is
remembered and re-loaded automatically after a reconnect, before
onConnect() callbacks run.
Search now ignores accents, so "seance" matches "séance" and vice versa.
Text is normalized with Unicode NFKD compatibility decomposition (which
also handles typographic ligatures, superscripts, full-width forms,
etc.) plus a small map for letters NFKD leaves alone (ø, œ, æ, ß, ...)
and the fraction slash, via Z.Utilities.Internal.normalizeForSearch().
The HTML tags we support in item fields are stripped, so markup isn't
matched (#81). Typographic quotes (#29, #1876) and dashes are folded to
ASCII.
Each searchable column gets a normalized shadow column --
itemDataValues.valueNormalized, tags.nameNormalized,
creators.firstNameNormalized/lastNameNormalized, and
itemAnnotations.textNormalized/commentNormalized -- populated at write
time and matched via COALESCE(normalized, raw) LIKE. NULL is stored when
normalizing only changes case, so plain-ASCII values are only stored
once. This covers the contains/doesNotContain/beginsWith operators in
both quick search and Advanced Search.
The new columns are local-only derived data and aren't synced. Older
clients will ignore them, so this doesn't break DB compatibility.
Existing rows are backfilled after the startup sync by
Zotero.Schema.populateNormalizedSearchColumns(), which should only take
a few seconds on most databases.
Closes#29, #81, #1300, #1876
Rename the sync-delivered full-text content processor's methods and
state to "sync content" (registerSyncContentProcessor,
processSyncedContent, indexSyncedContent, getSyncedContentCacheFile,
etc.), so it reads distinctly from the local index-building queues added
for content search. No behavior change.
The document-worker bump raised SDT_PROCESSOR_VERSIONS.pdf from 2 to 3,
so the test's "valid" pack fixture (embedding version 2) was rejected as
stale-processor during generation, failing "should generate the pack when
missing."
viewAttachment() fired the 'open' notification before launchFile() saved
a sniffed content type, so AttachmentReadObserver's concurrent lastRead
save could reload the item and discard the pending change, leaving the
attachment with a blank or incorrect content type.
The #focus() Tab/Shift-Tab tests wait on focus/blur events, which fire
only while the window is active. It intermittently loses activation in
CI's Xvfb, so call win.focus() in beforeEach when the window isn't
active.
Temporary diagnostic for the intermittent shift-tab focus-test timeout.
A central afterEach in runtests.js tracks when
Services.focus.activeWindow stops being the main window; the shift-tab
test logs the last-active and first-inactive test in its timeout
message, to identify what leaves the pane inactive in CI.
This reverts commit 03a610e696.
activate() foregrounds a window via the window manager, which CI's Xvfb
doesn't have, so activateZoteroPane() in beforeEach just hangs there
instead of fixing anything. Restore the temporary open-window logging to
try to catch what leaves the pane window inactive.
An image pasted into a note is stored as a hidden attachment, so a
matching search condition (e.g., "# of Tags" is 0) would pull the
image's parent item into the results with no visible match.
Notes and attachments are counted on regular items, and annotations on an
attachment or across a regular item's attachments; other rows are excluded
rather than always matching with a count of 0. Trashed children aren't
counted.
