The toolkit's default menuitem padding is em-based, so it balloons
with the UI font size in menulist dropdowns (e.g., the item type
menu), which inherit that font, forcing long lists to scroll. macOS
uses a fixed value on all menus and is unaffected. Match that on
Windows and Linux, slightly roomier, so every menu stays compact
regardless of font size.
https://forums.zotero.org/discussion/132797/request-for-compact-view-in-dropdown-menus
allowJavaScript: false didn't disable scripts for a system-principal
document, such as a blob: URL created from chrome code, which runs
scripts regardless of the allowJavascript flag. Sandbox the document
instead, forcing a null principal and blocking script execution.
The retraction checker fired a separate 'refresh' notification for
every matched item on each check, including already-known retractions
that hadn't changed. At startup this produced a burst of events that
forced observers like the item tree to rebuild repeatedly. Batch each
operation's changes into a single notification, and notify only for
items whose retraction state actually changed.
The candidate scan counted items whose Extra content parsed as fields
but couldn't actually be migrated (e.g., fields already set on the
item), so the window could appear on every global schema update without
changing anything.
Also use a dedicated headline instead of "Upgrading database…", and show
the window only if the migration is still running after 500 ms.
A stored file's path is normally 'storage:<filename>', which is parsed
as a string, but some ancient libraries apparently have relative paths
('../.../foo.pdf') that fell through to PathUtils.filename() and threw
NS_ERROR_FILE_UNRECOGNIZED_PATH, breaking syncing.
Resolve stored-file leaves as strings regardless of prefix and reserve
PathUtils.filename() for linked files, whose paths are genuine absolute
paths.
https://forums.zotero.org/discussion/132861/zotero-sync-issue-id-1987012678
The connector is now a Safari web extension, so the bundled appex is a
static wrapper stub whose placeholder resources are replaced at build
time with a connector build via $SAFARI_EXT_RESOURCES. The stub only
needs to be rebuilt when the native wrapper project changes, not for
connector releases.
ee68452b treated any backslash as a directory separator, in both the
attachmentPath setter and the userdata 128 migration. But backslashes
are technically valid on Linux/macOS and appear in real filenames (due
to LaTeX in titles, etc.), so that would wrongly throw on or truncate
them. Only a forward slash (never valid in a filename) or a Windows
absolute path (drive-letter or UNC prefix) reliably indicates a leaked
directory path, so treat just those as paths and leave bare backslashes
alone. We should make sure we're running getValidFileName() everywhere
to avoid saving those.
These conditions expanded into an OR-group across their underlying
fields, so a "does not contain"/"is not" operator matched almost every
item: any item missing one of the fields satisfied that field's negated
condition. Use an AND-group for negative operators, so the value must be
absent from every field.
https://forums.zotero.org/discussion/132835/
Selecting all libraries pulled in feed rows, which belong to a different
visibility group and so can't be shown together, collapsing the
selection back to a single row. Constrain the all-libraries scope to the
current selection's visibility group. Also preventDefault() the shortcut
so it no longer falls through to the cmd_selectAll key command, which
threw NS_ERROR_FAILURE.
https://forums.zotero.org/discussion/132843/beta-cannot-select-all-libraries
The download-rename test used a backslash as its sample invalid
character, which now throws in the attachmentFilename setter. Use a
colon instead, which still tests download-time filtering.
Someone ended up (via a plugin, presumably) with stored-file attachments
with a full path after 'storage:', which broke file syncing. Throw when
setting a stored-file path containing a slash, and strip paths from
existing filenames in a schema update step. No particular reason to
think that the file with that basename will exist in the storage dir,
but at least it will be looking for the right file and not be totally
broken.
Separately, the dataserver will clean up filenames with full paths and
block going forward.
https://forums.zotero.org/discussion/132822/reference-sychronization-error
Accented stop-words weren't matched by the existing unaccented
entries (e.g., "fur"), so automatic journal abbreviation kept and
capitalized them: "Jahrbuch für Heimatkunde" became "Jahrb. Für
Heimatkunde" instead of "Jahrb. Heimatkunde".
https://groups.google.com/g/zotero-dev/c/uP18QEKe2JU/m/AsoGWxd9AwAJ
If queued items can't be indexed and removed for any reason, the
startup and background drains would otherwise run forever. Stop after
three passes without the queues shrinking; the items are picked up
again on the next trigger.
If a queued attachment was missing its cache file and re-extraction
produced no text, nothing was recorded in the index state, so the
content-index queue selected the same items forever and the startup
drain never finished.
https://forums.zotero.org/discussion/132808/
Say "Attachments indexed"/"Partially indexed" instead of
"Indexed"/"Partial", reverse "File or full-text content not
available", and list notes last so the attachment rows are contiguous.
The progress popup was opened with alwaysontop, which is system-wide,
so a long-running popup (e.g., full-text indexing) stayed on top of
other apps' windows after switching away from Zotero. Open it as a
dependent window instead, which keeps it above only the Zotero window.
A term mixing CJK with non-ASCII words (e.g., Cyrillic plus Japanese)
was routed to the CJK index with only its CJK characters, matching
every document that contained those. Treat any non-CJK letter or digit
as making a term mixed-script, which falls back to a literal scan of
the cached text.
Replace the trigram FTS5 index for attachment content with a unicode61
word index, so terms match whole words with the final token as a prefix
("archive" matches "archives", but "ion" doesn't match "condition"), as
in the pre-FTS5 word index. A multi-word phrase gets adjacent-token
candidates from the index and is then verified against the cached text
of just those items, since FTS5 ignores what separates adjacent tokens;
the verification treats whitespace and hyphen runs as equivalent
(they're frequently extraction layout or styling) but requires other
punctuation to match literally. Notes keep the trigram index and CJK
matching is unchanged; the index database version is bumped so the
index is rebuilt.
Follow-up to #5979
JNA 5.x ships per-architecture macOS native libraries instead of a
single com/sun/jna/darwin/ directory, so the notarization re-signing
step failed with "filename not matched".
This step can be removed entirely once the plugin ships a jna.jar
without the macOS natives, which are never loaded.
SQLite replays a leftover -wal file (e.g., from a force-quit) into
whatever file next occupies the database path, so copying a backup
over zotero.sqlite produced a corrupted-database error, and the
automatic restore recreated the same mismatch and failed every time.
When corruption is detected, check if the database file is valid without
its journal files, and if so, save a verified copy and restart, swapping
the copy in at the next startup before the database is reopened. (The
copy is made before shutdown because SQLite automatically checkpoints
the WAL into the database file when the last connection closes, which
would write the stale WAL data into the file.) Otherwise, move journal
files along with the .damaged file to clear them from the main path
before restoring from the automatic backup or creating a new database.
Since a mismatched WAL can cause subtle data damage without errors, also
run a full integrity check at startup after an unclean shutdown. Skip
the explicit close-time WAL checkpoint once corruption has been flagged,
and truncate the WAL during idle maintenance to limit stale WAL data.
showZoteroPaneProgressMeter() now returns a token capturing the
previous message, meter mode and position, and owner, and
restoreZoteroPaneProgressMeter() restores that state if no other
operation has changed the display since, with nested tokens restoring
in reverse order.
The CSL locales repository replaced locales-sr-RS.xml with script
variants, so citing with a stored or default 'sr-RS' locale loaded no
terms and failed with an et-al error. Resolve unavailable locales to
the closest available CSL locale (sr-Cyrl-RS for sr-RS) when rendering
and in locale selectors.
https://forums.zotero.org/discussion/comment/515598/#Comment_515598
Backup copies preserve the database file's mtime, and the backup
interval is measured from the backup file's mtime, so a forced
pre-update backup of a long-idle database could be rotated out a day
early. Regular backups still keep the database mtime, which the
unchanged-database check relies on.
The backup before repairing integrity-check errors from the Advanced
pane has thrown a TypeError since backUpDatabase() started requiring an
options object in 8.0, aborting the repair. Also force the backup so a
recent automatic backup can't cause it to be skipped.
If a schema update committed the userdata upgrade but kept failing in a
later step, each restart forced another rotation backup of the
now-upgraded database, deleting the pre-upgrade backup after numBackups
restarts. Record the pending target versions in the database when
making a forced backup and skip the backup while they're unchanged, so
the pre-update backup survives the normal rotation period.
Replace the manually maintained minorUpdateFrom flag, which was easy to
forget to update when adding a slow upgrade step and couldn't account
for database size or hardware. The message is now shown for any upgrade
still running after 500 ms.
Versioned backups (zotero.sqlite.<version>.bak) were kept until the
next userdata upgrade, potentially bloating the data directory by
gigabytes, while minor upgrades made no backup at all. Now any userdata
upgrade, integrity check, or global schema update forces a rotation
backup. Since versioned backups ignored backup.numBackups, setting that
to 0 now fully disables backups, including before upgrades.
Show indexing progress as a bar (items in the search index out of those
that will be) that resolves to "up to date", alongside
Indexed/Partial/Notes counts and a "File or full-text content not
available" count for attachments with no local file or full-text
content. While the pane is open, trigger indexing directly so it
advances as the user watches.
Remove the Rebuild Index and Clear Index buttons. Rebuild Index marked
all content unsynced and re-uploaded it, triggering a server reindex and
re-download on other devices, and the per-item reindex paths now cover
what needed it -- raising a length limit re-extracts only the affected
items, and unindexed items are indexed automatically. Clear Index wiped
the local index and extracted-text cache files, a destructive action
that shouldn't be exposed.
Note content is indexed into fulltext.sqlite, making note searches
accent- and case-insensitive and matching the note's plain text rather
than its HTML markup. To avoid re-indexing on every auto-save, a save
flags the note for background indexing, and searches match a flagged
note from its normalized text in memory until it's indexed.
Closes#378
The condition `required` flag was removed in #5962, but the column was
kept so older clients could still read the database. The full-text
search changes bump the userdata compatibility version, locking out
those clients, so the column can now be dropped.
Index attachment content into a contentless trigram FTS5 table in a
separate, attached fulltext.sqlite, normalized so matching is accent-
and case-insensitive. For content containing CJK characters, a companion
'ascii'-tokenized table holds bigrams so 1-2 character CJK queries, which
the trigram tokenizer can't match, still work. The extracted text still
lives in the .zotero-ft-cache files, so the index is fully derived and
rebuildable.
Use the FTS index for the fulltextContent condition, falling back to the
cached-text scan for queries too short to index, and point quick
search's content matching at the FTS index in place of the now-removed
word index. (One side effect: quick search now matches attachment
content by substring rather than by word.)
Already-extracted content is migrated into the index at startup, slowing
down on active usage. A background queue then extracts not-yet-indexed
attachments gradually when Zotero is idle. Attachments with no local
file or full-text content are recorded as missing. Content downloaded
via sync is processed into the index immediately when the sync finishes,
rather than waiting for idle like it did before, so it's searchable
immediately in on-demand file-download mode.
The index DB is tied to the main DB via the local user key and rebuilt
if they don't match (e.g., after a delete-and-resync). We compact it by
running FTS5's 'optimize' command once the indexing queue drains, and we
vacuum the attached database when necessary to reclaim disk space.
Closes#2038, #2044
Addresses #1595
Code that attaches its own database to the main connection needs to hook
into that connection's lifecycle. Add onIdle() to run maintenance (e.g.,
vacuuming) during the main database's idle maintenance, and
addCorruptionHandler() to handle a corruption error for the attached
database (after confirming that the main database is actually corrupt,
so a corrupt attached database doesn't trigger main-database recovery).
