Update the global schema to 45, resolve a field's date type through its
base-field mapping in ItemFields.isDate() (to cover priorityDate), and
convert stored values of date-type fields to multipart dates on schema
upgrade.
Add a clientVersion column for items, collections, searches, and
libraries, incremented once per library per transaction on every
object save or deletion. The local API reports these versions instead
of synced versions -- in object JSON, format=versions, since=
filtering, and Last-Modified-Version -- since synced versions don't
reflect local changes and are 0 for unsynced objects. Group metadata
responses keep reporting the synced group version, which has no local
counterpart.
---------
Co-authored-by: Dan Stillman <dstillman@zotero.org>
Someone ended up (via a plugin, presumably) with stored-file attachments
with a full path after 'storage:', which broke file syncing. Throw when
setting a stored-file path containing a slash, and strip paths from
existing filenames in a schema update step. No particular reason to
think that the file with that basename will exist in the storage dir,
but at least it will be looking for the right file and not be totally
broken.
Separately, the dataserver will clean up filenames with full paths and
block going forward.
https://forums.zotero.org/discussion/132822/reference-sychronization-error
Accented stop-words weren't matched by the existing unaccented
entries (e.g., "fur"), so automatic journal abbreviation kept and
capitalized them: "Jahrbuch für Heimatkunde" became "Jahrb. Für
Heimatkunde" instead of "Jahrb. Heimatkunde".
https://groups.google.com/g/zotero-dev/c/uP18QEKe2JU/m/AsoGWxd9AwAJ
The condition `required` flag was removed in #5962, but the column was
kept so older clients could still read the database. The full-text
search changes bump the userdata compatibility version, locking out
those clients, so the column can now be dropped.
Index attachment content into a contentless trigram FTS5 table in a
separate, attached fulltext.sqlite, normalized so matching is accent-
and case-insensitive. For content containing CJK characters, a companion
'ascii'-tokenized table holds bigrams so 1-2 character CJK queries, which
the trigram tokenizer can't match, still work. The extracted text still
lives in the .zotero-ft-cache files, so the index is fully derived and
rebuildable.
Use the FTS index for the fulltextContent condition, falling back to the
cached-text scan for queries too short to index, and point quick
search's content matching at the FTS index in place of the now-removed
word index. (One side effect: quick search now matches attachment
content by substring rather than by word.)
Already-extracted content is migrated into the index at startup, slowing
down on active usage. A background queue then extracts not-yet-indexed
attachments gradually when Zotero is idle. Attachments with no local
file or full-text content are recorded as missing. Content downloaded
via sync is processed into the index immediately when the sync finishes,
rather than waiting for idle like it did before, so it's searchable
immediately in on-demand file-download mode.
The index DB is tied to the main DB via the local user key and rebuilt
if they don't match (e.g., after a delete-and-resync). We compact it by
running FTS5's 'optimize' command once the indexing queue drains, and we
vacuum the attached database when necessary to reclaim disk space.
Closes#2038, #2044
Addresses #1595
Search now ignores accents, so "seance" matches "séance" and vice versa.
Text is normalized with Unicode NFKD compatibility decomposition (which
also handles typographic ligatures, superscripts, full-width forms,
etc.) plus a small map for letters NFKD leaves alone (ø, œ, æ, ß, ...)
and the fraction slash, via Z.Utilities.Internal.normalizeForSearch().
The HTML tags we support in item fields are stripped, so markup isn't
matched (#81). Typographic quotes (#29, #1876) and dashes are folded to
ASCII.
Each searchable column gets a normalized shadow column --
itemDataValues.valueNormalized, tags.nameNormalized,
creators.firstNameNormalized/lastNameNormalized, and
itemAnnotations.textNormalized/commentNormalized -- populated at write
time and matched via COALESCE(normalized, raw) LIKE. NULL is stored when
normalizing only changes case, so plain-ASCII values are only stored
once. This covers the contains/doesNotContain/beginsWith operators in
both quick search and Advanced Search.
The new columns are local-only derived data and aren't synced. Older
clients will ignore them, so this doesn't break DB compatibility.
Existing rows are backfilled after the startup sync by
Zotero.Schema.populateNormalizedSearchColumns(), which should only take
a few seconds on most databases.
Closes#29, #81, #1300, #1876
example:
"Atmospheric Chemistry and Physics" is currently incorrectly abbreviated as "Atmospheric Chem. Phys." should be abbreviated "Atmos. Chem. Phys."
Track when attachments are last opened or read, storing a `lastRead` Unix timestamp on the attachment. For user library items, `lastRead` syncs as an attachment property in item JSON. For group library items, it syncs via a per-user synced setting (like `lastPageIndex`).
- Add `lastRead` column to `itemAttachments`
- Add `AttachmentReadObserver` to update `lastRead` on file open and page change (throttled to 5 min for page changes)
- Add "Recently Read" virtual collection (items read in last 14 days, sorted by `lastRead` descending)
- Add `lastRead` search condition with date operators
- Add `lastRead` item tree column with new `dependsOnChildren` property for parent item aggregation
- Add `getItemLastRead()` to return max `lastRead` across child attachments
Also:
- Generalize collection tree SCSS to support universal (context-fill) icons alongside themed icons
---------
Co-authored-by: Dan Stillman <dstillman@zotero.org>
After citation-style-language/styles#7928 renamed Vancouver styles to
NLM terminology, Zotero installations end up with both vancouver.csl
and nlm-citation-sequence.csl. To fix, on init, delete any installed style
whose ID appears in the renamed-styles mapping if the target style
also exists.
---------
Co-authored-by: Dan Stillman <dstillman@zotero.org>
