Add a clientVersion column for items, collections, searches, and
libraries, incremented once per library per transaction on every
object save or deletion. The local API reports these versions instead
of synced versions -- in object JSON, format=versions, since=
filtering, and Last-Modified-Version -- since synced versions don't
reflect local changes and are 0 for unsynced objects. Group metadata
responses keep reporting the synced group version, which has no local
counterpart.
---------
Co-authored-by: Dan Stillman <dstillman@zotero.org>
Someone ended up (via a plugin, presumably) with stored-file attachments
with a full path after 'storage:', which broke file syncing. Throw when
setting a stored-file path containing a slash, and strip paths from
existing filenames in a schema update step. No particular reason to
think that the file with that basename will exist in the storage dir,
but at least it will be looking for the right file and not be totally
broken.
Separately, the dataserver will clean up filenames with full paths and
block going forward.
https://forums.zotero.org/discussion/132822/reference-sychronization-error
The condition `required` flag was removed in #5962, but the column was
kept so older clients could still read the database. The full-text
search changes bump the userdata compatibility version, locking out
those clients, so the column can now be dropped.
Index attachment content into a contentless trigram FTS5 table in a
separate, attached fulltext.sqlite, normalized so matching is accent-
and case-insensitive. For content containing CJK characters, a companion
'ascii'-tokenized table holds bigrams so 1-2 character CJK queries, which
the trigram tokenizer can't match, still work. The extracted text still
lives in the .zotero-ft-cache files, so the index is fully derived and
rebuildable.
Use the FTS index for the fulltextContent condition, falling back to the
cached-text scan for queries too short to index, and point quick
search's content matching at the FTS index in place of the now-removed
word index. (One side effect: quick search now matches attachment
content by substring rather than by word.)
Already-extracted content is migrated into the index at startup, slowing
down on active usage. A background queue then extracts not-yet-indexed
attachments gradually when Zotero is idle. Attachments with no local
file or full-text content are recorded as missing. Content downloaded
via sync is processed into the index immediately when the sync finishes,
rather than waiting for idle like it did before, so it's searchable
immediately in on-demand file-download mode.
The index DB is tied to the main DB via the local user key and rebuilt
if they don't match (e.g., after a delete-and-resync). We compact it by
running FTS5's 'optimize' command once the indexing queue drains, and we
vacuum the attached database when necessary to reclaim disk space.
Closes#2038, #2044
Addresses #1595
Search now ignores accents, so "seance" matches "séance" and vice versa.
Text is normalized with Unicode NFKD compatibility decomposition (which
also handles typographic ligatures, superscripts, full-width forms,
etc.) plus a small map for letters NFKD leaves alone (ø, œ, æ, ß, ...)
and the fraction slash, via Z.Utilities.Internal.normalizeForSearch().
The HTML tags we support in item fields are stripped, so markup isn't
matched (#81). Typographic quotes (#29, #1876) and dashes are folded to
ASCII.
Each searchable column gets a normalized shadow column --
itemDataValues.valueNormalized, tags.nameNormalized,
creators.firstNameNormalized/lastNameNormalized, and
itemAnnotations.textNormalized/commentNormalized -- populated at write
time and matched via COALESCE(normalized, raw) LIKE. NULL is stored when
normalizing only changes case, so plain-ASCII values are only stored
once. This covers the contains/doesNotContain/beginsWith operators in
both quick search and Advanced Search.
The new columns are local-only derived data and aren't synced. Older
clients will ignore them, so this doesn't break DB compatibility.
Existing rows are backfilled after the startup sync by
Zotero.Schema.populateNormalizedSearchColumns(), which should only take
a few seconds on most databases.
Closes#29, #81, #1300, #1876
Track when attachments are last opened or read, storing a `lastRead` Unix timestamp on the attachment. For user library items, `lastRead` syncs as an attachment property in item JSON. For group library items, it syncs via a per-user synced setting (like `lastPageIndex`).
- Add `lastRead` column to `itemAttachments`
- Add `AttachmentReadObserver` to update `lastRead` on file open and page change (throttled to 5 min for page changes)
- Add "Recently Read" virtual collection (items read in last 14 days, sorted by `lastRead` descending)
- Add `lastRead` search condition with date operators
- Add `lastRead` item tree column with new `dependsOnChildren` property for parent item aggregation
- Add `getItemLastRead()` to return max `lastRead` across child attachments
Also:
- Generalize collection tree SCSS to support universal (context-fill) icons alongside themed icons
---------
Co-authored-by: Dan Stillman <dstillman@zotero.org>
And pass both `authorName` and `lastCreatedByUser` to the PDF reader.
The former can either come from `createdByUser` or be set directly on
the item (for group annotations dragged to personal libraries).
- We were updating global schema before migrating userdata, but a 4 → 5
upgrade involved a system.sql version bump, which wiped out itemTypes,
causing 'annotation' to not exist after the upgrade. This moves global
schema updates after userdata migration and bumps the global schema
version to repair DBs that were already upgraded and broken.
- A system.sql bump without a global schema update would result in empty
tables. This moves the global-schema-related tables to userdata.sql.
- The DB integrity check before userdata updates added in 5b9e6497a
could fail when coming from an older DB, because the checks assume
current schema. An integrity check is now done after a userdata update.
(We were already skipping the new table/index reconciliation stuff. If
old DBs are discovered to have problems that would cause a migration
step to fail, we'll fix those explicitly in the steps.)
Also:
- Make sure `version` is `versionNumber` in the `fields` table. It was
changed with a system.sql bump in 5.0, but hard-coded fields were later
removed from system.sql in favor of schema.json, meaning that anyone who
upgraded from 4.0 after that would never have `version` removed and so
would have both fields (one from before and one from schema.json).
The page index needs to be per-person in group libraries, and it should
still work in read-only libraries, so it doesn't make sense to store it
on the item. This uses a synced setting in the user's library instead.
```
var noteContents = item.note; // was item.getNote()
var schemaVersion = item.noteSchemaVersion;
item.setNote(contents) // default to Zotero.Notes.schemaVersion
item.setNote(contents, schemaVersion) - explicit version
```
This lays the groundwork for moving collections and searches to the
trash instead of deleting them outright. We're not doing that yet, so
the `deleted` property will never be set (except for items), but this
will allow clients from this point forward to sync collections and
searches with that property for when it's used in the future. For now,
such objects will just be hidden from the collections pane as if they
had been deleted.
Include flags in repo update checks indicating whether a database table
that was previously supposed to be removed was removed and whether a new
table was created. This should help us figure out whether we can safely
perform schema update steps or whether we need to figure out why some
schema update steps aren't being applied.
Prior to fdfa8052d1, it was possible to create an item in a feed
library using Add Item by Identifier. If you did that, and then copied
it to a collection in your personal library, it would somehow end up
with an owl:sameAs relation to a `/users/local` URI (probably because
the URI functions don't work properly on a feed library).
This will clean up such relations in a schema update.
This changes the way item types, item fields, creator types, and CSL
mappings are defined and handled, in preparation for updated types and
fields.
Instead of being predefined in SQL files or code, type/field info is
read from a bundled JSON file shared with other parts of the Zotero
ecosystem [1], referred to as the "global schema". Updates to the
bundled schema file are automatically applied to the database at first
run, allowing changes to be made consistently across apps.
When syncing, invalid JSON properties are now rejected instead of being
ignored and processed later, which will allow for schema changes to be
made without causing problems in existing clients. We considered many
alternative approaches, but this approach is by far the simplest,
safest, and most transparent to the user.
For now, there are no actual changes to types and fields, since we'll
first need to do a sync cut-off for earlier versions that don't reject
invalid properties.
For third-party code, the main change is that type and field IDs should
no longer be hard-coded, since they may not be consistent in new
installs. For example, code should use `Zotero.ItemTypes.getID('note')`
instead of hard-coding `1`.
[1] https://github.com/zotero/zotero-schema
Separate flags for hiding the retraction altogether and for hiding
citation warnings for it
New functions:
Zotero.Retractions.hideRetraction(item)
Zotero.Retractions.shouldShowCitationWarning(item)
Zotero.Retractions.disableCitationWarningsForItem(item)
Addresses #1710
This shouldn't be possible, but there've been a couple reports of people
ending up on version 103 without the table, so create it again with IF
NOT EXISTS. This is obviously a bad fix, but until we know how this
happened it's the best we can do.
- Check for retracted items using data from Retraction Watch
- Show an X next to retracted items in the items list, and show a
scary message at the top of the item pane with more info and links.
- Lookup is done in a privacy-preserving manner using k-anonymity --
the server is unable to determine the specific items that exist in
the client, so people who don't sync don't need to share any library
data (though the server doesn't log the lookups anyway).
TODO:
- Pop up an alert when new items are found
- Show a confirmation prompt when citing a retracted item
- Support items without DOIs or PMIDs
- Add a proper PMID field and expand DOI to more item types so these
values don't need to be parsed out of Extra
- Clear the banner immediately when all possible fields are cleared
instead of waiting a few seconds
A submitted database had a text userID with a trailing "A0. Not sure how
that happened -- it doesn't appear to be possible in current code -- but
it caused group permissions not to be properly synced.
If somebody switched accounts in a previous version, it was apparently
possible for related items to end up pointing at an item URI with the
old userID, which could cause a 403 on sync.
https://forums.zotero.org/discussion/70989/report-id-477331252
(5.0 deletes data when switching accounts to avoid exactly this sort of
bug.)
These should've all been replaced with dc:replaces relations in a schema
update step, so any that exist were likely synced down from the API
(since fixed) and should be obsolete/redundant.
This also adds a mechanism for indicating that DB userdata upgrades
after a certain version are minor and shouldn't show "Upgrading
database…" or create a pre-upgrade backup.
Instead of My Publications being a separate library, have it be a
special collection inside My Library. Top-level items can be dragged
into it as before, and child items can be toggled off and on with a
button in the item pane. Newly added child items won't be shown by
default.
For upgraders, items in the My Publications library will be moved into
My Library, which might result in their being duplicated if the items
weren't removed from My Library. The client will then upload those new
items into My Library.
The API endpoint will continue to show items in the separate My
Publications library until My Publications items are added to My
Library, so the profile page will continue to show them.
* Mark feedItems read in a single batch SQL update
* Automatically remove old feed items
* User-facing preference globally and per-feed for feed item expiration
Before 5.0 we performed a regexp on new item data values to determine if
they were integers and saved them natively in SQLite if so. We no longer
do that, but setField() used strict equality when checking for changes,
so an item could be marked as changed when comparing to a new string
value (e.g., from a write response from the API, which always returns
strings). To avoid that, this converts all old values in the DB to
strings and saves all incoming values as strings automatically. (This
should also help with searching and some other things.)
