vacuum() and the APFS-cloning offline backup path both close and
reopen the SQLite connection, which drops all ATTACHed databases --
including the in-memory "indexing" alias used for the fulltextWords
scratch table set up in Fulltext.init(). Once the connection is
reopened, indexing queries fail with "no such table:
indexing.fulltextWords", and indexItems()'s ignoreErrors path
routes the error to logError(), so indexing silently stops working
for the rest of the session.
Add an onConnect() hook on Zotero.DBConnection for per-connection
state that doesn't persist across reopens, and use it from
Fulltext.init() to re-attach the indexing DB on each reconnect.
Regression sources:
- 67288047f3 ("Use APFS cloning for file copies on macOS") flips
online idle backups to offline-with-clone on APFS, so every idle
backup interval (24h default) closes and reopens the connection.
Affects Mac users on APFS.
- b27c4cb023 ("Enable SQLite WAL mode and add periodic VACUUM
INTO") adds the vacuum path, which closes and reopens on the
first idle period in a session that passes the freelist/time
gates. Affects all users with sufficient DB churn, roughly once
per 14 days.
https://forums.zotero.org/discussion/131576/debug-id-d848621212-indexing-of-pdfs-fail-zotero-9-0-3https://forums.zotero.org/discussion/131718/possible-bug-regression-report-search-unusable-on-macos-zotero-9-0-3-with-large-library
(cherry picked from commit 70c4cadab1)
Add Zotero.File.copyFile(), which uses clonefile() on APFS with a
fallback to IOUtils.copy(), and use it for all significant file copies.
APFS is detected and cached for each parent folder via
Zotero.File.isAPFS(), which uses statfs().
On APFS, all database backups now use the offline (close/clone/reopen)
path instead of the SQLite online backup API. Cloning is nearly
instant, and backup files share disk blocks via copy-on-write, saving
potentially gigabytes of space.
Closes#5330
A new function, `Zotero.DB.backUpDatabase()`, by default makes offline
backups, by closing the database, doing a regular file copy, and
reopening the database. It takes an options object with an `online`
flag to make online, incremental backups that can take multiple minutes
to complete, though for now we're still running them on idle.
The old function, `Zotero.DB.backupDatabase(suffix, force)`, is
deprecated and proxies to the new function, making offline backups.
Fixes#4935
- Create userdata tables and indexes that are missing
- Delete tables and triggers that should no longer exist
- Run schema integrity check before user data migration
- Run schema integrity check after restart error
This is meant to address two problems:
1) Database damage, and subsequent use of the DB Repair Tool, that
results in missing tables
2) A small number of cases of schema update steps somehow not being
reflected in users' databases despite their having updated userdata
numbers, which are set within the same transaction. Until we figure
out how that's happening, we should start adding conditional versions
of schema update steps to the integrity check.
This is currently only running the update check after a restart error,
which might not occur for all missed schema update steps, so we might
want other triggers for calling setIntegrityCheckRequired().
- Add an 'exclusive' option to transactions that causes them to block other
transactions and wait for other transactions to finish before starting,
instead of nesting
- Resolve Zotero.DB.waitForTransaction() promise before returning from
executeTransaction()
- A side effect of the above: wait for a newly created item to be selected in
the middle pane and rendered in the right-hand pane before returning from
executeTransaction()
- Don't save items multiple times when adding/removing a non-final creator in
the Info pane
- Use a simpler, non-recursive method for focusing the next field in the Info
pane; this prevents "too much recursion" errors if something causes the
right-hand pane not to be rendered when expected
This fixes an issue where two transactions started around the same time
could run separately instead of nesting, causing the statements from one
to end up running not within a transaction
