These tested racing an offline backup against an in-progress online
backup, which wouldn't happen in practice (schema backups run at startup
before the idle observer, and someone is very unlikely to perform a DB
integrity check immediately after returning from an idle that triggered
a backup). The tests relied on fragile timing and a shared tmp file,
causing failures on Linux after the WAL checkpoint change. The tests
were already skipped on macOS after switching to APFS clones.
- Switch journal mode from DELETE to WAL for better write performance.
With EXCLUSIVE locking mode, SQLite uses heap memory for the WAL
index, avoiding an -shm file. Set synchronous=NORMAL (matching what
Mozilla uses for Places). Checkpoint WAL on database close so the
.sqlite file has all data (for copies or backups).
- Add periodic database compaction on idle (after DB backup) using
VACUUM INTO and do an atomic file swap back to zotero.sqlite if no
writes occurred during the operation. Check if vacuuming is needed
based on time interval (default 14 days) and freelist ratio (default
10% threshold).
- Disable auto_vacuum, which causes fragmentation and is unnecessary
with periodic VACUUM
- Remove the VACUUM call from the integrity check, which was always just
an awkward hack to let people trigger a VACUUM without having an
explicit button
Closes#652
Add Zotero.File.copyFile(), which uses clonefile() on APFS with a
fallback to IOUtils.copy(), and use it for all significant file copies.
APFS is detected and cached for each parent folder via
Zotero.File.isAPFS(), which uses statfs().
On APFS, all database backups now use the offline (close/clone/reopen)
path instead of the SQLite online backup API. Cloning is nearly
instant, and backup files share disk blocks via copy-on-write, saving
potentially gigabytes of space.
Closes#5330
A new function, `Zotero.DB.backUpDatabase()`, by default makes offline
backups, by closing the database, doing a regular file copy, and
reopening the database. It takes an options object with an `online`
flag to make online, incremental backups that can take multiple minutes
to complete, though for now we're still running them on idle.
The old function, `Zotero.DB.backupDatabase(suffix, force)`, is
deprecated and proxies to the new function, making offline backups.
Fixes#4935
- Create userdata tables and indexes that are missing
- Delete tables and triggers that should no longer exist
- Run schema integrity check before user data migration
- Run schema integrity check after restart error
This is meant to address two problems:
1) Database damage, and subsequent use of the DB Repair Tool, that
results in missing tables
2) A small number of cases of schema update steps somehow not being
reflected in users' databases despite their having updated userdata
numbers, which are set within the same transaction. Until we figure
out how that's happening, we should start adding conditional versions
of schema update steps to the integrity check.
This is currently only running the update check after a restart error,
which might not occur for all missed schema update steps, so we might
want other triggers for calling setIntegrityCheckRequired().
- Add an 'exclusive' option to transactions that causes them to block other
transactions and wait for other transactions to finish before starting,
instead of nesting
- Resolve Zotero.DB.waitForTransaction() promise before returning from
executeTransaction()
- A side effect of the above: wait for a newly created item to be selected in
the middle pane and rendered in the right-hand pane before returning from
executeTransaction()
- Don't save items multiple times when adding/removing a non-final creator in
the Info pane
- Use a simpler, non-recursive method for focusing the next field in the Info
pane; this prevents "too much recursion" errors if something causes the
right-hand pane not to be rendered when expected
This fixes an issue where two transactions started around the same time
could run separately instead of nesting, causing the statements from one
to end up running not within a transaction
