Firefox ESR 140.15 rejects loads from a content process for URLs that
process couldn't load on its own, including blob: URLs created by chrome
code. Full-text indexing loads HTML attachments through such a URL, so
indexing crashed Zotero with "Illegal load attempt of blob: URL from
web". Loads started by the parent are exempt, so start the load there
and use the child actor only to disable content retargeting.
https://forums.zotero.org/discussion/133661/
(cherry picked from commit e03920890a)
Editing a note flagged it stale, and the background drain indexed it and
then ran an FTS5 'optimize' -- a single statement that rewrites the
content index and can hold the shared database for over a minute --
because the queue was empty again. Note saves waiting on the connection
hit the transaction timeout and told the user to restart Zotero.
Merge the index in bounded steps instead, so no statement runs long
enough to keep other queries waiting, and only after enough items have
been indexed to be worth it.
https://forums.zotero.org/discussion/133298/
(cherry picked from commit 070ae8b615)
If a queued attachment was missing its cache file and re-extraction
produced no text, nothing was recorded in the index state, so the
content-index queue selected the same items forever and the startup
drain never finished.
https://forums.zotero.org/discussion/132808/
A term mixing CJK with non-ASCII words (e.g., Cyrillic plus Japanese)
was routed to the CJK index with only its CJK characters, matching
every document that contained those. Treat any non-CJK letter or digit
as making a term mixed-script, which falls back to a literal scan of
the cached text.
Replace the trigram FTS5 index for attachment content with a unicode61
word index, so terms match whole words with the final token as a prefix
("archive" matches "archives", but "ion" doesn't match "condition"), as
in the pre-FTS5 word index. A multi-word phrase gets adjacent-token
candidates from the index and is then verified against the cached text
of just those items, since FTS5 ignores what separates adjacent tokens;
the verification treats whitespace and hyphen runs as equivalent
(they're frequently extraction layout or styling) but requires other
punctuation to match literally. Notes keep the trigram index and CJK
matching is unchanged; the index database version is bumped so the
index is rebuilt.
Follow-up to #5979
Note content is indexed into fulltext.sqlite, making note searches
accent- and case-insensitive and matching the note's plain text rather
than its HTML markup. To avoid re-indexing on every auto-save, a save
flags the note for background indexing, and searches match a flagged
note from its normalized text in memory until it's indexed.
Closes#378
Index attachment content into a contentless trigram FTS5 table in a
separate, attached fulltext.sqlite, normalized so matching is accent-
and case-insensitive. For content containing CJK characters, a companion
'ascii'-tokenized table holds bigrams so 1-2 character CJK queries, which
the trigram tokenizer can't match, still work. The extracted text still
lives in the .zotero-ft-cache files, so the index is fully derived and
rebuildable.
Use the FTS index for the fulltextContent condition, falling back to the
cached-text scan for queries too short to index, and point quick
search's content matching at the FTS index in place of the now-removed
word index. (One side effect: quick search now matches attachment
content by substring rather than by word.)
Already-extracted content is migrated into the index at startup, slowing
down on active usage. A background queue then extracts not-yet-indexed
attachments gradually when Zotero is idle. Attachments with no local
file or full-text content are recorded as missing. Content downloaded
via sync is processed into the index immediately when the sync finishes,
rather than waiting for idle like it did before, so it's searchable
immediately in on-demand file-download mode.
The index DB is tied to the main DB via the local user key and rebuilt
if they don't match (e.g., after a delete-and-resync). We compact it by
running FTS5's 'optimize' command once the indexing queue drains, and we
vacuum the attached database when necessary to reclaim disk space.
Closes#2038, #2044
Addresses #1595
Rename the sync-delivered full-text content processor's methods and
state to "sync content" (registerSyncContentProcessor,
processSyncedContent, indexSyncedContent, getSyncedContentCacheFile,
etc.), so it reads distinctly from the local index-building queues added
for content search. No behavior change.
vacuum() and the APFS-cloning offline backup path both close and
reopen the SQLite connection, which drops all ATTACHed databases --
including the in-memory "indexing" alias used for the fulltextWords
scratch table set up in Fulltext.init(). Once the connection is
reopened, indexing queries fail with "no such table:
indexing.fulltextWords", and indexItems()'s ignoreErrors path
routes the error to logError(), so indexing silently stops working
for the rest of the session.
Add an onConnect() hook on Zotero.DBConnection for per-connection
state that doesn't persist across reopens, and use it from
Fulltext.init() to re-attach the indexing DB on each reconnect.
Regression sources:
- 67288047f3 ("Use APFS cloning for file copies on macOS") flips
online idle backups to offline-with-clone on APFS, so every idle
backup interval (24h default) closes and reopens the connection.
Affects Mac users on APFS.
- b27c4cb023 ("Enable SQLite WAL mode and add periodic VACUUM
INTO") adds the vacuum path, which closes and reopens on the
first idle period in a session that passes the freelist/time
gates. Affects all users with sufficient DB churn, roughly once
per 14 days.
https://forums.zotero.org/discussion/131576/debug-id-d848621212-indexing-of-pdfs-fail-zotero-9-0-3https://forums.zotero.org/discussion/131718/possible-bug-regression-report-search-unusable-on-macos-zotero-9-0-3-with-large-library
A browser window was no longer actually needed for charset detection
(on macOS, at least, and hopefully elsewhere) because we switched to a
HiddenFrame-based hidden browser. Remaining uses now call
`loadZoteroWindow()`.
- Use full-text cache file from syncing if available when reindexing via
info pane or Rebuild Index → Index Unindexed Items. Only discard it for
full index rebuild. This allows Index Unindexed Items to be used to
force immediate processing of queued content from syncing and avoids
unnecessary syncing back of identical content. Previously, the cache
file was used for a manual index only when the local file didn't exist.
- When rebuilding index, don't clear indexed items with missing local
file that are missing stats due to a pre-411180ef bug.
- indexItems() now takes an 'options' object as its second parameter
- Minor code cleanup
- Don't clear item's index stats (and show "Unknown") when an item is
reindexed remotely and the content matches the local content
- Always update an item's state and its stats in the same query, to
avoid incorrect feedback immediately after indexing
- Clean up `setItemContent()` tests
The items will still match full-text word searches, but they won't match
phrase searches (because those require cache files for non-text
attachments) and the full-text won't sync to other computers, so they
should really be reindexed.
The Zotero.DataDirectory equivalents return string paths instead of nsIFile
instances, so some of these calls now just use Zotero.File.pathToFile(), which
can be removed when the surrounding code is updated to OS.File,
This is the recommended approach (since NetUtil can still do some main-thread
I/O for files) and avoids warnings in the console.
For getContentsAsync(), also sends nsIURIs and string URIs to
Zotero.HTTP.request(), which should be used instead.
This makes getBinaryContentsAsync() much slower (due to the conversion from an
array of bytes to a binary string), but it's only used in tests. For one test
that compares two large files, use MD5 instead.
If a version is returned for an item's full-text content but a 404 is returned
for the content itself (because it's missing in Elasticsearch for some reason),
don't throw an error.
Also remove legacy array comprehensions in fulltext and syncFullTextEngine test
files, which apparently weren't being run.
The test runner now downloads and caches the PDF tools for the current
platform within the test data directory and only redownloads them when
out of date, and it updates the download URL so that the full-text code
pulls from the cache directory via a file:// URL.
The installPDFTools() support function now installs the files directly
instead of going through the prefs, and a new uninstallPDFTools()
function removes the tools. Since the presence of the PDF tools can
affect other tests, tests that need the tools should install them in a
before() and uninstall them in an after(), leaving most tests to run
without PDF indexing.
This also adds a callback to the waitForWindow() support function. If a
modal dialog is opened, it blocks the next promise handler from running,
so a callback has to be used to interact with and close the dialog
immediately.
