* fix(analyze): make incremental analyze skip the derived layers it can reuse (#3016)
A warm incremental run only ever wrote a handful of files, but it still
paid for the whole graph on the way out: Leiden ran over every node,
flow extraction re-derived every process, and all FTS indexes were
dropped and rebuilt from scratch. On a small edit that tail dominated
the run, which is why "incremental" did not feel incremental.
Reuse what the previous run already derived when the write plan allows
it. The pipeline holds back community detection and flow extraction
whenever the persisted metadata says this run is a candidate for a
surgical write; the DB keeps its Community/Process rows instead of a
wipe-and-rewrite; and the FTS sweep is narrowed to the indexes the run
actually has to touch.
The bet is placed before the pipeline and settled after it. Any plan
that turns out to need a freshly derived layer — full rebuild, escalated
write, or an incremental diff with deleted files — runs the held-back
phases through `runDeferredDerivedPhases`, against the same graph and
phase outputs, so its output is identical to never having skipped them.
Correctness details worth naming, since each one silently loses data if
got wrong:
- The MEMBER_OF / STEP_IN_PROCESS edges of the changed files are snapshotted
before the DETACH DELETE and reattached after the subgraph load. Both
endpoints are matched by explicit label: `labels(n)[0]` over an unlabelled
match returns an empty string on this engine, which produced a snapshot
that restored nothing.
- The FTS narrowing unions three sets — what the writeback deletes (a DB
probe, because a symbol the edit removed is in no fresh graph but is
still a row), what it inserts (the fresh graph), and what is missing
right now (else a prior escalation's dropped indexes would never come
back). An unreadable index catalog withdraws the narrowing entirely.
- Deletions disqualify reuse outright: persisted derived rows can reference
nodes this run removes, and nothing short of re-deriving can tell which.
Covered by the existing incremental suites, including the
incremental-equals-force byte-equivalence test and the #2589
drop-before-delete ordering test, plus unit tests for the new helpers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(analyze): address #3102 review on derived reuse and FTS narrowing
Re-run Leiden/flows unless the file-hash diff is empty, restore ENTRY_POINT_OF
on the preserve path, always drop class_fts before Spring synthetic Class DML,
and reject seeded duplicate phase names. Prettier and exact FTS drop-ordering
assertions unblock CI and pin the #2589/#3016 contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(analyze): reuse FileHashDiff for derived-layer preserve
Drop the count DTO, share phase-name uniqueness, and remove the File
FTS sentinel that Class already makes unreachable.
Refs #3102
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(analyze): prettier-wrap shouldPreservePersistedDerivedGraph
quality / format failed on the Pick<FileHashDiff> signature wrapping.
Refs #3102
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
- Anchor isBenignDropFtsIndexError to the START of the message
(startsWith, not includes) so a future genuine failure that merely
mentions "Binder exception" or "Catalog exception" mid-message can't
be misclassified as benign. New test proves the old substring match
would have swallowed such a message.
- incremental-fts-drop-ordering.test.ts: probe FTS availability once in
beforeAll and skip VISIBLY via ctx.skip() in beforeEach (matching the
withTestLbugDB/lbug-vector-extension convention) instead of a silent
console.warn+return inside the test body, which reported a false pass
with zero coverage of the ordering invariant when FTS was unavailable.
The post-first-run FTS-index-built check is now a hard assertion
instead of a second soft skip, since the beforeEach gate already
proved the extension loads.
Fixes#2589: incremental analyze intermittently crashed with "FTS
index 'file_fts' is inconsistent: term is missing during delete" after
markdown-only commits, and --repair-fts also failed in that state.
deleteNodesForFiles' batched DETACH DELETE ran against tables that
still carried the FTS index built at the end of the PREVIOUS analyze
run -- createSearchFTSIndexes only drops+rebuilds every index in Phase
3, well after that delete already ran. LadybugDB's FTS extension is
not proven to survive DML against an indexed table (its own docs never
demonstrate the sequence). Call the new dropSearchFTSIndexes() up front
in the non-escalated incremental branch, before deleteNodesForFiles --
Phase 3 still rebuilds every index from the final row set regardless.
New end-to-end test drives a real runFullAnalysis full+incremental
cycle and confirms it fails without this change (file_fts and 50
sibling indexes still present at delete time) and passes with it.
