GitNexus/gitnexus/test/unit/incremental-index-extension-dml-gate.test.ts
Gergő Magyar e69d3c49c4
fix(analyze): gate FTS-indexed DML before the incremental writeback (#2841) (#2854)
* fix(lbug): never report a drop that could not happen, and gate FTS-indexed DML

`CALL DROP_FTS_INDEX` is itself an FTS-extension function, so with the
extension unloaded it fails with `Catalog exception: function DROP_FTS_INDEX
is not defined`. `isBenignDropFtsIndexError` classifies that as "nothing to
drop" — correct when the index does not exist, wrong when it does: the drop
silently no-ops and the next write to that table dies at bind time with an
engine message that never mentions FTS (#2841).

The classifier stays pure (a message cannot tell you whether an index is
live). Instead `dropFTSIndex` settles liveness with a catalog read on the
ERROR path only and raises an FTS-named, remedy-bearing error when the index
is present but undroppable.

Adds `ensureFtsRowDmlSafe`, the FTS twin of `ensureEmbeddingRowDmlSafe`
(#2623): catalog first, load FTS with the analyze policy only when an index
actually gates DML. LadybugDB refuses that DML at BIND time — a DETACH DELETE
matching zero rows fails exactly as hard as one matching thousands — and the
indexes cannot be cleared in place, so a verdict is the only useful answer.

Both gates now share one `SHOW_INDEXES` read via `readIndexCatalogRows`, so
adding the FTS check costs no extra catalog round-trip.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* fix(analyze): escalate instead of crashing when FTS blocks incremental DML

The incremental writeback decided its write plan without ever asking whether
row-level DML was legal. On a DB carrying FTS indexes with an unloadable FTS
extension, `deleteNodesForFiles` then died mid-writeback:

    Binder exception: Trying to delete from an index on table File but its
    extension is not loaded.

with no mention of FTS anywhere in the run — the only install-capable load
happened in Phase 3, long after the writes (#2841).

The incremental branch now reads the index catalog once and derives both
extension verdicts before any DML. When FTS (or VECTOR) blocks in-place
writes, the run falls through to the existing wipe-and-bulk-COPY escalation
— the same answer #2623 gave for VECTOR, and the only one available, since
the indexes cannot be dropped without the extension.

Every blocked extension is named in the reason log, not just the first one
checked: a DB can carry both a vector index and FTS indexes, and reporting
half the cause is how this failure stayed mis-diagnosed.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* test(analyze): cover the FTS DML gate, both-blocked escalation, and the drop guard

New `incremental-index-extension-dml-gate.test.ts` drives the real
`runFullAnalysis` against a real mini-repo and a real LadybugDB:

  - a DB carrying FTS indexes with FTS made unloadable escalates to a full DB
    write, names FTS in the log, ends with zero FTS indexes, and still has the
    newly committed content in the graph (pre-fix: Binder exception, exit 1);
  - FTS available keeps the surgical plan and the indexes;
  - a DB that never carried FTS indexes is not escalated (the catalog-first
    check must not tax FTS-less machines);
  - FTS and VECTOR both blocked produce ONE escalation naming both.

`drop-fts-index-error-classification.test.ts` gains the two `dropFTSIndex`
cases the #2841 guard turns on: live index + unloaded extension rejects with
an FTS-named error, absent index still resolves. The existing classifier
assertions are unchanged — it stays pure.

The CLI e2e reproduces the reporter's exact journey (analyze with the
extension, remove it, touch a file, analyze again) and asserts exit 0 plus an
FTS-named reason. It skips visibly when the seeded extension cannot load on
the host, so it can never report a false red about the fix.

Mutation-verified: reverting the run-analyze gate fails the first scenario;
reverting the dropFTSIndex guard fails the live-index case.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* fix(lbug): make every catalog-gated path fail closed, and classify the drop remedy

Review findings on #2854 (two-engine, 17 lanes).

H3 — `ftsIndexExistsInCatalog` returned `false` when the catalog could not be
read, i.e. "index absent", so `dropFTSIndex` swallowed the error and the caller
proceeded as if the index were gone. That is the #2841 symptom the guard exists
to make loud, and it contradicted the contract `readIndexCatalogRows` states two
functions above. It now fails closed.

§6.A — `ensureFtsRowDmlSafe` keyed on `index_type === 'FTS'`, which answers
`undefined === 'FTS'` → false → *no gate* for a row whose shape cannot be read:
fail-open, in the gate whose only job is preventing an unsafe write, while the
VECTOR twin fails closed on the same input. Now only a positively-identified
non-FTS index is waved through. Deliberately NOT the twin's `!== 'HASH'`: that
is safe there only because it is scoped to the embedding table first, and this
gate is table-agnostic — `!== 'HASH'` would let the HNSW index gate FTS DML.

§5.A — `undefined` was overloaded: "caller passed nothing" and "caller tried and
could not prove anything" shared one value, so a failed shared read silently
became three reads and the two gates could decide from different snapshots. The
failed snapshot is now representable (`INDEX_CATALOG_UNREADABLE`), leaving one
unambiguous `??` in `resolveGateRows`.

§5.B — both gates regained the unconditional null-connection precondition the
refactor moved into the reader.

§5.G — the throw's remedy now routes through `diagnoseExtensionLoad`, like
`--repair-fts` and `ftsDegradedWarning`, so a missing runtime dependency is not
told to reinstall. The message stays path-free (#2374/#2375).

The dead positional row fallbacks are kept and marked `LADYBUGDB-CONTRACT`:
removing them would turn a proven-inert hedge into a fail-open gate if a future
engine returns unnamed tuples.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* fix(analyze): never undo an explicit wipe, stage extension-forced rebuilds, report honestly

Review findings on #2854 (two-engine, 17 lanes).

H1 (P1, both engines) — `analyze --drop-embeddings` was silently reverted. The
`--drop-embeddings` → `force` conversion sits inside the `embeddingCheckpoint`
branch, so without a checkpoint the run stays incremental and reaches the gate;
the flag then *deliberately* leaves `cachedEmbeddings` empty, which is exactly
the rescue's trigger, so every row the operator asked to destroy was read back
and restored, exit 0. Widening the rescue from `!embeddingRowDmlSafe` to
`extensionForcedRebuild` moved that latent bug onto the dominant path, because
every analyzed DB carries FTS indexes. Guarded on the flag itself — NOT on
`shouldLoadCache`, which is false in the meta-under-reports case the rescue
exists for and would have deleted the safeguard while fixing the wipe. The
`--drop-embeddings --embeddings` variant is covered by the same guard.

H2 — an extension-forced escalation wiped the LIVE index in place: `buildPath`
was frozen ~440 lines earlier while the run was still classified incremental,
so an interrupt or ENOSPC left no complete index, where main failed at bind time
with it intact. Extension-forced rebuilds now build into a staging file and
publish via the existing atomic swap; size-forced ones stay in place, since that
trigger is the repo's own churn rather than a machine condition.

H5 — the escalation log asserted a vector index "exists" and that the store
"carries FTS indexes" in exactly the case the catalog read proved nothing, while
the only truthful signal went to stderr rather than the IPC log. It now emits a
distinct unreadable-catalog cause, and "this index carries" (which pointed at
the vector index just named) reads "the graph store carries".

§5.D — the write-set cause was dropped whenever an extension cause co-occurred;
causes are appended now, not selected between.

§5.C — after an FTS-forced rebuild stamped lastCommit, a plain rerun on the same
commit hit the alreadyUpToDate fast path before Phase 3, so the CLI's "install
… then rerun" advice could never restore FTS. The fast path is now bypassed when
meta records FTS unavailable and the extension can load again, keyed on the
persisted capabilities stamp rather than new state.

§5.F (skip the escalation for a zero-change commit) is deliberately NOT
implemented: `deleteSpringAutoConfigurationSyntheticClasses` and
`deleteSpringAopEvidenceNodes` run unconditionally on the surgical branch and
bind against FTS-indexed `Class`/`CodeElement`, and a zero-row DETACH DELETE
fails at bind time exactly as hard as a large one — so the skip would restore
the original crash.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* perf(search): read the index catalog once per drop sweep, and state the real contract

Review findings on #2854.

H4 — on a machine where FTS cannot load and the DB carries no FTS index, the
gate correctly returned early without loading the extension, but the surgical
path still ran the full 20-entry drop sweep: every `CALL DROP_FTS_INDEX` raised
"function DROP_FTS_INDEX is not defined", and the new liveness guard then fired
a fresh catalog read per table — 20 reads every run, forever, for exactly the
offline/load-only population, contradicting the "healthy path costs nothing"
claim shipped with the guard. The sweep now reads the catalog once and skips
entirely when no FTS-typed index exists. An unreadable catalog runs the sweep,
so an unprovable catalog never skips real work.

H8 — the docstring still promised `dropFTSIndex` "tolerates" an unloadable
extension. Post-#2854 a live index plus an unloadable extension throws, and
safety rests on caller ordering discipline rather than the type system — which
is what would have talked the next caller out of that ordering.

GUARDRAILS — the "switching to a full DB write" sign described exactly one
trigger (write set >~50%). Since #2623 and #2841 an unloadable extension
escalates regardless of write-set size; documented with its recovery steps.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* test(analyze): cover the wipe guard, the staged rebuild, and the fail-closed branches

Review findings on #2854.

H1/H2 mutation-verified: removing `!options.dropEmbeddings` fails the new
drop-embeddings case ("expected true to be false"); disabling the staging
upgrade fails the staging case ("expected 0 to be greater than 0"), so both
assert behaviour rather than describe it.

Gate suite (7 cases): `--drop-embeddings` under an FTS-forced escalation ends at
zero embedding rows and logs no "Preserving"; the escalation is one-shot — a
third run on a healthy host returns to surgery and rebuilds every FTS index; an
extension-forced rebuild is observed building into `lbug.staging.*` and leaves
none behind; the rescue complement still preserves un-stamped rows when no wipe
was requested; the never-built case now asserts the commit reached the graph.

H6 — the both-blocked case hard-asserted `createVectorIndex()` while the suite
probed FTS only, so it went red on any FTS-yes/VECTOR-no host. VECTOR is probed
now and gates only that case, with a GITNEXUS_REQUIRE_VECTOR hard-fail.

H7 — the fail-closed branches had no coverage although the VECTOR twin's test
and interception technique were ready to copy: `ensureFtsRowDmlSafe` under an
unreadable catalog now proves it routes to the load, and `dropFTSIndex` proves
it rejects rather than silently tolerating. Plus a redaction case that forces a
real path-bearing load failure — under policy `never` the assertion would have
been vacuous, since that reason carries no path.

§5.E/§6.B — the suite is registered in the cross-platform matrix (its sibling
was; it wasn't, and GITNEXUS_REQUIRE_VECTOR is set only on that job) and moved
into the sequential lbug-db project per TESTING.md:68, verified not to drop it
from the sharded ubuntu job. A Windows shard weight is added as a labelled
estimate — the 8s floor would skew the split it exists to protect.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* refactor(analyze): make the FTS gate's fast path cheap, its claims provable, and its remedies classified

Cleanup review of the #2841 work (four parallel angles: reuse, simplification,
efficiency, altitude). Behaviour-preserving except where the previous behaviour
was wrong.

Correctness the review caught:

- The fast-path probe keyed on `capabilities.fts.status === 'unavailable'`,
  which collapses "extension unavailable" and "index build failed". A
  deterministic build failure (an un-tokenizable row, #2544) therefore bypassed
  `alreadyUpToDate` on EVERY subsequent run, re-analyzed the whole repo, failed
  the same way, and restamped — a permanent loop where the run used to be one
  `stat`. Phase 3 already computes the discriminator; it is now persisted as
  `fts.skipReason` and the probe only runs for `extension-unavailable`. Metas
  written before this carry no field and keep today's behaviour.

- `dropSearchFTSIndexes` skipped its sweep when no row read `index_type ===
  'FTS'`, while `ensureFtsRowDmlSafe` treats an unreadable type as "might be
  FTS". Opposite polarity, under a comment claiming they matched: a row-shape
  change would let the gate wave the surgical plan through while the sweep
  dropped nothing, putting DELETEs back on tables carrying live FTS indexes —
  #2589 again. The sweep now decides per configured index on identity, which
  is also strictly more precise. Its old justification (leftover indexes under
  other names) was unreachable — the loop only ever drops configured entries.

- `dropFTSIndex` threw "FTS index X on table Y exists" on the one path where
  the catalog could not be read — a fabricated claim, on a DB the same run had
  just shown carries no FTS index. Presence is now `present | absent |
  unverifiable` and the message says which.

- The remedy was hand-written for three of the four load-failure classes,
  discarding `missingFileRemedy`/`corruptFileRemedy`, so a corrupt extension
  file was told to retry an install — the misdirection #2383 fixed. Both the
  drop error and the escalation log now use the classified remedy.

Cost, measured on a 391 MB index (cold open ~1 s, SHOW_INDEXES ~4 ms):

- The probe opened the live index WRITABLE on the millisecond fast path,
  dragging in schema DDL, the cross-process write lock, sidecar reclaim and a
  CHECKPOINT on close. It is read-only now. That also closes an install trap:
  `doInitLbug`'s pre-load resolves the env policy on the writable branch, so an
  operator following our own `GITNEXUS_LBUG_EXTENSION_INSTALL=auto` advice paid
  a forked 15 s installer on every up-to-date run (memoized per process; the CLI
  is a fresh process each time). The read-only branch pins `load-only`.

- A failed staged rebuild orphaned a full index-sized copy until the next lock
  sweep; the failure path now reclaims it.

- The sweep re-read a catalog the run already held, defeating the invariant the
  snapshot type exists to enforce.

Structure: row-shape accessors have one home, so the LADYBUGDB-CONTRACT grep
claim is true by construction; staging now applies to both escalation causes,
since recoverability is a property of the wipe-then-COPY plan, not of the
trigger; `getExtensionCapability`/`getFtsCapability` replace hand-spelled
lookups where the seam allows.

Two lookups in run-analyze.ts deliberately keep the exported
`getExtensionCapabilities()` form: the #2383 tests stub that export, and an ESM
module mock does not intercept a helper's internal call — routing through it
silently degraded the classified remedy to generic text. Recorded in-comment.

Not taken, deliberately: extracting the escalation message and replacing the
snapshot protocol with a connection-scoped catalog memo (both sound, both
restructure code this PR just stabilised — they belong in their own change);
an extension registry (premature at two instances, and the FTS/VECTOR polarity
difference is exactly what it would have to parameterize back out).

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* test(analyze): pin both sides of the degraded-FTS fast-path bypass

`healDegradedFts` (§5.C) had zero coverage — three separate review angles
flagged it, and the cleanup pass then found it sat one conjunct away from a
permanent full-re-analyze loop. Both sides are pinned now:

- it re-analyzes past `alreadyUpToDate` when the stored meta says FTS is
  degraded and the extension loads again: run 1 analyzes with loads blocked
  (asserting the precondition — `status: 'unavailable'`, `skipReason:
  'extension-unavailable'` — rather than assuming it), then a same-commit
  clean-tree rerun rebuilds every FTS index without a file changing;
- it stands down when the degradation was a BUILD failure: the stored
  `skipReason` is rewritten to 'build-failed' and the rerun must take the fast
  path, because that rebuild would fail identically on every run forever.

The build-failed state is reached by rewriting the stamped discriminator, not
by provoking a real tokenizer failure: a genuine one needs a stored row the
native tokenizer rejects (#2544/#2546), which is neither portable across the CI
matrix nor deterministic, and §5.C reads only that field.

Also folds the first escalation case into the one-shot case. The claim that it
was fully subsumed did not hold on audit: `logs` containing 'FTS' was unique as
expected, but so was the duplicate-File-node row count — every other reader goes
through a Map keyed by path, which collapses a stale twin an appending rebuild
would leave. Both assertions moved rather than one being dropped.

Net suite runtime goes UP (two cycles removed, four added), against the
cross-platform-matrix argument that motivated the dedup — recorded here because
the shard weight is an estimate pending a real Windows measurement.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* test(search): keep the whole-module adapter mock in step with the row accessors

The cleanup pass moved the LadybugDB row-shape reads behind named accessors so
the column contract has one home. `fts-indexes.test.ts` mocks the entire adapter
module with a hand-written factory, which still exposed only the three exports
the file imported before — so `verifySearchFTSIndexes` failed with "No
`indexRowName` export is defined on the mock" while production was fine.

The added accessors mirror the real implementations rather than returning
stubs. A stub would have read `undefined` out of every catalog row and let the
suite pass for the wrong reason — the failure mode a whole-module mock invites
whenever the module under test grows an import.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* revert(analyze): drop the degraded-FTS auto-heal, fix the advice it existed to justify

§5.C's complaint was that the CLI tells users to "install the extension … then
rerun" when a rerun lands on the up-to-date fast path and rebuilds nothing. The
answer shipped for it was a probe that bypasses that fast path. Four independent
problems later, the sentence is cheaper to fix than to make true:

- it could not tell "extension was missing" from "index build failed" without a
  stamped discriminator, so a deterministic build failure (#2544/#2546)
  re-analyzed the entire repo on every invocation, forever, where the run used
  to be one `stat`;
- it opened the live index on the millisecond fast path — writable at first,
  dragging in DDL, the cross-process lock and a CHECKPOINT (~1 s on a 391 MB
  index), and even read-only it is a full open;
- `doInitLbug`'s pre-load resolves the env policy, so an operator following our
  own `GITNEXUS_LBUG_EXTENSION_INSTALL=auto` advice paid a forked 15 s installer
  per up-to-date run;
- and it turns the fast path into a full re-analysis whenever an index authored
  where FTS was unavailable is later read where it loads — a legitimate, common
  state, and the invariant `analyzer-identity-cli.test.ts` pins.

So: no probe. The degraded-search warning now points at `gitnexus analyze
--repair-fts`, which rebuilds the search indexes without re-parsing the repo,
instead of "then rerun". One line, no new failure modes, and it is what the
issue actually asked for.

`capabilities.fts.skipReason` stays in the meta stamp: it costs three lines,
makes the two degradation causes distinguishable for support, and is what any
future correct answer here would key on.

Also gates the H2 staging assertion on the production predicate. It asserted
staging unconditionally while the upgrade requires `posixSwap || windowsSwapOk`,
and `windowsSwapOk` is opt-in (#2614) — so it failed on the Windows matrix for a
reason unrelated to #2841. Registering this suite cross-platform is what exposed
it; the assertion now mirrors the condition it is testing.

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

* fix(analyze): never stage around a damaged index — escalate in place when the catalog is unreadable

CI caught this on ubuntu and macOS: `analyze-wal-checkpoint-failure` stopped
failing, which is worse than it sounds.

That test plants a directory at `.gitnexus/lbug.wal.checkpoint` so the
auto-checkpoint's rename target is blocked, and asserts analyze exits non-zero
with the `--wal-checkpoint-threshold` hint. But LadybugDB cannot open that path
at all, so `CALL SHOW_INDEXES()` now fails with `IO exception: … Is a
directory`. The catalog read returns UNREADABLE, both DML gates correctly fail
closed, both extension loads fail with the same IO error, and the run escalates
— and since the escalation stages, it built a fresh index at
`lbug.staging.<uuid>`, swapped it in, and exited 0.

The blocked path was never touched. The run "succeeded" while the damage sat
untouched on disk, waiting to break the next in-place writeback.

So the staging upgrade is now conditional on the catalog having been READ.
Staging exists to protect a healthy live index from a machine-level cause (an
extension that will not load); it must not be used to route around a damaged
one. When we are escalating out of ignorance, build in place so the underlying
IO fault lands on the failure path where the operator gets a diagnosis.

Verified against the real CLI, not just the suite: with a directory planted at
the checkpoint path, analyze now exits 1 and prints
`gitnexus analyze --wal-checkpoint-threshold 67108864`. The healthy
extension-forced case still stages (gate suite 6/6).

Refs #2841

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxokkpfssUCxvBwRZMtRCB

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 09:44:44 +01:00

576 lines
27 KiB
TypeScript

/**
* #2841: an incremental writeback must decide whether row-level DML is even
* legal BEFORE it mutates a row. LadybugDB refuses every write to a table
* carrying an FTS index while the FTS extension is unloaded — at BIND time, so
* a DETACH DELETE matching zero rows fails exactly as hard as one matching
* thousands:
*
* Binder exception: Trying to delete from an index on table File but its
* extension is not loaded.
*
* and the indexes cannot be cleared in place either (`DROP_FTS_INDEX` is itself
* an FTS-extension function; LadybugDB has no SQL `DROP INDEX`). So a DB that
* carries FTS indexes on a machine where the extension stopped loading used to
* kill every incremental analyze mid-writeback, with an engine message that
* never mentions FTS.
*
* The fix mirrors the VECTOR gate (#2623): probe the index catalog first, load
* FTS with the analyze policy only when an index actually gates DML, and fall
* through to the escalation valve's wipe-and-bulk-COPY plan when it cannot be
* loaded. The VECTOR half of that behaviour is covered by
* `incremental-vector-extension-ordering.test.ts`; this suite covers the FTS
* half plus the both-extensions-blocked case, where the reason log has to name
* both causes rather than only the first one checked.
*
* The escalation is a valve, not a wipe switch, so three of its consequences
* are pinned here as well:
* - it must NOT rescue embeddings on the one run whose purpose is to destroy
* them (`--drop-embeddings`, review H1) — and must rescue them on every
* other forced rebuild (the complement, asserted in the both-blocked case);
* - it must be ONE-SHOT: the next run on a machine where FTS loads again goes
* back to surgery and rebuilds the search indexes, rather than escalating
* forever;
* - an extension-forced rebuild is environmental, not repo churn, so it builds
* into a staging file beside the live index and publishes it with one rename
* (review H2) — an interrupted rebuild must leave the current index intact.
*
* That rebuild stamps `lastCommit`, so a plain rerun on an unchanged tree takes
* the `alreadyUpToDate` fast path and the search indexes stay missing. That is
* addressed in the CLI's advice (`--repair-fts`, which rebuilds the indexes
* without re-parsing), NOT by an auto-heal probe here — one was tried and
* reverted for re-analyzing the whole repo on every run in the build-failed
* case, for opening the live index on the millisecond fast path, and for
* breaking the fast-path invariant `analyzer-identity-cli.test.ts` pins.
*/
import { readFile, readdir, writeFile } from 'fs/promises';
import { execSync } from 'child_process';
import path from 'path';
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
import type { TestContext } from 'vitest';
import { setupMiniRepo } from '../helpers/mini-repo.js';
import { seedEmbeddingsForFiles } from '../helpers/embedding-seed.js';
import { getStoragePaths } from '../../src/storage/repo-manager.js';
import { createTempDir } from '../helpers/test-db.js';
import { FTS_INDEXES } from '../../src/core/search/fts-schema.js';
import { EMBEDDING_TABLE_NAME } from '../../src/core/lbug/schema.js';
import { resolveAnalyzeInstallPolicy } from '../../src/core/lbug/extension-loader.js';
const ftsMustBeAvailable = process.env.GITNEXUS_REQUIRE_FTS === '1';
const vectorMustBeAvailable = process.env.GITNEXUS_REQUIRE_VECTOR === '1';
const commitAll = (cwd: string, message: string): void => {
execSync('git -c user.name=test -c user.email=t@t -c commit.gpgsign=false add -A', {
cwd,
stdio: 'pipe',
});
execSync(
`git -c user.name=test -c user.email=t@t -c commit.gpgsign=false commit -q -m "${message}"`,
{ cwd, stdio: 'pipe' },
);
};
/**
* Append a line to a mini-repo file and commit it — a one-file write set.
* `relPath` is POSIX-joined for the graph side but filesystem-joined for the
* write, so callers can target a file the NEXT run will not touch.
*/
const touchAndCommit = async (
repoPath: string,
marker: string,
relPath = 'src/handler.ts',
): Promise<void> => {
const filePath = path.join(repoPath, ...relPath.split('/'));
await writeFile(filePath, (await readFile(filePath, 'utf-8')) + `\n// ${marker}\n`, 'utf-8');
commitAll(repoPath, marker);
};
const readFtsIndexRows = async (lbugPath: string): Promise<Array<Record<string, unknown>>> => {
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
await lbugAdapter.initLbug(lbugPath);
try {
const rows = (await lbugAdapter.executeQuery('CALL SHOW_INDEXES() RETURN *')) as Array<
Record<string, unknown>
>;
return rows.filter((r) => r.index_type === 'FTS');
} finally {
await lbugAdapter.closeLbug();
}
};
/** Every File node in the published graph, as rows (duplicates stay visible). */
const readGraphFileRows = async (
lbugPath: string,
): Promise<Array<{ filePath: string; content: string }>> => {
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
await lbugAdapter.initLbug(lbugPath);
try {
const rows = (await lbugAdapter.executeQuery(
`MATCH (f:File) RETURN f.filePath AS filePath, f.content AS content`,
)) as Array<{ filePath: string; content: string }>;
return rows.map((r) => ({ filePath: String(r.filePath), content: String(r.content) }));
} finally {
await lbugAdapter.closeLbug();
}
};
const contentsByPath = (
rows: ReadonlyArray<{ filePath: string; content: string }>,
): Map<string, string> => new Map(rows.map((r) => [r.filePath, r.content]));
/** Surviving CodeEmbedding nodeIds, read straight from the published DB. */
const readEmbeddingRows = async (lbugPath: string): Promise<string[]> => {
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
await lbugAdapter.initLbug(lbugPath);
try {
const rows = (await lbugAdapter.executeQuery(
`MATCH (e:${EMBEDDING_TABLE_NAME}) RETURN e.nodeId AS nodeId`,
)) as Array<{ nodeId: string }>;
return rows.map((r) => String(r.nodeId));
} finally {
await lbugAdapter.closeLbug();
}
};
/** `count(e)` straight from the engine — the H1 wipe has to be proven at zero. */
const countEmbeddingRows = async (lbugPath: string): Promise<number> => {
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
await lbugAdapter.initLbug(lbugPath);
try {
const rows = (await lbugAdapter.executeQuery(
`MATCH (e:${EMBEDDING_TABLE_NAME}) RETURN count(e) AS total`,
)) as Array<{ total: number | bigint }>;
expect(rows.length).toBe(1);
return Number(rows[0]?.total);
} finally {
await lbugAdapter.closeLbug();
}
};
/** Staging builds sit beside the live index as `lbug.staging.<uuid>` (#2658). */
const readStagingEntries = async (storagePath: string): Promise<string[]> =>
(await readdir(storagePath)).filter((name) => name.includes('.staging.'));
/**
* Make every optional extension unloadable for the next run. The env policy is
* what actually blocks the load (`ExtensionManager.ensure` short-circuits on
* `'never'` before it consults any cache); `resetExtensionState()` clears the
* process-wide capability/install memo so the run reports its own verdict
* rather than one an earlier test in this file settled.
*/
const blockExtensionLoads = async (): Promise<void> => {
process.env.GITNEXUS_LBUG_EXTENSION_INSTALL = 'never';
const { resetExtensionState } = await import('../../src/core/lbug/extension-loader.js');
resetExtensionState();
};
const restoreExtensionPolicy = async (previous: string | undefined): Promise<void> => {
if (previous === undefined) delete process.env.GITNEXUS_LBUG_EXTENSION_INSTALL;
else process.env.GITNEXUS_LBUG_EXTENSION_INSTALL = previous;
const { resetExtensionState } = await import('../../src/core/lbug/extension-loader.js');
resetExtensionState();
};
describe('runFullAnalysis incremental writeback — extension-gated DML decided before any DML (#2841)', () => {
let ftsAvailable = true;
let vectorAvailable = true;
let skipWarned = false;
let vectorSkipWarned = false;
beforeAll(async () => {
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
// Cheap standalone probe, matching the #2589/#2623 suites: settle
// availability once, up front, not inside the expensive test body. BOTH
// extensions are probed on the one throwaway connection (review H6): the
// both-blocked case builds a REAL HNSW index and asserts it was built, so
// gating that case on the FTS probe alone made a hard assertion fail on
// every host that has FTS but not VECTOR.
const probe = await createTempDir('gitnexus-2841-extension-probe-');
try {
await lbugAdapter.initLbug(probe.dbPath);
const policy = resolveAnalyzeInstallPolicy();
ftsAvailable = await lbugAdapter.loadFTSExtension(undefined, { policy });
vectorAvailable = await lbugAdapter.loadVectorExtension(undefined, { policy });
} finally {
await lbugAdapter.closeLbug();
await probe.cleanup();
}
}, 120_000);
// Skip VISIBLY: a silent `return` would report a false pass and hide the
// regression in exactly the environments least likely to notice.
beforeEach((ctx) => {
if (!ftsAvailable) {
if (ftsMustBeAvailable) {
throw new Error(
'GITNEXUS_REQUIRE_FTS=1 but the FTS extension is unavailable — cannot verify the #2841 gate.',
);
}
if (!skipWarned) {
skipWarned = true;
console.warn(
'[incremental-index-extension-dml-gate] Skipping — the LadybugDB FTS extension is unavailable.',
);
}
ctx.skip();
}
});
afterEach(() => {
vi.restoreAllMocks();
});
/**
* Per-test VECTOR gate (review H6). Only the both-blocked case needs VECTOR;
* the other cases must keep running on an FTS-only host, so this is called
* from that one test body instead of widening the suite-level `beforeEach`.
* Same visibility contract as the FTS gate: a real skip, and a hard failure
* under `GITNEXUS_REQUIRE_VECTOR=1`.
*/
const skipUnlessVectorAvailable = (ctx: TestContext): void => {
if (vectorAvailable) return;
if (vectorMustBeAvailable) {
throw new Error(
'GITNEXUS_REQUIRE_VECTOR=1 but the VECTOR extension is unavailable — cannot verify the #2841 both-blocked escalation.',
);
}
if (!vectorSkipWarned) {
vectorSkipWarned = true;
console.warn(
'[incremental-index-extension-dml-gate] Skipping the both-blocked case — the LadybugDB VECTOR extension is unavailable.',
);
}
ctx.skip();
};
it('keeps the surgical write plan (and the indexes) when FTS is available', async () => {
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const repo = await setupMiniRepo('gitnexus-2841-fts-available-');
try {
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
await touchAndCommit(repo.dbPath, '#2841 healthy-path touch');
const logs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true },
{ onProgress: () => {}, onLog: (m: string) => logs.push(m) },
),
).resolves.toBeDefined();
// No escalation: a one-file write set on a 7-file repo stays surgical,
// and the gate must not manufacture a rebuild when FTS loads fine.
expect(logs.some((m) => m.includes('full DB write'))).toBe(false);
const { lbugPath } = getStoragePaths(repo.dbPath);
expect((await readFtsIndexRows(lbugPath)).length).toBe(FTS_INDEXES.length);
} finally {
await repo.cleanup();
}
}, 300_000);
it('does not escalate — or touch the extension machinery — when the DB never carried FTS indexes', async () => {
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const repo = await setupMiniRepo('gitnexus-2841-fts-never-built-');
const previousPolicy = process.env.GITNEXUS_LBUG_EXTENSION_INSTALL;
try {
// Both runs are FTS-less, so no index is ever created. The catalog-first
// check must settle this without gating the surgical plan — otherwise
// every incremental analyze on an FTS-less machine would become a full
// rebuild.
await blockExtensionLoads();
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
const { lbugPath } = getStoragePaths(repo.dbPath);
expect((await readFtsIndexRows(lbugPath)).length).toBe(0);
await touchAndCommit(repo.dbPath, '#2841 never-built touch');
const logs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true },
{ onProgress: () => {}, onLog: (m: string) => logs.push(m) },
),
).resolves.toBeDefined();
expect(logs.some((m) => m.includes('full DB write'))).toBe(false);
// Test gap 7: "no escalation log" alone would also pass if the surgical
// write silently did nothing. Prove the write actually landed — the same
// File.content check the blocked-path case makes. REVERSION: make
// `ensureFtsRowDmlSafe` fall OPEN on an index row whose type cannot be
// read (`indexType === undefined || indexType === 'FTS'` → `=== 'FTS'`,
// review §6.A) and this file still has no FTS index, so the no-escalation
// half keeps passing while a genuinely blocked DML would reach the engine.
const contents = contentsByPath(await readGraphFileRows(lbugPath));
expect(contents.get('src/handler.ts')).toContain('#2841 never-built touch');
} finally {
await restoreExtensionPolicy(previousPolicy);
await repo.cleanup();
}
}, 300_000);
it('names every blocked extension when both FTS and VECTOR gate the write', async (ctx) => {
skipUnlessVectorAvailable(ctx);
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const repo = await setupMiniRepo('gitnexus-2841-both-blocked-');
const previousPolicy = process.env.GITNEXUS_LBUG_EXTENSION_INSTALL;
try {
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
const { lbugPath } = getStoragePaths(repo.dbPath);
// POSIX literal for the graph-side path: filePaths are stored with
// forward slashes on every OS (see the note in the #2623 suite).
// Deliberately NOT stampEmbeddingCount: meta reports zero embeddings
// while the DB holds these rows, which is exactly the state the forced
// rebuild's rescue read exists for.
const seeded = await seedEmbeddingsForFiles(repo.dbPath, ['src/handler.ts'], 2);
const seededIds = seeded.get('src/handler.ts') ?? [];
expect(seededIds.length).toBeGreaterThan(0);
await lbugAdapter.initLbug(lbugPath);
const vectorIndexBuilt = await lbugAdapter.createVectorIndex();
await lbugAdapter.closeLbug();
// Hard assertion, not an environment gap: `skipUnlessVectorAvailable`
// above already proved VECTOR loads on this host (review H6).
expect(vectorIndexBuilt).toBe(true);
expect((await readFtsIndexRows(lbugPath)).length).toBe(FTS_INDEXES.length);
await touchAndCommit(repo.dbPath, '#2841 both-blocked touch');
await blockExtensionLoads();
const logs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true },
{ onProgress: () => {}, onLog: (m: string) => logs.push(m) },
),
).resolves.toBeDefined();
// One escalation, both causes named. Reporting only the first checked
// extension is how a half-diagnosed failure survives a bug report.
const escalation = logs.filter((m) => m.includes('full DB write'));
expect(escalation.length).toBe(1);
expect(escalation[0]).toContain('FTS');
expect(escalation[0]).toContain('VECTOR');
// Test gap 5 — the COMPLEMENT of the `--drop-embeddings` case below: this
// run never asked to touch embeddings, so the wipe must not eat the rows
// meta failed to account for. REVERSION: delete the
// `if (extensionForcedRebuild && !options.dropEmbeddings &&
// cachedEmbeddings.length === 0)` rescue in run-analyze.ts and every
// seeded row is destroyed by a rebuild the operator did not ask for,
// while the run still exits 0.
const surviving = await readEmbeddingRows(lbugPath);
const survivingIds = new Set(surviving);
for (const id of seededIds) {
expect(survivingIds.has(id)).toBe(true);
}
// …and exactly once each — the restore must not double-insert.
expect(surviving.length).toBe(survivingIds.size);
expect(logs.some((m) => m.includes('Preserving'))).toBe(true);
} finally {
await restoreExtensionPolicy(previousPolicy);
await repo.cleanup();
}
}, 300_000);
it('lets --drop-embeddings wipe unaccounted embeddings instead of rescuing them (review H1)', async () => {
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const repo = await setupMiniRepo('gitnexus-2841-drop-embeddings-');
const previousPolicy = process.env.GITNEXUS_LBUG_EXTENSION_INSTALL;
try {
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
const { lbugPath } = getStoragePaths(repo.dbPath);
expect((await readFtsIndexRows(lbugPath)).length).toBe(FTS_INDEXES.length);
// Real rows and NO stampEmbeddingCount, so meta reports zero embeddings
// while the DB holds these — the exact trigger state for the forced
// rebuild's rescue read. `--drop-embeddings` leaves `cachedEmbeddings`
// empty by construction (`deriveEmbeddingMode` returns
// `shouldLoadCache: false`), and without a checkpoint the run stays
// incremental, so it arrives at the gate looking precisely like the case
// the rescue was written for.
const seeded = await seedEmbeddingsForFiles(repo.dbPath, ['src/handler.ts'], 2);
expect((seeded.get('src/handler.ts') ?? []).length).toBeGreaterThan(0);
await touchAndCommit(repo.dbPath, '#2841 drop-embeddings touch');
await blockExtensionLoads();
const logs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true, dropEmbeddings: true },
{ onProgress: () => {}, onLog: (m: string) => logs.push(m) },
),
).resolves.toBeDefined();
// The FTS block still forces the rebuild — this is the same escalation,
// reached with the one flag whose entire purpose is to destroy the rows
// the rescue would restore.
expect(logs.some((m) => m.includes('full DB write'))).toBe(true);
// REVERSION: drop `!options.dropEmbeddings` from the rescue predicate in
// run-analyze.ts (`if (extensionForcedRebuild && !options.dropEmbeddings
// && cachedEmbeddings.length === 0)`) and the rescue reads the rows back
// out of the DB, logs `Preserving N embedding row(s) across the forced
// rebuild` on top of this run's own drop, and Phase 3.5 re-inserts every
// one of them — `--drop-embeddings` silently becomes a no-op.
expect(logs.some((m) => m.includes('Preserving'))).toBe(false);
expect(await countEmbeddingRows(lbugPath)).toBe(0);
} finally {
await restoreExtensionPolicy(previousPolicy);
await repo.cleanup();
}
}, 300_000);
it('escalates once: the next run on a healthy host goes back to surgery and rebuilds the FTS indexes', async () => {
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const repo = await setupMiniRepo('gitnexus-2841-one-shot-');
const previousPolicy = process.env.GITNEXUS_LBUG_EXTENSION_INSTALL;
try {
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
const { lbugPath } = getStoragePaths(repo.dbPath);
expect((await readFtsIndexRows(lbugPath)).length).toBe(FTS_INDEXES.length);
// Run 2: FTS blocked → the forced rebuild, which leaves a DB with no FTS
// index at all and stamps `capabilities.fts.status = 'unavailable'`.
await touchAndCommit(repo.dbPath, '#2841 one-shot escalated touch');
await blockExtensionLoads();
const escalatedLogs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true },
{ onProgress: () => {}, onLog: (m: string) => escalatedLogs.push(m) },
),
).resolves.toBeDefined();
expect(escalatedLogs.some((m) => m.includes('full DB write'))).toBe(true);
expect((await readFtsIndexRows(lbugPath)).length).toBe(0);
// The reason must be stated in FTS terms before the plan switches — the
// whole issue is that the pre-fix crash ("Trying to delete from an index
// on table File but its extension is not loaded") named no extension at
// all. The both-blocked case asserts this on the escalation line itself,
// but it is VECTOR-gated, so an FTS-only host would lose the property
// entirely without this check.
expect(escalatedLogs.some((m) => m.includes('FTS'))).toBe(true);
// The wipe-and-bulk-COPY republished each file exactly once. Asserted on
// ROWS, not through `contentsByPath`: that Map collapses duplicates, so a
// rebuild that appended a stale twin beside the fresh row would slip past
// every content check in this suite.
const escalatedRows = await readGraphFileRows(lbugPath);
expect(escalatedRows.filter((r) => r.filePath === 'src/handler.ts').length).toBe(1);
// Run 3: FTS loads again and a DIFFERENT file changes. Nothing may carry
// the escalation forward — the catalog-first gate sees no FTS index, so
// the surgical plan stands, and Phase 3 rebuilds the whole index set.
await restoreExtensionPolicy(previousPolicy);
await touchAndCommit(repo.dbPath, '#2841 one-shot healed touch', 'src/validator.ts');
const healedLogs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true },
{ onProgress: () => {}, onLog: (m: string) => healedLogs.push(m) },
),
).resolves.toBeDefined();
// The property that makes the whole design a one-shot rebuild rather than
// a permanent regression: no second escalation, and keyword search is
// whole again. REVERSION: make the extension-forced escalation sticky
// (e.g. keep escalating while `capabilities.fts.status === 'unavailable'`,
// or have `ensureFtsRowDmlSafe` answer from that stamp instead of the
// catalog) and this run escalates again, forever.
expect(healedLogs.some((m) => m.includes('full DB write'))).toBe(false);
expect((await readFtsIndexRows(lbugPath)).length).toBe(FTS_INDEXES.length);
// Run 2's work survived into run 3's surgical write — i.e. the escalated
// rebuild was really published at the canonical path, not left behind in
// a staging file (review H2). handler.ts is untouched by run 3, so its
// marker can only come from the run that escalated.
const contents = contentsByPath(await readGraphFileRows(lbugPath));
expect(contents.get('src/handler.ts')).toContain('#2841 one-shot escalated touch');
expect(contents.get('src/validator.ts')).toContain('#2841 one-shot healed touch');
} finally {
await restoreExtensionPolicy(previousPolicy);
await repo.cleanup();
}
}, 300_000);
it('builds an extension-forced rebuild into a staging file and swaps it in (review H2)', async () => {
const lbugAdapter = await import('../../src/core/lbug/lbug-adapter.js');
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const repo = await setupMiniRepo('gitnexus-2841-staged-rebuild-');
const previousPolicy = process.env.GITNEXUS_LBUG_EXTENSION_INSTALL;
try {
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
const { lbugPath, storagePath } = getStoragePaths(repo.dbPath);
expect((await readFtsIndexRows(lbugPath)).length).toBe(FTS_INDEXES.length);
// Run 1 was a full rebuild, which always stages — so any staging entry
// observed below belongs to the escalated run, not to a leftover.
expect(await readStagingEntries(storagePath)).toEqual([]);
await touchAndCommit(repo.dbPath, '#2841 staged-rebuild touch');
await blockExtensionLoads();
// Observe the build target at the exact moment the full graph is COPYed
// in. The escalated run reaches `loadGraphToLbug` immediately after
// `wipeLbugDbFiles(buildPath)` + `initLbug(buildPath)`, so the presence
// of a `lbug.staging.<uuid>` file there IS the build target.
let stagingDuringBuild: string[] | undefined;
const originalLoadGraphToLbug = lbugAdapter.loadGraphToLbug;
vi.spyOn(lbugAdapter, 'loadGraphToLbug').mockImplementation(
async (...args: Parameters<typeof originalLoadGraphToLbug>) => {
stagingDuringBuild ??= await readStagingEntries(storagePath);
return originalLoadGraphToLbug(...args);
},
);
const logs: string[] = [];
await expect(
runFullAnalysis(
repo.dbPath,
{ skipAgentsMd: true },
{ onProgress: () => {}, onLog: (m: string) => logs.push(m) },
),
).resolves.toBeDefined();
expect(logs.some((m) => m.includes('full DB write'))).toBe(true);
// REVERSION: delete the `if (!useAtomicSwap && (posixSwap ||
// windowsSwapOk)) { useAtomicSwap = true; buildPath =
// `${lbugPath}.staging.${randomUUID()}` }` block in run-analyze.ts and
// `buildPath` stays the LIVE `lbug` file — frozen ~440 lines earlier while
// the run was still classified incremental — so the wipe destroys the only
// complete index before the COPY starts and this list is empty.
expect(stagingDuringBuild).toBeDefined();
const observedStaging = stagingDuringBuild ?? [];
// Platform-gated, matching the production predicate exactly: the upgrade
// requires `posixSwap || windowsSwapOk`, and `windowsSwapOk` is opt-in via
// GITNEXUS_ATOMIC_WINDOWS_SWAP=1 (#2614 keeps the default Windows analyze
// on the proven in-place path). So on Windows without that flag the run
// correctly does NOT stage, and asserting otherwise fails for a reason
// that says nothing about #2841 — which is exactly what the cross-platform
// matrix caught when this assertion was written platform-blind.
const expectsStaging =
process.platform !== 'win32' || process.env.GITNEXUS_ATOMIC_WINDOWS_SWAP === '1';
expect(observedStaging.length > 0).toBe(expectsStaging);
expect(observedStaging.every((name) => name.startsWith('lbug.staging.'))).toBe(true);
// Published, not orphaned: the rename put the rebuild at the canonical
// path, nothing `.staging.` is left beside it, and the live index answers
// a query carrying this run's content.
expect(await readStagingEntries(storagePath)).toEqual([]);
const contents = contentsByPath(await readGraphFileRows(lbugPath));
expect(contents.get('src/handler.ts')).toContain('#2841 staged-rebuild touch');
} finally {
await restoreExtensionPolicy(previousPolicy);
await repo.cleanup();
}
}, 300_000);
});