GitNexus/gitnexus/test/helpers/fts-availability.ts
Gergő Magyar 21a52af1d4
fix(lbug): ship FTS per-platform and recover in-place native aborts (#3274)
* fix(lbug): pin Ladybug core so Dependabot cannot ship a skewed FTS artifact

The extension version is a separate upstream constant. Ignore daily core bumps and fail the pairing gate when the committed manifest does not name the installed core.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): make doctor and CI FTS gates resolve the packaged artifact

Doctor and the REQUIRE_FTS file gates still treated an empty ~/.lbdb as
unavailable, which would turn three CI jobs red once analyze stops
installing into that tree.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): name native-abort and tuple-missing so analyze cannot mis-advise

The CLI summary's trailing else treated every unknown skip reason as a
missing extension. New crash and platform causes must get their own
remedies, not a network-install hint.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): delete the dead read-path FTS index create

ensureFTSIndex had no production callers and swallowed read-only
CREATE_FTS_INDEX failures, which hid the only signal that a reader
tried to write.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): vendor per-platform FTS artifacts so analyze needs no host install

Keyword search depended on a CDN fetch into ~/.lbdb. Shipping the five
published tuples inside the package makes air-gapped and ignore-scripts
installs load the same artifact the publish gate checksums.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): load the packaged FTS artifact before any network install

Analyze still required a CDN fetch into ~/.lbdb even when the package
already shipped the file. FTS now path-loads the vendored tuple first
and records source labels so a later truncated home copy cannot steal
the diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): diagnose a core/extension version skew instead of a missing runtime

A structurally valid FTS artifact whose path version disagrees with the
packaged pin must name both versions, not prescribe VC++ or OpenSSL.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): stamp an FTS phase so repair stays usable after an in-place abort

A native CREATE_FTS_INDEX abort leaves no skip reason; the next run infers
it from the dirty flag, and --repair-fts must not treat that phase as a
half-written graph.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): park an in-place FTS crash WAL without wiping the graph

An FTS abort after a successful checkpoint must reopen the live index on
macOS, Windows, and Linux. Staging never parks the live WAL; readers keep
today's large-WAL refusal.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): refuse read-only opens of an FTS-poisoned WAL

MCP and serve cannot repair a leftover in-place abort. Fail before the
native open and name --repair-fts, on macOS, Windows, and Linux.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): name a vendor-neutral Windows OpenSSL prerequisite

OQ1 is unanswered here so GitNexus does not ship OpenSSL DLLs. Windows
FTS now asks for a system OpenSSL 3 runtime instead of Git Bash PATH.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): inject the FTS vendor root and redact it on HTTP and MCP

Path-loaded artifacts no longer vary with HOME. Tests pass an injected
vendor tree and assert search warnings never leak a filesystem path.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(lbug): document load-only as the global FTS install default

Analyze still overrides to auto. Packaged per-platform artifacts load
before any network install on macOS, Windows, and Linux.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(lbug): format the FTS install-policy README table

Prettier does not run on Markdown in pre-commit, so the U10 table wrap
needs its own formatting commit.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): skip FTS CREATE after a persisted native abort

A recovered analyze run was retrying CREATE_FTS_INDEX from skipReason
alone. Keep that skip until --repair-fts, fail closed on unsupported
tuples, and honor the checkpoint warrant for park/repair.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): honor checkpoint flushed warrant and align FTS tests with packaged vendor

A no-op CHECKPOINT must not satisfy the FTS park warrant, and CI still asserted HOME-only FTS isolation after analyze started path-LOADing the packaged artifact.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): accept a nonempty incremental write set in the #2790 recovery check

FTS-phase recovery can incremental-add files (changed=0, added=1). That is not the #2790 empty-diff wipe skip.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): compare FTS home versions to the core pin and tighten the publish filename gate

Ladybug's ~/.lbdb/extension directory is the runtime/core version; treating it as the artifact version false-diagnosed skew. The publish guard now rejects a path-escaping filename the same way the fetch script does.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): seed FTS e2e fixtures from the packaged vendor artifact

A machine with no ~/.lbdb copy should still run the vendor-survivorship cases; the seed no longer depends on HOME or a network install.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3274)

Keep in-place FTS abort evidence after persist so a second CREATE abort
cannot fail-open readers, and close the CLI, loader, embed, and e2e gaps
the review called out.

Note: full npm test hit Ladybug worker-pool startup failures under memory
pressure; tsc and 180 targeted unit tests passed.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3274)

Run the vendored-path symlink guard on the OS matrix, put e2e HOME
fixtures on Ladybug's real extension layout, pin the embed crash-WAL
gate before the writable open, and let analyze writers park through
missing-shadow recovery.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): keep --repair-fts CI green after vendored-first FTS

Never-installed warning fixtures must not inspect a packaged vendor binary, and a failed dirty restamp must not abort an otherwise successful --repair-fts run.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(cli): give the #1169 analyze e2e the same 90s Windows budget as its sibling

The first #1169 persist-meta case was still on a 60s spawn/it budget and was killed banner-only on windows-latest after the FTS warning fixture no longer failed the shard first.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(ci): reweight Windows shards after the FTS e2e grew

Vendored-first HOME fixtures pushed fts-extension-e2e to ~6 minutes on windows-latest, so the old 146s weight packed it with skills-e2e and blew the 20-minute watchdog.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-14 08:52:24 +01:00

122 lines
5.9 KiB
TypeScript

import { existsSync, readdirSync, statSync } from 'node:fs';
import { homedir } from 'node:os';
import { join } from 'node:path';
import { resolveVendoredFtsPath } from '../../src/core/lbug/vendored-extension-path.js';
/** A valid `libfts.lbug_extension` is ~2.2MB; anything smaller is truncated/corrupt. */
const MIN_VALID_FTS_EXTENSION_BYTES = 1024 * 1024;
/**
* Find the installed FTS extension file under a `.lbdb/extension` root,
* discovering the version directory instead of assuming it equals the npm
* `@ladybugdb/core` package version. LadybugDB's native INSTALL/LOAD resolves
* its own extension-ABI version directory, which does not always track the
* npm package version — e.g. #2587: bumping the package from 0.18.1 to 0.18.2
* still installs into a `0.18.1` directory, because the underlying
* extension-ABI build did not change with that patch release.
*
* Scans every version subdirectory for a `<platform>/fts/libfts.lbug_extension`
* file and returns the most recently modified one (the one an install/load
* actually just resolved), or null when nothing is installed.
*/
export const findInstalledFtsExtension = (extensionRoot: string): string | null => {
// Fail closed on any FS error (permission quirks, AV file locks on Windows,
// a directory vanishing mid-scan) — same contract as the callers this
// replaces: "not found" is a valid outcome, a thrown exception is not.
try {
if (!existsSync(extensionRoot)) return null;
let best: { path: string; mtimeMs: number } | null = null;
for (const versionEntry of readdirSync(extensionRoot)) {
const versionDir = join(extensionRoot, versionEntry);
if (!statSync(versionDir).isDirectory()) continue;
for (const platformEntry of readdirSync(versionDir)) {
const candidate = join(versionDir, platformEntry, 'fts', 'libfts.lbug_extension');
if (!existsSync(candidate)) continue;
const stat = statSync(candidate);
if (stat.size < MIN_VALID_FTS_EXTENSION_BYTES) continue;
if (!best || stat.mtimeMs > best.mtimeMs) best = { path: candidate, mtimeMs: stat.mtimeMs };
}
}
return best?.path ?? null;
} catch {
return null;
}
};
/**
* Resolve the FTS extension the same way doctor/analyze will after vendoring:
* packaged artifact first, then a `~/.lbdb` install. File-path CI gates must
* accept the vendored file so they stay green when ensure-fts no longer
* populates the home cache.
*/
export const resolveFtsExtension = (opts?: {
vendorRoot?: string;
homeExtensionRoot?: string;
}): string | null =>
resolveVendoredFtsPath({ vendorRoot: opts?.vendorRoot }) ??
findInstalledFtsExtension(opts?.homeExtensionRoot ?? join(homedir(), '.lbdb', 'extension'));
export const FTS_UNAVAILABLE_NOTE =
'FTS extension unavailable (load-only policy; LOAD failed on this machine)';
/**
* Dynamically skip an FTS-primitive test when the extension cannot load.
* `ctx.skip()` aborts the test, so callers should `await` this first thing.
*
* Honors GITNEXUS_REQUIRE_FTS=1 the same way `withTestLbugDB` does (see
* test/helpers/test-indexed-db.ts): when CI sets it, an unavailable extension is
* a HARD FAILURE, never a silent skip — otherwise these FTS-primitive tests
* (registered in LBUG_NATIVE, so they run on the ubuntu/macOS/windows jobs that
* all set GITNEXUS_REQUIRE_FTS=1) could vanish from a green run. Offline/local
* runs (no env var) still skip gracefully (#2299).
*
* Self-sufficient under sharding: the default load path is `load-only`, so these
* primitives only pass when *some other* test already installed FTS into the
* shared home. That co-location is not guaranteed once the cross-platform suite
* is sharded (a load-only file can land in a shard with no installer sibling —
* exactly what broke `lbug-core-adapter` on shard 2/3). So under REQUIRE_FTS we
* install-on-miss with `auto` (LOAD-first, then one bounded network INSTALL),
* matching `withTestIndexedDB`, before treating it as a hard failure.
*/
export const skipUnlessFtsAvailable = async (ctx: {
skip: (note?: string) => void;
}): Promise<void> => {
const { loadFTSExtension } = await import('../../src/core/lbug/lbug-adapter.js');
if (await loadFTSExtension()) return;
if (process.env.GITNEXUS_REQUIRE_FTS === '1') {
// Not pre-installed in this (possibly-sharded) CI VM — install it once, then
// it stays available for the rest of this file's tests. `auto` is LOAD-first
// so a pre-installed extension still costs no network.
if (await loadFTSExtension(undefined, { policy: 'auto' })) return;
throw new Error(
'FTS extension is required (GITNEXUS_REQUIRE_FTS=1) but could not be loaded or installed. ' +
'FTS-dependent tests must not be silently skipped in CI — install/repair the LadybugDB ' +
'FTS extension (see `gitnexus doctor`) or unset GITNEXUS_REQUIRE_FTS for offline/local runs.',
);
}
ctx.skip(FTS_UNAVAILABLE_NOTE);
};
/**
* Skip a structural FTS test when a required on-disk artifact (the vendored
* extension, a home install, or the native addon) is not resolvable — but
* HARD-FAIL under GITNEXUS_REQUIRE_FTS=1 (#2299, #2383 F6d) so it never
* silently vanishes from a green CI run. Used by tests that inspect the
* extension *file* directly and so need its path rather than a loaded
* connection (skipUnlessFtsAvailable needs an initialized LadybugDB).
*/
export const requireFtsResourceOrSkip = (
ctx: { skip: (note?: string) => void },
resource: string | null,
note: string,
): void => {
if (resource) return;
if (process.env.GITNEXUS_REQUIRE_FTS === '1') {
throw new Error(
`${note} is required (GITNEXUS_REQUIRE_FTS=1) but was not found on this machine. ` +
'FTS-dependent tests must not be silently skipped in CI — install/repair the LadybugDB ' +
'FTS extension (see `gitnexus doctor`) or unset GITNEXUS_REQUIRE_FTS for offline/local runs.',
);
}
ctx.skip(`${note} unavailable`);
};