scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-10-05 02:43:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

feat(analyze): add explicit FTS opt-out (#3205)

Browse source 
* feat(analyze): add explicit FTS opt-out

* chore(autofix): apply prettier + eslint fixes via /autofix command

* fix(analyze): treat flag and env FTS opt-out as one mode

Avoid a same-commit rebuild when only the skipReason discriminator
changes, and advertise disablement on dirty meta before leftover
indexes are wiped.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
azizur100389 2026-09-11 15:05:28 +01:00 • committed by GitHub
parent d1a3edd333
commit f8036ac349
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
27 changed files with 1389 additions and 245 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
README.md
View file

@ -407,7 +407,7 @@ backoff. Invalid `.gitnexusrc` or ignore-file reloads pause ordinary refreshes
until the control file is fixed. Stop the watcher with Ctrl+C.
Watch mode accepts `--debounce`, `--workers`, `--worker-timeout`,
`--max-file-size`, `--branch`, `--pdg`, `--name`, `--allow-duplicate-name`, and
`--max-file-size`, `--branch`, `--pdg`, `--skip-fts`, `--name`, `--allow-duplicate-name`, and
`--verbose`. Explicit one-shot options such as `--force`, `--repair-fts`,
embedding flags, `--skills`, `--self-commit`, `--index-only`, and `--skip-git`
are rejected. Unsupported defaults from `.gitnexusrc` are ignored with a
@ -441,6 +441,7 @@ The token may be set in the shell, `.env.local`, or `.env` in the working direct
gitnexus analyze --force # Full graph + FTS rebuild (reuses unchanged parser output)
gitnexus analyze --no-parse-cache # Full rebuild that re-parses every source file
gitnexus analyze --repair-fts # Fast path: rebuild/verify only FTS indexes on existing index data
gitnexus analyze --skip-fts # Index graph/embeddings without loading FTS or building keyword indexes
gitnexus analyze --skills # Generate repo-specific skill files from detected communities
gitnexus analyze --skip-embeddings # Skip embedding generation (faster)
gitnexus analyze --embeddings [limit] # Enable embedding generation (slower, better search)
@ -458,6 +459,8 @@ gitnexus analyze --wal-checkpoint-threshold 67108864 # LadybugDB WAL auto-check
# (default 67108864 = 64 MiB; -1 keeps Ladybug stock ~16 MiB)
```
`--skip-fts` (or `GITNEXUS_SKIP_FTS=1`) disables FTS extension loading and keyword-index construction for this analysis. Graph queries, communities, processes, and existing embeddings remain available. Status and search report "FTS disabled for this index". Remove both the flag and environment setting and run `analyze` again to restore keyword search, even at the same commit. Only the exact environment value `1` enables the opt-out; the flag takes precedence. It cannot be combined with `--repair-fts`. Disabling an existing FTS index may require one graph-store rebuild to avoid unsafe writes through native indexes.
`--spring-actuator` is explicitly opt-in and accepts either a JSON bundle keyed by `mappings`, `beans`, `conditions`, `configprops`, and/or `env`, or a directory containing endpoint-named JSON files. It confirms matching static nodes and adds conservative runtime-only routes, beans, and property keys. The configured input is excluded from source scanning; only normalized repository-relative exclusions are retained for future scans, never absolute paths. Env/configprops values, origins, condition messages, and source names are never persisted or printed. Because snapshots are external runtime state, an enabled run always rebuilds; the first later run without the option rebuilds once to remove runtime evidence. The same path can be set as `springActuator` in `.gitnexusrc`.
`--asyncapi-spec` is explicitly opt-in and accepts a directory of AsyncAPI documents or a single document; the path is resolved against the repository root, so a committed `docs/asyncapi` and an absolute cache written by something else both work. Each `operations[]` entry of an **AsyncAPI 3.x** document can contribute a `Destination` node keyed by broker and address, with `action: send` emitting `PUBLISHES_TO` and `action: receive` emitting `CONSUMES_FROM`, so a document and source code that name one address on one broker land on the same node. Edges start at the document, not at a callable — a document states that the service talks to an address, not which method does — and no address a document names is ever attached to an unresolved source site.
@ -595,6 +598,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
| `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` | `30000` | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000. | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s. |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget in milliseconds for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. | Slow or heavily loaded hosts where a full pool cold-starting concurrently needs more than 5s, and analyze aborts with "did not report ready within 5000ms". |
| `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. |
| `GITNEXUS_SKIP_FTS` | unset | When exactly `1`, skips FTS extension loading and keyword index creation during analyze. Equivalent to `--skip-fts`; a later analyze without either option restores FTS. | Graph-only consumers with their own retrieval, or short-lived indexes that do not need keyword search. |
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold <bytes>`. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. |
| `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling in bytes for every GitNexus database (analyze, MCP server, serve, group bridges). `0` restores LadybugDB's native unbounded default of 80% of system RAM; invalid values warn and fall back to the default (#2557). During `analyze` the pool is right-sized to the graph, scaled on non-4 KiB-page hosts by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. | A long-lived `gitnexus mcp` or a big incremental `analyze` uses too much memory, or a huge repo's working set genuinely needs a pool larger than 2 GiB. |
| `GITNEXUS_LBUG_MAX_DB_SIZE` | `17179869184` (16 GiB) | Maximum size in bytes of a single LadybugDB database file — an mmap/disk-address-space ceiling, not a memory limit (it does not constrain the buffer pool). Invalid values silently fall back to the default. | Indexing a genuinely huge monorepo whose on-disk graph index approaches 16 GiB. |

1
gitnexus/scripts/cross-platform-tests.ts
View file

@ -142,6 +142,7 @@ const PLATFORM_LOGIC = [
// N-API addon which has known platform-specific behavior (Windows
// file-lock lag after close, macOS N-API destructor segfaults)
const LBUG_NATIVE = [
'test/integration/skip-fts.test.ts',
'test/integration/lbug-core-adapter.test.ts',
'test/integration/lbug-vector-extension.test.ts',
'test/integration/lbug-pool.test.ts',

1
gitnexus/src/cli/analyze-options.ts
View file

@ -23,6 +23,7 @@ export interface AnalyzeOptions {
/** Commander negated flag: false only when --no-parse-cache is passed. */
parseCache?: boolean;
repairFts?: boolean;
skipFts?: boolean;
/**
* Embedding generation toggle. Commander parses `--embeddings [limit]` as:
* - `undefined` when the flag is omitted

1
gitnexus/src/cli/analyze-watch.ts
View file

@ -176,6 +176,7 @@ export async function resolveWatchOptions(
return {
pdg: merged.pdg,
skipFts: merged.skipFts,
branch,
registryName: merged.name,
allowDuplicateName: merged.allowDuplicateName,

7
gitnexus/src/cli/analyze.ts
View file

@ -13,6 +13,7 @@ import os from 'os';
import { spawn } from 'child_process';
import v8 from 'v8';
import cliProgress from 'cli-progress';
import { FTS_DISABLED_MESSAGE, isExplicitFtsDisablement } from '../core/search/fts-policy.js';
import { isLbugReady, LbugWipeError } from '../core/lbug/lbug-adapter.js';
import { boundedCheckpointBeforeExit } from '../core/lbug/shutdown-helpers.js';
import { findUndeclaredRelationPairError } from '../core/lbug/rel-pair-routing.js';
@ -1349,6 +1350,7 @@ const analyzeCommandImpl = async (
force: options.force || options.skills || options.parseCache === false,
useParseCache: options.parseCache !== false,
repairFts: options.repairFts,
skipFts: options.skipFts,
embeddings: embeddingsEnabled,
embeddingsNodeLimit,
dropEmbeddings: options.dropEmbeddings,
@ -1450,6 +1452,7 @@ const analyzeCommandImpl = async (
console.error = origError;
bar.stop();
console.log(' Already up to date\n');
if (result.ftsSkipped) console.log(` ${FTS_DISABLED_MESSAGE}\n`);
if (runOptions.registryName) {
console.log(` Registry name: ${result.repoName}\n`);
}
@ -1610,7 +1613,9 @@ const analyzeCommandImpl = async (
if (result.ftsSkipped) {
// #2658 review L2: a build/verify failure is NOT an extension-unavailable
// problem — sending the user to install the extension is the wrong remedy.
if (result.ftsSkipReason === 'build-failed') {
if (isExplicitFtsDisablement(result.ftsSkipReason)) {
console.log(`\n ${FTS_DISABLED_MESSAGE}`);
} else if (result.ftsSkipReason === 'build-failed') {
console.log(
`\n Warning: full-text/BM25 search is disabled — the search index build failed this run.\n` +
` The FTS extension is available; rerun \`gitnexus analyze --repair-fts\`. If it persists,\n` +

1
gitnexus/src/cli/index.ts
View file

@ -81,6 +81,7 @@ program
'Re-parse every source file instead of replaying cached parser output',
)
.option('--repair-fts', 'Repair/rebuild search FTS indexes without full re-analysis')
.option('--skip-fts', 'Skip FTS extension loading and keyword search indexes')
.option(
'--embeddings [limit]',
'Enable embedding generation for semantic search (off by default). ' +

3
gitnexus/src/cli/status.ts
View file

@ -18,6 +18,7 @@ import {
resolveAnalyzerRunnerIdentity,
} from '../core/analyzer-identity.js';
import { getIndexIncompleteReasons } from '../core/index-freshness.js';
import { getFtsDisabledReason, FTS_DISABLED_MESSAGE } from '../core/search/fts-policy.js';
import { detectIndexContentDrift, type IndexContentDrift } from '../core/index-content-drift.js';
import { t } from './i18n/index.js';
@ -186,6 +187,7 @@ export const statusCommand = async (options: StatusOptions = {}) => {
index: {
indexedAt: activeMeta.indexedAt,
commit: activeMeta.lastCommit,
...(activeMeta.capabilities ? { capabilities: activeMeta.capabilities } : {}),
runnerIdentity: activeMeta.runnerIdentity ?? null,
runnerIdentityStatus: runnerIdentityIsCurrent ? 'current' : 'stale-or-unknown',
incompleteReasons,
@ -211,6 +213,7 @@ export const statusCommand = async (options: StatusOptions = {}) => {
console.log(`${t('status.indexed')}: ${new Date(activeMeta.indexedAt).toLocaleString()}`);
console.log(`${t('status.indexedCommit')}: ${activeMeta.lastCommit?.slice(0, 7)}`);
console.log(`${t('status.currentCommit')}: ${currentCommit?.slice(0, 7)}`);
if (getFtsDisabledReason(activeMeta.capabilities?.fts)) console.log(FTS_DISABLED_MESSAGE);
// Emit the complete, versioned receipt as JSON so humans can inspect it and
// automation can compare it without reverse-engineering a display string.
// `null` is the backward-compatible signal for pre-receipt metadata.

46
gitnexus/src/core/lbug/lbug-adapter.ts
View file

@ -709,8 +709,8 @@ const runSchemaCreationQueries = async (dbPath: string): Promise<unknown | null>
return null;
};
export const initLbug = async (dbPath: string) => {
return runWithSessionLock(() => ensureLbugInitialized(dbPath));
export const initLbug = async (dbPath: string, options: { skipFts?: boolean } = {}) => {
return runWithSessionLock(() => ensureLbugInitialized(dbPath, options));
};
/**
@ -724,14 +724,14 @@ export const initLbug = async (dbPath: string) => {
export const withLbugDb = async <T>(
dbPath: string,
operation: () => Promise<T>,
options: { readOnly?: boolean } = {},
options: { readOnly?: boolean; skipFts?: boolean } = {},
): Promise<T> => {
let lastError: unknown;
const readOnly = options.readOnly === true;
for (let attempt = 1; attempt <= DB_LOCK_RETRY_ATTEMPTS; attempt++) {
try {
return await runWithSessionLock(async () => {
await ensureLbugInitialized(dbPath, readOnly);
await ensureLbugInitialized(dbPath, { readOnly, skipFts: options.skipFts });
return operation();
});
} catch (err) {
@ -762,15 +762,32 @@ export const withLbugDb = async <T>(
throw lastError;
};
const ensureLbugInitialized = async (dbPath: string, readOnly: boolean = false) => {
if (conn && currentDbPath === dbPath && currentDbReadOnly === readOnly) {
let currentDbSkipFts = false;
const ensureLbugInitialized = async (
dbPath: string,
options: { readOnly?: boolean; skipFts?: boolean } = {},
) => {
const readOnly = options.readOnly === true;
const skipFts = options.skipFts === true;
if (
conn &&
currentDbPath === dbPath &&
currentDbReadOnly === readOnly &&
currentDbSkipFts === skipFts
) {
return { db, conn };
}
await doInitLbug(dbPath, readOnly);
await doInitLbug(dbPath, { readOnly, skipFts });
return { db, conn };
};
const doInitLbug = async (dbPath: string, readOnly: boolean = false) => {
const doInitLbug = async (
dbPath: string,
options: { readOnly?: boolean; skipFts?: boolean } = {},
) => {
const readOnly = options.readOnly === true;
const skipFts = options.skipFts === true;
// Different database requested — close the old one first
if (conn || db) {
await safeClose();
@ -942,8 +959,11 @@ const doInitLbug = async (dbPath: string, readOnly: boolean = false) => {
// Phase 3 installs it moments later in the same run. Warning here reported a
// degradation that never happened — the run went on to build every FTS index.
// Phase 3 (and the read-only branch) still warn for real failures.
await loadFTSExtension(undefined, readOnly ? { policy: 'load-only' } : { quiet: true });
if (!skipFts) {
await loadFTSExtension(undefined, readOnly ? { policy: 'load-only' } : { quiet: true });
}
currentDbSkipFts = skipFts;
currentDbPath = dbPath;
return { db, conn };
};
@ -3679,7 +3699,10 @@ export const ensureEmbeddingRowDmlSafe = async (
* read once per run. {@link INDEX_CATALOG_UNREADABLE} fails closed here without
* a second read; omitting the argument makes the gate read for itself.
*/
export const ensureFtsRowDmlSafe = async (indexRows?: IndexCatalogSnapshot): Promise<boolean> => {
export const ensureFtsRowDmlSafe = async (
indexRows?: IndexCatalogSnapshot,
options: { skipFts?: boolean } = {},
): Promise<boolean> => {
// Unconditional precondition, same regression as the VECTOR twin's (#2841
// review §5.B): a caller-supplied snapshot must not let a closed DB be
// answered `true`.
@ -3706,6 +3729,9 @@ export const ensureFtsRowDmlSafe = async (indexRows?: IndexCatalogSnapshot): Pro
return indexType === undefined || indexType === 'FTS';
});
if (!indexGatesDml) return true;
// Existing/unknown native indexes still gate writes. Rebuild into a fresh
// database rather than loading FTS or issuing unsafe DML when opted out.
if (options.skipFts) return false;
return await loadFTSExtension(undefined, { policy: resolveAnalyzeInstallPolicy() });
};

101
gitnexus/src/core/run-analyze.ts
View file

@ -10,6 +10,13 @@
*/
import { detectGraphWriteCollapse, type GraphWriteCollapseVerdict } from './index-freshness.js';
import {
resolveFtsDisableReason,
getFtsDisabledReason,
withExplicitFtsDisablement,
FTS_DISABLED_MESSAGE,
type FtsSkipReason,
} from './search/fts-policy.js';
import { PDG_EDGE_TYPES } from './lbug/pdg-emit-sink.js';
import path from 'path';
import fs from 'fs/promises';
@ -353,6 +360,7 @@ export interface AnalyzeOptions {
useParseCache?: boolean;
/** Repair only search indexes without re-running full parsing/indexing. */
repairFts?: boolean;
skipFts?: boolean;
/** Emit per-index FTS create logs. */
verbose?: boolean;
embeddings?: boolean;
@ -559,8 +567,10 @@ export interface AnalyzeResult {
* extension loaded but the index build/verify failed non-fatally — remedied by
* `--repair-fts`, not by installing the extension). Lets the CLI show the
* correct recovery hint instead of always blaming a missing extension.
* `disabled-by-flag` and `disabled-by-env` record intentional opt-out;
* neither calls for extension installation or repair.
*/
ftsSkipReason?: 'extension-unavailable' | 'build-failed';
ftsSkipReason?: FtsSkipReason;
/**
* True when the index this run produced/validated is the flat workspace
* slot (#2106 R2, inverted by #2354 to follow the checked-out branch).
@ -1077,6 +1087,9 @@ export async function runFullAnalysis(
// Validate operator-provided FTS config before anything else — a typo fails
// here in ms, without taking the lock. (createSearchFTSIndexes reuses the
// cached value via getSearchFTSStemmer.)
if (options.repairFts && resolveFtsDisableReason(options.skipFts)) {
throw new Error('--repair-fts cannot be used with --skip-fts or GITNEXUS_SKIP_FTS=1.');
}
initialiseSearchFTSStemmer();
initialiseSearchFTSCjkSegmentation();
// Scope the degraded-parse log throttle to this run (module-level counter
@ -1145,6 +1158,9 @@ async function runFullAnalysisInner(
writeTarget: WriteTarget,
runnerIdentityAtBootstrap?: AnalyzerRunnerIdentity,
): Promise<AnalyzeResult> {
const ftsDisabledReason = resolveFtsDisableReason(options.skipFts);
const initAnalysisLbug = (dbPath: string) =>
ftsDisabledReason ? initLbug(dbPath, { skipFts: true }) : initLbug(dbPath);
const log = (msg: string) => callbacks.onLog?.(stripControlCharacters(msg));
const progress = (phase: string, percent: number, message: string) =>
callbacks.onProgress(phase, percent, message);
@ -1185,7 +1201,19 @@ async function runFullAnalysisInner(
log(`Metadata reconciliation failed (non-critical${code ? `, ${code}` : ''}); continuing.`);
}
const existingMeta = await loadMeta(metaDir);
const loadedMeta = await loadMeta(metaDir);
const previousFtsDisabledReason = getFtsDisabledReason(loadedMeta?.capabilities?.fts);
// Flag and env are equivalent disablements. Only a true enable↔disable flip
// needs a write plan; a discriminator-only change restamps on the
// already-up-to-date path.
const ftsModeChanged = Boolean(ftsDisabledReason) !== Boolean(previousFtsDisabledReason);
// Fold explicit disablement into the in-memory prior meta so every later
// saveMeta that spreads it (dirty flag, incremental phase stamps) advertises
// "FTS disabled" instead of leftover available/build-failed while a wipe is
// in flight. Re-enable leaves the prior stamp untouched.
const existingMeta = loadedMeta
? withExplicitFtsDisablement(loadedMeta, ftsDisabledReason)
: undefined;
// ── FTS-only repair path ────────────────────────────────────────────
if (options.repairFts) {
@ -1238,7 +1266,7 @@ async function runFullAnalysisInner(
);
}
try {
await initLbug(lbugPath);
await initAnalysisLbug(lbugPath);
// Gate on FTS availability BEFORE touching any index. createSearchFTSIndexes
// now DROPs each index before recreating it (so schema changes reach existing
// DBs); if the extension were unavailable, the drops would run and leave the
@ -1783,7 +1811,8 @@ async function runFullAnalysisInner(
existingMeta &&
!existingMeta.embeddingCheckpoint &&
!options.force &&
existingMeta.lastCommit === currentCommit
existingMeta.lastCommit === currentCommit &&
!ftsModeChanged
) {
// Non-git folders have currentCommit = '' — always rebuild since we can't detect changes
if (currentCommit !== '') {
@ -1892,6 +1921,19 @@ async function runFullAnalysisInner(
`Warning: could not restamp the workspace branch label (${reason}); will retry on the next run.`,
);
}
} else if (ftsDisabledReason && ftsDisabledReason !== previousFtsDisabledReason) {
// Discriminator-only restamp (flag↔env). `existingMeta` already
// carries the folded skipReason; persist it without a write plan.
try {
await saveMeta(metaDir, existingMeta);
} catch (err) {
const reason = isReadOnlyFilesystemError(err)
? `${(err as Error).message} — storage may be read-only (#1549)`
: (err as Error).message;
log(
`Warning: could not restamp the FTS skip reason (${reason}); will retry on the next run.`,
);
}
}
await ensureGitNexusIgnored(repoPath);
return {
@ -1905,6 +1947,7 @@ async function runFullAnalysisInner(
repoPath,
stats: existingMeta.stats ?? {},
alreadyUpToDate: true,
...(ftsDisabledReason ? { ftsSkipped: true, ftsSkipReason: ftsDisabledReason } : {}),
isPrimaryBranch: !placement.branch,
};
}
@ -1975,7 +2018,7 @@ async function runFullAnalysisInner(
if (shouldLoadCache && existingMeta) {
try {
progress('embeddings', 0, 'Caching embeddings...');
await initLbug(lbugPath);
await initAnalysisLbug(lbugPath);
const cached = await loadCachedEmbeddings();
cachedEmbeddingNodeIds = cached.embeddingNodeIds;
cachedEmbeddings = cached.embeddings;
@ -2356,7 +2399,7 @@ async function runFullAnalysisInner(
// Full rebuild (POSIX) builds into the temp `buildPath`; incremental and
// Windows use `buildPath === lbugPath` in place.
try {
await initLbug(buildPath);
await initAnalysisLbug(buildPath);
} catch (error) {
if (liveIndexMutationStarted) recordLiveIndexMutationRisk(error);
throw error;
@ -2673,7 +2716,9 @@ async function runFullAnalysisInner(
// creates or drops an index.
const indexCatalogRows = await readIndexCatalogSnapshot();
const embeddingRowDmlSafe = await ensureEmbeddingRowDmlSafe(indexCatalogRows);
const ftsRowDmlSafe = await ensureFtsRowDmlSafe(indexCatalogRows);
const ftsRowDmlSafe = ftsDisabledReason
? await ensureFtsRowDmlSafe(indexCatalogRows, { skipFts: true })
: await ensureFtsRowDmlSafe(indexCatalogRows);
const extensionForcedRebuild = !embeddingRowDmlSafe || !ftsRowDmlSafe;
// `!options.dropEmbeddings` (H1): this rescue reads the rows back OUT of
// the DB, so it must never fire on the one path whose entire purpose is to
@ -2763,7 +2808,7 @@ async function runFullAnalysisInner(
// catalog read happened to fail still had both gates answer "safe"
// (both extensions loaded), and claiming otherwise would trade one
// invented cause for another.
if (extensionForcedRebuild && indexCatalogUnreadable) {
if (extensionForcedRebuild && indexCatalogUnreadable && !ftsDisabledReason) {
const blockedExtensions = [
!embeddingRowDmlSafe ? 'VECTOR' : undefined,
!ftsRowDmlSafe ? 'FTS' : undefined,
@ -2787,7 +2832,12 @@ async function runFullAnalysisInner(
'Semantic search falls back to exact scan until VECTOR is available.',
);
}
if (!ftsRowDmlSafe) {
if (!ftsRowDmlSafe && ftsDisabledReason) {
escalationCauses.push(
'FTS is explicitly disabled and existing search indexes could not be ruled out; ' +
'a fresh graph store is required before writing rows without the extension',
);
} else if (!ftsRowDmlSafe) {
if (!indexCatalogUnreadable) {
// Self-contained subject (H5): `join('; and ')` used to render "…the
// CodeEmbedding vector index exists … and THIS INDEX carries FTS
@ -2838,7 +2888,9 @@ async function runFullAnalysisInner(
!embeddingRowDmlSafe
? { reason: getExtensionCapability('VECTOR')?.reason, label: 'VECTOR' }
: undefined,
!ftsRowDmlSafe ? { reason: getFtsCapability()?.reason, label: 'FTS' } : undefined,
!ftsRowDmlSafe && !ftsDisabledReason
? { reason: getFtsCapability()?.reason, label: 'FTS' }
: undefined,
]
.filter((e): e is { reason: string | undefined; label: string } => e !== undefined)
.map(({ reason, label }) => diagnoseExtensionLoad(reason, label).remedy);
@ -2921,7 +2973,7 @@ async function runFullAnalysisInner(
await closeLbug();
if (buildPath === lbugPath) liveIndexMutationStarted = true;
await wipeLbugDbFiles(buildPath);
await initLbug(buildPath);
await initAnalysisLbug(buildPath);
walCheckpointDriver = startWalCheckpointDriver();
await loadGraphToLbug(pipelineResult.graph, pipelineResult.repoPath, storagePath, (msg) => {
lbugMsgCount++;
@ -3139,20 +3191,28 @@ async function runFullAnalysisInner(
// analyze still produces a fully queryable graph; only full-text/BM25
// search falls back. `--repair-fts` (whose sole job is FTS) still fails
// loudly on its own path above.
progress('fts', 85, 'Creating search indexes...');
const ftsAvailable = await loadFTSExtension(undefined, {
policy: resolveAnalyzeInstallPolicy(),
});
progress(
'fts',
85,
ftsDisabledReason ? 'Skipping search indexes...' : 'Creating search indexes...',
);
const ftsAvailable =
!ftsDisabledReason &&
(await loadFTSExtension(undefined, {
policy: resolveAnalyzeInstallPolicy(),
}));
// Tracks whether search indexes actually ended up usable this run — starts
// as ftsAvailable (extension loaded) but flips to false below when the
// build/verify step itself fails, so capabilities.fts.status / ftsSkipped
// stay honest even though that failure no longer aborts the whole analyze.
let ftsReady = ftsAvailable;
// Why FTS ended up skipped (#2658 review L2): extension-unavailable up front,
// or build-failed in the degrade branch below.
let ftsSkipReason: 'extension-unavailable' | 'build-failed' | undefined = ftsAvailable
// Why FTS ended up skipped (#2658 review L2): an explicit opt-out
// (`disabled-by-flag` / `disabled-by-env`, #3091) when one was recorded,
// else extension-unavailable up front, or build-failed in the degrade
// branch below.
let ftsSkipReason: FtsSkipReason | undefined = ftsAvailable
? undefined
: 'extension-unavailable';
: (ftsDisabledReason ?? 'extension-unavailable');
if (ftsAvailable) {
// Degrade rather than throw: createSearchFTSIndexes re-tokenizes every
// stored row on every run, so a native tokenizer error on a single
@ -3193,6 +3253,9 @@ async function runFullAnalysisInner(
);
progress('fts', 90, 'Search indexes skipped (build failed)');
}
} else if (ftsDisabledReason) {
log(FTS_DISABLED_MESSAGE);
progress('fts', 90, 'Search indexes skipped (explicitly disabled)');
} else {
// For a missing runtime dependency (#2374) the file is present, so the
// generic "install it with network access" tail in FTS_UNAVAILABLE_MESSAGE

16
gitnexus/src/core/search/bm25-index.ts
View file

@ -2,7 +2,10 @@
* Full-Text Search via LadybugDB FTS
*
* Uses LadybugDB's built-in full-text search indexes for keyword-based search.
* Always reads from the database (no cached state to drift).
* Reads from the database on every query it runs (no cached state to drift).
* The one exception is an explicit opt-out (#3091): when a caller passes an
* `FtsDisabledReason`, the search short-circuits to an empty, unavailable
* response without opening or querying the database at all.
*/
// tri-review Residual-1: `classifyFtsQueryError` now lives in lbug-adapter.ts
@ -16,6 +19,7 @@ import { normalizeFtsText } from '../lbug/csv-generator.js';
import { getExtensionCapabilities } from '../lbug/extension-loader.js';
import { redactPaths } from './fts-indexes.js';
import { FTS_INDEXES } from './fts-schema.js';
import type { FtsDisabledReason } from './fts-policy.js';
import {
applyCjkSegmentationIfEnabled,
MAX_CJK_SEGMENTATION_QUERY_LENGTH,
@ -95,21 +99,29 @@ async function queryFTSViaExecutor(
}
/**
* Search using LadybugDB's built-in FTS (always fresh, reads from disk)
* Search using LadybugDB's built-in FTS (fresh, reads from disk)
*
* Queries multiple node tables (File, Function, Class, Method) in parallel
* and merges results by filePath, summing scores for the same file.
*
* When `disabledReason` is set the index intentionally has no FTS (#3091), so
* this returns an empty `ftsAvailable: false` response immediately and never
* touches the database — callers render that as deliberate disablement rather
* than as a missing index or a failed extension load.
*
* @param query - Search query string
* @param limit - Maximum results
* @param repoId - If provided, queries will be routed via the MCP connection pool
* @param disabledReason - Explicit FTS opt-out recorded for this index; short-circuits the search
* @returns Ranked search results from FTS indexes
*/
export const searchFTSFromLbug = async (
query: string,
limit: number = 20,
repoId?: string,
disabledReason?: FtsDisabledReason,
): Promise<FTSSearchResponse> => {
if (disabledReason) return { results: [], ftsAvailable: false };
// Applied once, up front, so every downstream branch searches with the
// same text the index was built from (#2331/#2339) — index-time and
// query-time text transforms must never diverge, since QUERY_FTS_INDEX

7
gitnexus/src/core/search/fts-indexes.ts
View file

@ -8,6 +8,7 @@ import {
type IndexCatalogSnapshot,
} from '../lbug/lbug-adapter.js';
import { getFtsCapability } from '../lbug/extension-loader.js';
import { FTS_DISABLED_MESSAGE, type FtsDisabledReason } from './fts-policy.js';
import { classifyExtensionLoadError } from '../lbug/extension-load-error.js';
import { FTS_INDEXES } from './fts-schema.js';
@ -73,8 +74,12 @@ const formatWarningContext = (context: FtsWarningContext): string => {
* text itself (#2767). Optional and additive: omitting it reproduces today's
* exact message.
*/
export const ftsDegradedWarning = (context?: FtsWarningContext): string => {
export const ftsDegradedWarning = (
context?: FtsWarningContext,
disabledReason?: FtsDisabledReason,
): string => {
const suffix = context ? formatWarningContext(context) : '';
if (disabledReason) return FTS_DISABLED_MESSAGE + suffix;
const fts = getFtsCapability();
if (fts && !fts.loaded) {
const reason = fts.reason ? redactPaths(fts.reason).replace(/\.$/, '') : undefined;

76
gitnexus/src/core/search/fts-policy.ts Normal file
View file

@ -0,0 +1,76 @@
import type { RepoMeta } from '../../storage/repo-meta.js';
export type FtsDisabledReason = 'disabled-by-flag' | 'disabled-by-env';
export type FtsSkipReason = FtsDisabledReason | 'extension-unavailable' | 'build-failed';
type RepoCapabilities = NonNullable<RepoMeta['capabilities']>;
const DEFAULT_GRAPH_CAPABILITY: RepoCapabilities['graph'] = {
provider: 'ladybugdb',
status: 'available',
};
const DEFAULT_VECTOR_SEARCH_CAPABILITY: RepoCapabilities['vectorSearch'] = {
provider: 'exact-scan',
status: 'unavailable',
exactScanLimit: 0,
};
export function resolveFtsDisableReason(
skipFts?: boolean,
envValue = process.env.GITNEXUS_SKIP_FTS,
): FtsDisabledReason | undefined {
if (skipFts === true) return 'disabled-by-flag';
if (envValue === '1') return 'disabled-by-env';
return undefined;
}
export function isExplicitFtsDisablement(reason: string | undefined): reason is FtsDisabledReason {
return reason === 'disabled-by-flag' || reason === 'disabled-by-env';
}
export function getFtsDisabledReason(
capability: RepoCapabilities['fts'] | undefined,
): FtsDisabledReason | undefined {
if (capability?.status !== 'unavailable') return undefined;
return isExplicitFtsDisablement(capability.skipReason) ? capability.skipReason : undefined;
}
/**
* Overlay an explicit FTS opt-out onto an existing meta snapshot without
* touching freshness (`indexedAt` / `lastCommit`) or sibling capabilities.
* Flag and env are equivalent disablements; only the discriminator changes.
* Returns `meta` unchanged when `reason` is absent (re-enable) or already stamped.
*/
export function withExplicitFtsDisablement(
meta: RepoMeta,
reason: FtsDisabledReason | undefined,
): RepoMeta {
if (!reason) return meta;
const existing = meta.capabilities;
const existingFts = existing?.fts;
if (
existingFts?.status === 'unavailable' &&
existingFts.skipReason === reason &&
existing?.graph &&
existing.vectorSearch
) {
return meta;
}
return {
...meta,
capabilities: {
graph: existing?.graph ?? DEFAULT_GRAPH_CAPABILITY,
fts: {
provider: existingFts?.provider ?? 'ladybugdb-fts',
status: 'unavailable',
skipReason: reason,
},
vectorSearch: existing?.vectorSearch ?? DEFAULT_VECTOR_SEARCH_CAPABILITY,
},
};
}
export const FTS_DISABLED_MESSAGE =
'FTS disabled for this index. To enable keyword search, run gitnexus analyze ' +
'without --skip-fts and with GITNEXUS_SKIP_FTS unset.';

15
gitnexus/src/core/search/hybrid-search.ts
View file

@ -10,6 +10,7 @@
import { searchFTSFromLbug, type BM25SearchResult } from './bm25-index.js';
import type { SemanticSearchResult } from '../embeddings/types.js';
import type { FtsDisabledReason } from './fts-policy.js';
/**
* RRF constant - standard value used in the literature
@ -153,11 +154,13 @@ export const formatHybridResults = (results: HybridSearchResult[]): string => {
/**
* Execute BM25 + semantic search and merge with RRF.
* Uses LadybugDB FTS for always-fresh BM25 results (no cached data).
* Uses LadybugDB FTS for fresh BM25 results (no cached data).
* The semanticSearch function is injected to keep this module environment-agnostic.
*
* When FTS is unavailable (e.g. read-only MCP connection, missing indexes),
* falls back to semantic-only results instead of crashing (#1489).
* When FTS is unavailable (e.g. read-only MCP connection, missing indexes) or
* explicitly disabled for this index (`disabledReason`, #3091), falls back to
* semantic-only results instead of crashing (#1489). In the disabled case no
* BM25 query is issued at all.
*/
export const hybridSearch = async (
query: string,
@ -168,13 +171,15 @@ export const hybridSearch = async (
query: string,
k?: number,
) => Promise<SemanticSearchResult[]>,
disabledReason?: FtsDisabledReason,
): Promise<HybridSearchResult[]> => {
// Use LadybugDB FTS for always-fresh BM25 results.
// Use LadybugDB FTS for fresh BM25 results — skipped entirely when this
// index recorded an explicit FTS opt-out (`disabledReason`, #3091).
// If FTS fails (e.g. extension not loaded in MCP process), fall back to
// semantic-only search instead of crashing with "bm25Results is not iterable".
let bm25Results: BM25SearchResult[] = [];
try {
const ftsResponse = await searchFTSFromLbug(query, limit);
const ftsResponse = await searchFTSFromLbug(query, limit, undefined, disabledReason);
bm25Results = ftsResponse?.results ?? [];
} catch {
// FTS unavailable — continue with semantic-only search

10
gitnexus/src/mcp/local/local-backend.ts
View file

@ -90,6 +90,7 @@ import {
import { getExactScanLimit } from '../../core/platform/capabilities.js';
import { PhaseTimer } from '../../core/search/phase-timer.js';
import { ftsDegradedWarning, ftsQueryFailedWarning } from '../../core/search/fts-indexes.js';
import { getFtsDisabledReason, type FtsDisabledReason } from '../../core/search/fts-policy.js';
import {
cjkSegmentationModeMismatch,
containsSegmentableCjkRun,
@ -2915,8 +2916,10 @@ export class LocalBackend {
// each so both get independent wall-time records without fighting
// over a single `current` phase slot.
const searchLimit = processLimit * maxSymbolsPerProcess; // fetch enough raw results
const meta = await loadMeta(path.dirname(repo.lbugPath));
const ftsDisabledReason = getFtsDisabledReason(meta?.capabilities?.fts);
const [bm25SearchResult, semanticResults] = await Promise.all([
timer.time('bm25', this.bm25Search(repo, searchQuery, searchLimit)),
timer.time('bm25', this.bm25Search(repo, searchQuery, searchLimit, ftsDisabledReason)),
timer.time('vector', this.semanticSearch(repo, searchQuery, searchLimit)),
]);
@ -3248,7 +3251,7 @@ export class LocalBackend {
warnings.push(
ftsQueryErrors
? ftsQueryFailedWarning({ ...warningContext, lastErrorRedacted: ftsQueryErrors[0] })
: ftsDegradedWarning(warningContext),
: ftsDegradedWarning(warningContext, ftsDisabledReason),
);
} else if (ftsQueryErrors) {
// #2767: at least one FTS table succeeded (ftsUsed=true) but another
@ -3302,7 +3305,6 @@ export class LocalBackend {
// GITNEXUS_FTS_CJK_SEGMENTATION (the only thing that actually throws in
// there) cannot take an unrelated diagnostic down with it. Needs no guard
// of its own: loadMeta() returns null on any read/parse failure.
const meta = await loadMeta(path.dirname(repo.lbugPath));
try {
// meta.json is on-disk state inside the analyzed repo, read via a
// schema-less JSON.parse — not trusted input. Validate before
@ -3408,7 +3410,9 @@ export class LocalBackend {
repo: RepoHandle,
query: string,
limit: number,
disabledReason?: FtsDisabledReason,
): Promise<{ results: any[]; ftsUsed: boolean; nonBenignErrors?: string[] }> {
if (disabledReason) return { results: [], ftsUsed: false };
let searchFTSFromLbug;
try {
({ searchFTSFromLbug } = await import('../../core/search/bm25-index.js'));

425
gitnexus/src/server/api.ts
View file

@ -37,6 +37,7 @@ import { NODE_TABLES, type GraphNode, type GraphRelationship } from 'gitnexus-sh
import { searchFTSFromLbug } from '../core/search/bm25-index.js';
import { hybridSearch } from '../core/search/hybrid-search.js';
import { ftsDegradedWarning } from '../core/search/fts-indexes.js';
import { getFtsDisabledReason, type FtsDisabledReason } from '../core/search/fts-policy.js';
import { LocalBackend } from '../mcp/local/local-backend.js';
import { installServeMcpAuth, mountMCPEndpoints } from './mcp-http.js';
import { fileURLToPath } from 'url';
@ -724,6 +725,24 @@ export const handleFileRequest = async (
}
};
async function loadFtsSession(storagePath: string): Promise<{
meta: Awaited<ReturnType<typeof loadMeta>>;
ftsDisabledReason: FtsDisabledReason | undefined;
skipFts?: true;
}> {
const meta = await loadMeta(storagePath);
const ftsDisabledReason = getFtsDisabledReason(meta?.capabilities?.fts);
return {
meta,
ftsDisabledReason,
...(ftsDisabledReason ? { skipFts: true as const } : {}),
};
}
function readOnlyFtsOptions(skipFts?: true): { readOnly: true; skipFts?: true } {
return skipFts ? { readOnly: true, skipFts: true } : { readOnly: true };
}
export const handleQueryRequest = async (
req: express.Request,
res: express.Response,
@ -749,9 +768,12 @@ export const handleQueryRequest = async (
return;
}
const lbugPath = path.join(entry.storagePath, 'lbug');
const result = await withLbugDb(lbugPath, () => executePrepared(cypher, queryParams ?? {}), {
readOnly: true,
});
const { skipFts } = await loadFtsSession(entry.storagePath);
const result = await withLbugDb(
lbugPath,
() => executePrepared(cypher, queryParams ?? {}),
readOnlyFtsOptions(skipFts),
);
res.json({ result });
} catch (err: any) {
if (isReadOnlyDbError(err)) {
@ -1151,6 +1173,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
const lbugPath = path.join(entry.storagePath, 'lbug');
const includeContent = req.query.includeContent === 'true';
const stream = req.query.stream === 'true';
const { skipFts } = await loadFtsSession(entry.storagePath);
if (stream) {
const abortController = new AbortController();
@ -1182,7 +1205,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
await withLbugDb(
lbugPath,
async () => streamGraphNdjson(res, includeContent, abortController.signal),
{ readOnly: true },
readOnlyFtsOptions(skipFts),
);
if (!abortController.signal.aborted && !res.writableEnded) {
res.end();
@ -1195,9 +1218,11 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
return;
}
const graph = await withLbugDb(lbugPath, async () => buildGraph(includeContent), {
readOnly: true,
});
const graph = await withLbugDb(
lbugPath,
async () => buildGraph(includeContent),
readOnlyFtsOptions(skipFts),
);
res.json(graph);
} catch (err: any) {
if (err instanceof ClientDisconnectedError) {
@ -1238,6 +1263,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
}
const lbugPath = path.join(entry.storagePath, 'lbug');
const parsedLimit = Number(req.body.limit ?? 10);
const { ftsDisabledReason, skipFts } = await loadFtsSession(entry.storagePath);
const limit = Number.isFinite(parsedLimit)
? Math.max(1, Math.min(100, Math.trunc(parsedLimit)))
: 10;
@ -1266,7 +1292,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
sources: ['semantic'],
}));
} else if (mode === 'bm25') {
const ftsResponse = await searchFTSFromLbug(query, limit);
const ftsResponse = await searchFTSFromLbug(query, limit, undefined, ftsDisabledReason);
ftsAvailable = ftsResponse.ftsAvailable;
searchResults = ftsResponse.results.map((r: any, i: number) => ({
...r,
@ -1279,9 +1305,21 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
if (isEmbedderReady()) {
const { semanticSearch: semSearch } =
await import('../core/embeddings/embedding-pipeline.js');
searchResults = await hybridSearch(query, limit, executeQuery, semSearch);
searchResults = await hybridSearch(
query,
limit,
executeQuery,
semSearch,
ftsDisabledReason,
);
if (ftsDisabledReason) ftsAvailable = false;
} else {
const ftsResponse = await searchFTSFromLbug(query, limit);
const ftsResponse = await searchFTSFromLbug(
query,
limit,
undefined,
ftsDisabledReason,
);
ftsAvailable = ftsResponse.ftsAvailable;
searchResults = ftsResponse.results;
}
@ -1375,11 +1413,11 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
return { searchResults: enriched, ftsAvailable };
},
{ readOnly: true },
readOnlyFtsOptions(skipFts),
);
const response: any = { results: results.searchResults ?? results };
if (results.ftsAvailable === false) {
response.warning = ftsDegradedWarning();
response.warning = ftsDegradedWarning(undefined, ftsDisabledReason);
}
res.json(response);
} catch (err: any) {
@ -1415,13 +1453,14 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
// cut a stuck regex.test() when the wall-clock budget expires.
const { regex, fileFilter, limit } = parseGrepQuery(req.query as Record<string, unknown>);
const repoRoot = path.resolve(entry.path);
const { skipFts } = await loadFtsSession(entry.storagePath);
const lbugPath = path.join(entry.storagePath, 'lbug');
const fileRows = await withLbugDb(
lbugPath,
() =>
executeQuery(`MATCH (n:File) WHERE n.content IS NOT NULL RETURN n.filePath AS filePath`),
{ readOnly: true },
readOnlyFtsOptions(skipFts),
);
const filePaths: string[] = [];
@ -1811,187 +1850,197 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
let partialRunDetail: AnalyzeJobPartialOutcome | undefined;
try {
const lbugPath = path.join(entry.storagePath, 'lbug');
await withLbugDb(lbugPath, async () => {
const { runEmbeddingPipeline } =
await import('../core/embeddings/embedding-pipeline.js');
const { resolveEmbeddingIdentity } =
await import('../core/embeddings/embedding-identity.js');
const embeddingIdentity = resolveEmbeddingIdentity();
let embeddingMeta = await loadMeta(entry.storagePath);
if (!embeddingMeta) {
throw new Error('Repository metadata is missing; run gitnexus analyze first');
}
const priorCheckpoint = embeddingMeta.embeddingCheckpoint;
// The SAME decision the CLI's resume gate makes
// (core/embedding-checkpoint.ts). This route used to hard-throw on
// any identity mismatch and ignore `attempts` entirely, so a
// `'partial'` marker written by `gitnexus analyze` and resumed
// here hit exactly the permanent wedge `kind` exists to remove:
// two readers of one record disagreeing about the rule it encodes.
// No `force`/`--drop-embeddings` equivalent exists on this route,
// so the flag options go unset and `'discard'` is unreachable —
// it is folded into the abandon arm rather than given an invented
// flag. `maxAttempts` is left to the shared default.
const resume = priorCheckpoint
? decideEmbeddingResume(priorCheckpoint, embeddingIdentity)
: undefined;
if (resume?.action === 'abort') throw new Error(resume.error);
if (resume?.action === 'abandon' || resume?.action === 'discard') {
logger.warn({ repo: entry.name }, resume.log);
}
const forceReembedNodeIds: ReadonlySet<string> =
resume?.action === 'resume' ? resume.pendingNodeIds : new Set<string>();
const saveEmbeddingCheckpoint = async (
checkpoint: {
nodesProcessed: number;
totalNodes: number;
chunksProcessed: number;
},
pendingNodeIds: string[],
embeddings?: PersistedEmbeddingCount,
): Promise<void> => {
// tri-review NEW-2: re-read immediately before writing (mirrors
// the pattern in run-analyze.ts's --repair-fts stamp) instead of
// spreading the stale `embeddingMeta` snapshot captured once at
// job start. This job can run up to EMBED_TIMEOUT_MS (30 min);
// without a fresh read, a concurrent writer's update (e.g. a
// --repair-fts capability stamp) would be silently reverted on
// every checkpoint save for the job's whole lifetime.
const latestMeta = (await loadMeta(entry.storagePath)) ?? embeddingMeta;
// `stats.embeddings` only moves when the caller MEASURED the
// live count (the post-flush `onCheckpoint`). The window-start
// callback measures nothing and passes nothing: restating the
// old count there would re-publish a stale number and clobber
// what a preceding `onCheckpoint` just wrote (same split as
// run-analyze.ts's checkpoint writer).
embeddingMeta = withMeasuredEmbeddingCount(
{
...latestMeta,
// In flight ⇒ `kind: 'interrupted'` (embedding-checkpoint.ts).
embeddingCheckpoint: mintInterruptedCheckpoint(
embeddingIdentity,
checkpoint,
pendingNodeIds,
),
const ftsSession = await loadFtsSession(entry.storagePath);
let embeddingMeta = ftsSession.meta;
await withLbugDb(
lbugPath,
async () => {
const { runEmbeddingPipeline } =
await import('../core/embeddings/embedding-pipeline.js');
const { resolveEmbeddingIdentity } =
await import('../core/embeddings/embedding-identity.js');
const embeddingIdentity = resolveEmbeddingIdentity();
if (!embeddingMeta) {
throw new Error('Repository metadata is missing; run gitnexus analyze first');
}
const priorCheckpoint = embeddingMeta.embeddingCheckpoint;
// The SAME decision the CLI's resume gate makes
// (core/embedding-checkpoint.ts). This route used to hard-throw on
// any identity mismatch and ignore `attempts` entirely, so a
// `'partial'` marker written by `gitnexus analyze` and resumed
// here hit exactly the permanent wedge `kind` exists to remove:
// two readers of one record disagreeing about the rule it encodes.
// No `force`/`--drop-embeddings` equivalent exists on this route,
// so the flag options go unset and `'discard'` is unreachable —
// it is folded into the abandon arm rather than given an invented
// flag. `maxAttempts` is left to the shared default.
const resume = priorCheckpoint
? decideEmbeddingResume(priorCheckpoint, embeddingIdentity)
: undefined;
if (resume?.action === 'abort') throw new Error(resume.error);
if (resume?.action === 'abandon' || resume?.action === 'discard') {
logger.warn({ repo: entry.name }, resume.log);
}
const forceReembedNodeIds: ReadonlySet<string> =
resume?.action === 'resume' ? resume.pendingNodeIds : new Set<string>();
const saveEmbeddingCheckpoint = async (
checkpoint: {
nodesProcessed: number;
totalNodes: number;
chunksProcessed: number;
},
embeddings,
);
await saveMeta(entry.storagePath, embeddingMeta);
};
/**
* Count the persisted rows, or report the answer never arrived.
* The TRI-STATE is carried to the fold rather than collapsed here:
* `unknown` is not 0, and only the fold knows what to carry
* forward instead (core/embedding-count.ts).
*/
const countPersistedEmbeddings = async (): Promise<PersistedEmbeddingCount> => {
const counted = await measurePersistedEmbeddingCount(executeQuery);
if (counted.kind === 'unknown') {
logger.warn(
{ reason: counted.reason },
'[embed] could not count persisted embeddings; leaving stats.embeddings untouched',
pendingNodeIds: string[],
embeddings?: PersistedEmbeddingCount,
): Promise<void> => {
// tri-review NEW-2: re-read immediately before writing (mirrors
// the pattern in run-analyze.ts's --repair-fts stamp) instead of
// spreading the stale `embeddingMeta` snapshot captured once at
// job start. This job can run up to EMBED_TIMEOUT_MS (30 min);
// without a fresh read, a concurrent writer's update (e.g. a
// --repair-fts capability stamp) would be silently reverted on
// every checkpoint save for the job's whole lifetime.
const latestMeta = (await loadMeta(entry.storagePath)) ?? embeddingMeta;
// `stats.embeddings` only moves when the caller MEASURED the
// live count (the post-flush `onCheckpoint`). The window-start
// callback measures nothing and passes nothing: restating the
// old count there would re-publish a stale number and clobber
// what a preceding `onCheckpoint` just wrote (same split as
// run-analyze.ts's checkpoint writer).
embeddingMeta = withMeasuredEmbeddingCount(
{
...latestMeta,
// In flight ⇒ `kind: 'interrupted'` (embedding-checkpoint.ts).
embeddingCheckpoint: mintInterruptedCheckpoint(
embeddingIdentity,
checkpoint,
pendingNodeIds,
),
},
embeddings,
);
await saveMeta(entry.storagePath, embeddingMeta);
};
/**
* Count the persisted rows, or report the answer never arrived.
* The TRI-STATE is carried to the fold rather than collapsed here:
* `unknown` is not 0, and only the fold knows what to carry
* forward instead (core/embedding-count.ts).
*/
const countPersistedEmbeddings = async (): Promise<PersistedEmbeddingCount> => {
const counted = await measurePersistedEmbeddingCount(executeQuery);
if (counted.kind === 'unknown') {
logger.warn(
{ reason: counted.reason },
'[embed] could not count persisted embeddings; leaving stats.embeddings untouched',
);
}
return counted;
};
// Fetch existing content hashes for incremental embedding.
// Delegated to lbug-adapter which owns the DB query logic and legacy-fallback handling.
const { fetchExistingEmbeddingHashes } =
await import('../core/lbug/lbug-adapter.js');
const existingEmbeddings = await fetchExistingEmbeddingHashes(executeQuery);
if (existingEmbeddings && existingEmbeddings.size > 0) {
console.log(
`[embed] ${existingEmbeddings.size} nodes already embedded — incremental run with content-hash comparison`,
);
}
return counted;
};
// Fetch existing content hashes for incremental embedding.
// Delegated to lbug-adapter which owns the DB query logic and legacy-fallback handling.
const { fetchExistingEmbeddingHashes } = await import('../core/lbug/lbug-adapter.js');
const existingEmbeddings = await fetchExistingEmbeddingHashes(executeQuery);
if (existingEmbeddings && existingEmbeddings.size > 0) {
console.log(
`[embed] ${existingEmbeddings.size} nodes already embedded — incremental run with content-hash comparison`,
);
}
const pipelineResult = await runEmbeddingPipeline(
executeQuery,
executeWithReusedStatement,
(p) => {
embedJobManager.updateJob(job.id, {
progress: {
// `ready` maps to 'finalizing', NOT 'complete' (#2790).
// The pipeline emits `ready`/100% unconditionally before
// returning — including when it dropped nodes to endpoint
// failures — and the route has not measured the index or
// decided the outcome yet, so 'complete' here would make
// the job record contradict itself (`status: 'analyzing'`,
// `progress.phase: 'complete'`).
phase:
p.phase === 'ready'
? 'finalizing'
: p.phase === 'error'
? 'failed'
: p.phase,
percent: p.percent,
message:
p.phase === 'loading-model'
? 'Loading embedding model...'
: p.phase === 'embedding'
? `Embedding nodes (${p.percent}%)...`
: p.phase === 'indexing'
? 'Creating vector index...'
: p.phase === 'ready'
? 'Finalizing embeddings...'
: `${p.phase} (${p.percent}%)`,
const pipelineResult = await runEmbeddingPipeline(
executeQuery,
executeWithReusedStatement,
(p) => {
embedJobManager.updateJob(job.id, {
progress: {
// `ready` maps to 'finalizing', NOT 'complete' (#2790).
// The pipeline emits `ready`/100% unconditionally before
// returning — including when it dropped nodes to endpoint
// failures — and the route has not measured the index or
// decided the outcome yet, so 'complete' here would make
// the job record contradict itself (`status: 'analyzing'`,
// `progress.phase: 'complete'`).
phase:
p.phase === 'ready'
? 'finalizing'
: p.phase === 'error'
? 'failed'
: p.phase,
percent: p.percent,
message:
p.phase === 'loading-model'
? 'Loading embedding model...'
: p.phase === 'embedding'
? `Embedding nodes (${p.percent}%)...`
: p.phase === 'indexing'
? 'Creating vector index...'
: p.phase === 'ready'
? 'Finalizing embeddings...'
: `${p.phase} (${p.percent}%)`,
},
});
},
{}, // config: use defaults
undefined, // skipNodeIds
existingEmbeddings,
{
signal: embedController.signal,
forceReembedNodeIds,
onCheckpointWindowStart: async ({ nodeIds, ...checkpoint }) => {
await saveEmbeddingCheckpoint(checkpoint, nodeIds);
},
onCheckpoint: async (checkpoint) => {
// Count AFTER the flush, so the number describes rows that
// are durable rather than rows still pending in the WAL.
await flushWAL();
await saveEmbeddingCheckpoint(
checkpoint,
[],
await countPersistedEmbeddings(),
);
},
});
},
{}, // config: use defaults
undefined, // skipNodeIds
existingEmbeddings,
{
signal: embedController.signal,
forceReembedNodeIds,
onCheckpointWindowStart: async ({ nodeIds, ...checkpoint }) => {
await saveEmbeddingCheckpoint(checkpoint, nodeIds);
},
onCheckpoint: async (checkpoint) => {
// Count AFTER the flush, so the number describes rows that
// are durable rather than rows still pending in the WAL.
await flushWAL();
await saveEmbeddingCheckpoint(checkpoint, [], await countPersistedEmbeddings());
},
},
);
);
// Flush WAL so subsequent /api/search requests see the new
// embeddings immediately (#1149). In the CLI path closeLbug()
// handles this during process exit, but the server keeps the
// connection open for other routes — a CHECKPOINT is enough.
await flushWAL();
// Measure inside withLbugDb, after the flush and while the
// connection is still open — this is the route's only chance to
// stamp `stats.embeddings` (embed-run-outcome.ts). A partial run
// gets the same stamp: an honest count of a partial index is what
// makes it survivable.
const measuredEmbeddings = await countPersistedEmbeddings();
// Same re-read-before-write reasoning as saveEmbeddingCheckpoint
// above — and the outcome decision reads it too: its
// `embeddingCheckpoint` is the marker this run's own mid-run
// writer saved, which is the only record of the work when the
// count query could not answer.
const finalMeta = (await loadMeta(entry.storagePath)) ?? embeddingMeta;
const finalizeContext: EmbedRunFinalizeContext = {
measuredEmbeddings: persistedEmbeddingCountOrUndefined(measuredEmbeddings),
onDisk: finalMeta,
// The marker the job STARTED from — `finalMeta`'s has since been
// overwritten by the in-flight writer, so only this one carries
// the `'partial'` attempt chain.
resumedFrom: priorCheckpoint,
};
const outcome = resolveEmbedRunOutcome(
embeddingIdentity,
pipelineResult,
finalizeContext,
);
partialRunError = outcome.error;
partialRunDetail = outcome.partial;
embeddingMeta = withMeasuredEmbeddingCount(
{ ...finalMeta, embeddingCheckpoint: outcome.checkpoint },
measuredEmbeddings,
);
await saveMeta(entry.storagePath, embeddingMeta);
});
// Flush WAL so subsequent /api/search requests see the new
// embeddings immediately (#1149). In the CLI path closeLbug()
// handles this during process exit, but the server keeps the
// connection open for other routes — a CHECKPOINT is enough.
await flushWAL();
// Measure inside withLbugDb, after the flush and while the
// connection is still open — this is the route's only chance to
// stamp `stats.embeddings` (embed-run-outcome.ts). A partial run
// gets the same stamp: an honest count of a partial index is what
// makes it survivable.
const measuredEmbeddings = await countPersistedEmbeddings();
// Same re-read-before-write reasoning as saveEmbeddingCheckpoint
// above — and the outcome decision reads it too: its
// `embeddingCheckpoint` is the marker this run's own mid-run
// writer saved, which is the only record of the work when the
// count query could not answer.
const finalMeta = (await loadMeta(entry.storagePath)) ?? embeddingMeta;
const finalizeContext: EmbedRunFinalizeContext = {
measuredEmbeddings: persistedEmbeddingCountOrUndefined(measuredEmbeddings),
onDisk: finalMeta,
// The marker the job STARTED from — `finalMeta`'s has since been
// overwritten by the in-flight writer, so only this one carries
// the `'partial'` attempt chain.
resumedFrom: priorCheckpoint,
};
const outcome = resolveEmbedRunOutcome(
embeddingIdentity,
pipelineResult,
finalizeContext,
);
partialRunError = outcome.error;
partialRunDetail = outcome.partial;
embeddingMeta = withMeasuredEmbeddingCount(
{ ...finalMeta, embeddingCheckpoint: outcome.checkpoint },
measuredEmbeddings,
);
await saveMeta(entry.storagePath, embeddingMeta);
},
{ ...(ftsSession.skipFts ? { skipFts: true } : {}) },
);
// Don't overwrite 'failed' if the job was cancelled while the pipeline was running
const current = embedJobManager.getJob(job.id);

13
gitnexus/src/storage/repo-meta.ts
View file

@ -164,23 +164,28 @@ export interface RepoMeta {
* `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in
* core/run-analyze.ts — the same discriminator that surface already
* reports to the CLI, persisted rather than re-derived because the two
* causes need OPPOSITE handling on the next run:
* causes need distinct diagnostics and recovery handling:
*
* - `extension-unavailable` — the FTS extension could not load. Healable
* from outside the repo (install it), so the up-to-date fast path
* probes whether it loads now and re-analyzes when it does.
* from outside the repo (install it), then rebuild with --repair-fts.
* - `build-failed` — the extension loaded fine and the index BUILD
* failed (e.g. one un-tokenizable pre-existing row, #2544/#2546).
* Deterministic: the same probe would "heal" it into a full
* re-analysis that degrades identically and restamps, forever. Only
* `--repair-fts` or a content change addresses it.
* - `disabled-by-flag` / `disabled-by-env` — deliberate opt-out.
* A later analyze without the opt-out rebuilds FTS at the same commit.
*
* Collapsing both into `status: 'unavailable'` is exactly what made that
* loop reachable. ABSENT on indexes written before #2841 and on the
* `--repair-fts` stamp (which writes `status: 'available'`); `undefined`
* therefore reads as "cause unknown" and keeps the pre-#2841 behaviour.
*/
skipReason?: 'extension-unavailable' | 'build-failed';
skipReason?:
| 'extension-unavailable'
| 'build-failed'
| 'disabled-by-flag'
| 'disabled-by-env';
};
vectorSearch: {
provider: string;

380
gitnexus/test/integration/skip-fts.test.ts Normal file
View file

@ -0,0 +1,380 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { setupMiniRepo } from '../helpers/mini-repo.js';
import { createTempDir } from '../helpers/test-db.js';
import { runFullAnalysis } from '../../src/core/run-analyze.js';
import * as adapter from '../../src/core/lbug/lbug-adapter.js';
import { extensionManager } from '../../src/core/lbug/extension-loader.js';
import { getStoragePaths, loadMeta, saveMeta } from '../../src/storage/repo-manager.js';
import { batchInsertEmbeddings } from '../../src/core/embeddings/embedding-pipeline.js';
import { EMBEDDING_DIMS } from '../../src/core/lbug/schema.js';
import { searchFTSFromLbug } from '../../src/core/search/bm25-index.js';
import { LocalBackend } from '../../src/mcp/local/local-backend.js';
import { execFile, spawn } from 'node:child_process';
import { promisify } from 'node:util';
import { createServer } from 'node:net';
import { CLI_SPAWN_PREFIX } from '../helpers/cli-entry.js';
import fs from 'node:fs/promises';
import path from 'node:path';
import { createHash } from 'node:crypto';
describe('FTS opt-out analysis lifecycle (#3091)', () => {
let repo: Awaited<ReturnType<typeof setupMiniRepo>>;
let home: Awaited<ReturnType<typeof createTempDir>>;
const options = { skipAgentsMd: true, registryName: 'skip-fts-fixture' };
const callbacks = { onProgress() {} };
beforeEach(async () => {
repo = await setupMiniRepo();
home = await createTempDir();
vi.stubEnv('GITNEXUS_HOME', home.dbPath);
vi.stubEnv('GITNEXUS_SKIP_FTS', undefined);
});
afterEach(async () => {
await adapter.closeLbug();
vi.restoreAllMocks();
vi.unstubAllEnvs();
await repo.cleanup();
await home.cleanup();
});
async function graphSnapshot() {
const { lbugPath, storagePath } = getStoragePaths(repo.dbPath);
const meta = await loadMeta(storagePath);
const cacheKeys = [...(meta?.cacheKeys ?? [])].sort();
expect(cacheKeys).not.toEqual([]);
const cache = [];
for (const key of cacheKeys) {
const bytes = await fs.readFile(path.join(storagePath, 'parse-cache', `${key}.v8`));
cache.push([key, createHash('sha256').update(bytes).digest('hex')]);
}
await adapter.initLbug(lbugPath, { skipFts: true });
try {
return {
functions: await adapter.executeQuery(
'MATCH (n:Function) RETURN n.id AS id, n.name AS name ORDER BY n.id',
),
edges: await adapter.executeQuery(
'MATCH (n)-[r:CodeRelation]->(m) RETURN n.id AS source, m.id AS target, r.type AS type ORDER BY source, target, type',
),
communities: await adapter.executeQuery(
'MATCH (n:Community) RETURN n.id AS id, n.label AS label, n.heuristicLabel AS heuristicLabel, n.keywords AS keywords, n.cohesion AS cohesion, n.symbolCount AS symbolCount ORDER BY id',
),
processes: await adapter.executeQuery(
'MATCH (n:Process) RETURN n.id AS id, n.label AS label, n.heuristicLabel AS heuristicLabel, n.processType AS processType, n.stepCount AS stepCount, n.communities AS communities, n.entryPointId AS entryPointId, n.terminalId AS terminalId ORDER BY id',
),
cache,
embeddings: await adapter.loadCachedEmbeddings(),
indexes: await adapter.executeQuery('CALL SHOW_INDEXES() RETURN *'),
};
} finally {
await adapter.closeLbug();
}
}
it('indexes without any FTS load and reports intentional disablement through MCP', async () => {
const ensure = vi.spyOn(extensionManager, 'ensure');
const result = await runFullAnalysis(repo.dbPath, { ...options, skipFts: true }, callbacks);
expect(result.ftsSkipped).toBe(true);
expect(result.ftsSkipReason).toBe('disabled-by-flag');
expect(ensure.mock.calls.filter((call) => call[1] === 'fts')).toEqual([]);
const meta = await loadMeta(getStoragePaths(repo.dbPath).storagePath);
expect(meta?.capabilities?.fts).toEqual({
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-flag',
});
expect(meta?.capabilities?.graph.status).toBe('available');
const snapshot = await graphSnapshot();
expect(snapshot.functions).toContainEqual(expect.objectContaining({ name: 'createHandler' }));
expect(snapshot.edges).toContainEqual(expect.objectContaining({ type: 'CALLS' }));
expect(snapshot.indexes.filter((row) => row.index_type === 'FTS')).toEqual([]);
const repeat = await runFullAnalysis(repo.dbPath, { ...options, skipFts: true }, callbacks);
expect(repeat.alreadyUpToDate).toBe(true);
expect(repeat.ftsSkipReason).toBe('disabled-by-flag');
expect(ensure.mock.calls.filter((call) => call[1] === 'fts')).toEqual([]);
vi.stubEnv('GITNEXUS_SKIP_FTS', '1');
const switched = await runFullAnalysis(repo.dbPath, options, callbacks);
expect(switched.alreadyUpToDate).toBe(true);
expect(switched.ftsSkipReason).toBe('disabled-by-env');
expect(ensure.mock.calls.filter((call) => call[1] === 'fts')).toEqual([]);
expect((await loadMeta(getStoragePaths(repo.dbPath).storagePath))?.capabilities?.fts).toEqual({
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-env',
});
vi.stubEnv('GITNEXUS_SKIP_FTS', undefined);
const backend = new LocalBackend();
try {
expect(await backend.init()).toBe(true);
const query = await backend.callTool('query', {
repo: options.registryName,
search_query: 'createHandler',
});
expect(query.error).toBeUndefined();
expect(query.warning).toContain('FTS disabled for this index');
expect(query.warning).not.toMatch(/failed to load|indexes missing|reinstall|repair-fts/);
} finally {
await backend.dispose();
}
}, 180_000);
it('re-enables at the same commit and safely disables existing native indexes while preserving graph and embeddings', async () => {
vi.stubEnv('GITNEXUS_SKIP_FTS', '1');
const disabled = await runFullAnalysis(repo.dbPath, options, callbacks);
expect(disabled.ftsSkipReason).toBe('disabled-by-env');
const { storagePath, lbugPath } = getStoragePaths(repo.dbPath);
const before = await graphSnapshot();
expect(before.communities).not.toEqual([]);
expect(before.processes).not.toEqual([]);
const node = before.functions.find((row) => row.name === 'createHandler');
expect(node).toBeDefined();
await adapter.initLbug(lbugPath, { skipFts: true });
const embedding = {
nodeId: String(node!.id),
chunkIndex: 0,
startLine: 13,
endLine: 15,
embedding: Array.from({ length: EMBEDDING_DIMS }, (_, i) => (i === 0 ? 1 : 0)),
contentHash: 'seed-preserved',
};
await batchInsertEmbeddings(adapter.executeWithReusedStatement, [embedding]);
await adapter.flushWAL();
await adapter.closeLbug();
const meta = (await loadMeta(storagePath))!;
await saveMeta(storagePath, {
...meta,
stats: { ...meta.stats, embeddings: 1 },
embeddingDims: EMBEDDING_DIMS,
});
vi.stubEnv('GITNEXUS_SKIP_FTS', undefined);
const enabled = await runFullAnalysis(repo.dbPath, options, callbacks);
expect(enabled.alreadyUpToDate).not.toBe(true);
expect(enabled.ftsSkipped).toBe(false);
expect((await loadMeta(storagePath))?.lastCommit).toBe(meta.lastCommit);
await adapter.initLbug(lbugPath);
const found = await searchFTSFromLbug('createHandler');
expect(found.ftsAvailable).toBe(true);
expect(found.results.some((r) => r.filePath === 'src/handler.ts')).toBe(true);
await adapter.closeLbug();
const afterEnable = await graphSnapshot();
expect(afterEnable.functions).toEqual(before.functions);
expect(afterEnable.edges).toEqual(before.edges);
expect(afterEnable.communities).toEqual(before.communities);
expect(afterEnable.processes).toEqual(before.processes);
expect(afterEnable.cache).toEqual(before.cache);
expect(afterEnable.embeddings.embeddings).toEqual([embedding]);
const ensure = vi.spyOn(extensionManager, 'ensure');
const enabledMeta = (await loadMeta(storagePath))!;
let dirtyDuringDisable: Promise<Awaited<ReturnType<typeof loadMeta>>> | undefined;
const disabledAgain = await runFullAnalysis(
repo.dbPath,
{ ...options, skipFts: true },
{
onProgress() {},
onLog(msg) {
// vi.spyOn cannot intercept run-analyze's ESM saveMeta binding.
// The escalation log is emitted after the dirty stamp and before wipe.
if (
!dirtyDuringDisable &&
msg.includes('FTS is explicitly disabled and existing search indexes')
) {
dirtyDuringDisable = loadMeta(storagePath);
}
},
},
);
expect(disabledAgain.ftsSkipReason).toBe('disabled-by-flag');
expect(dirtyDuringDisable).toBeDefined();
const dirty = await dirtyDuringDisable!;
expect(dirty?.incrementalInProgress).toBeDefined();
expect(dirty?.indexedAt).toBe(enabledMeta.indexedAt);
expect(dirty?.capabilities?.fts).toEqual({
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-flag',
});
expect(ensure.mock.calls.filter((call) => call[1] === 'fts')).toEqual([]);
const after = await graphSnapshot();
expect(after.functions).toEqual(before.functions);
expect(after.edges).toEqual(before.edges);
expect(after.communities).toEqual(before.communities);
expect(after.processes).toEqual(before.processes);
expect(after.cache).toEqual(before.cache);
expect(after.embeddings.embeddings).toEqual([embedding]);
expect(after.indexes.filter((row) => row.index_type === 'FTS')).toEqual([]);
expect((await loadMeta(storagePath))?.stats?.embeddings).toBe(1);
}, 240_000);
it('refreshes an existing MCP session across both FTS mode transitions without losing context', async () => {
await runFullAnalysis(repo.dbPath, { ...options, skipFts: true }, callbacks);
const backend = new LocalBackend();
const uid = 'Function:src/handler.ts:createHandler';
const queryArgs = { repo: options.registryName, search_query: 'createHandler' };
const contextArgs = { repo: options.registryName, uid };
try {
expect(await backend.init()).toBe(true);
expect((await backend.callTool('query', queryArgs)).warning).toContain(
'FTS disabled for this index',
);
const initialContext = await backend.callTool('context', contextArgs);
expect(initialContext).toMatchObject({ status: 'found', symbol: { uid } });
await runFullAnalysis(repo.dbPath, options, callbacks);
// Observe the production staleness throttle without resetting the backend.
await vi.waitFor(
async () => {
const result = await backend.callTool('query', queryArgs);
expect(result.error).toBeUndefined();
expect(result.warning).toBeUndefined();
const hits = [...(result.definitions ?? []), ...(result.process_symbols ?? [])];
expect(hits.map((hit) => hit.id)).toContain(uid);
},
{ timeout: 15_000, interval: 300 },
);
await runFullAnalysis(repo.dbPath, { ...options, skipFts: true }, callbacks);
await vi.waitFor(
async () => {
const result = await backend.callTool('query', queryArgs);
expect(result.error).toBeUndefined();
expect(result.warning).toContain('FTS disabled for this index');
},
{ timeout: 15_000, interval: 300 },
);
const finalContext = await backend.callTool('context', contextArgs);
expect(finalContext.status).toBe('found');
expect(finalContext.symbol).toEqual(initialContext.symbol);
} finally {
await backend.dispose();
}
}, 240_000);
it('fails closed for unknown catalog state without loading FTS', async () => {
const { lbugPath } = getStoragePaths(repo.dbPath);
const ensure = vi.spyOn(extensionManager, 'ensure');
await adapter.initLbug(lbugPath, { skipFts: true });
expect(await adapter.ensureFtsRowDmlSafe([], { skipFts: true })).toBe(true);
expect(
await adapter.ensureFtsRowDmlSafe(adapter.INDEX_CATALOG_UNREADABLE, { skipFts: true }),
).toBe(false);
expect(await adapter.ensureFtsRowDmlSafe([{ index_type: 'FTS' }], { skipFts: true })).toBe(
false,
);
expect(ensure.mock.calls.filter((call) => call[1] === 'fts')).toEqual([]);
});
it('exposes the explicit CLI flag, status capability and HTTP disabled response', async () => {
const cli = (args: string[]) =>
promisify(execFile)(process.execPath, [...CLI_SPAWN_PREFIX, ...args], {
cwd: repo.dbPath,
env: { ...process.env, GITNEXUS_MEMORY: 'off' },
timeout: 120_000,
maxBuffer: 2_000_000,
});
const analyze = await cli([
'analyze',
'--skip-fts',
'--skip-agents-md',
'--name',
options.registryName,
]);
expect(analyze.stdout + analyze.stderr).toContain('FTS disabled for this index');
const status = await cli(['status', '--json']);
const statusJson = JSON.parse(
status.stdout.split(/\r?\n/).find((line) => line.startsWith('{'))!,
);
expect(statusJson.index.capabilities.fts).toMatchObject({
status: 'unavailable',
skipReason: 'disabled-by-flag',
});
expect((await cli(['status'])).stdout).toContain('FTS disabled for this index');
// Spawned `serve` socket readiness is not reliable on Windows — the child can
// report ready before its listen socket is reachable from the parent, which is
// why both sibling spawned-server suites skip there (server-http-startup.test.ts,
// server-analyze-token-validation.test.ts). The CLI flag and status-capability
// assertions above already ran on every platform.
if (process.platform === 'win32') return;
const pickPort = () =>
new Promise<number>((resolve, reject) => {
const probe = createServer();
probe.once('error', reject);
probe.listen(0, '127.0.0.1', () => {
const address = probe.address();
if (!address || typeof address === 'string') {
probe.close();
reject(new Error('No port'));
return;
}
probe.close((error) => (error ? reject(error) : resolve(address.port)));
});
});
// The probe must release its ephemeral port before the child can bind it, so
// another process can win that gap and the child dies with EADDRINUSE. Re-pick
// and respawn a bounded number of times rather than failing on a lost race.
const startServe = async () => {
for (let attempt = 1; attempt <= 3; attempt++) {
const port = await pickPort();
let output = '';
const child = spawn(
process.execPath,
[...CLI_SPAWN_PREFIX, 'serve', '--port', String(port), '--host', '127.0.0.1'],
{
cwd: repo.dbPath,
env: { ...process.env, GITNEXUS_MEMORY: 'off', GITNEXUS_NO_UPDATE_NOTIFIER: '1' },
stdio: ['ignore', 'pipe', 'pipe'],
},
);
child.stdout.on('data', (data) => {
output += String(data);
});
child.stderr.on('data', (data) => {
output += String(data);
});
const childExited = new Promise<void>((resolve) => child.once('exit', () => resolve()));
const healthy = vi.waitFor(
async () => {
const health = await fetch(`http://127.0.0.1:${port}/api/health`, {
signal: AbortSignal.timeout(2_000),
});
expect(health.status).toBe(200);
},
{ timeout: 60_000, interval: 250 },
);
// Race readiness against the child dying so a lost port does not burn the
// whole health-wait timeout before retrying.
const outcome = await Promise.race([
healthy.then(() => 'ready' as const),
childExited.then(() => 'exited' as const),
]);
if (outcome === 'ready') return { child, port, exited: childExited };
healthy.catch(() => {});
if (attempt === 3 || !/EADDRINUSE/i.test(output)) {
throw new Error(`Server exited: ${output}`);
}
}
throw new Error('unreachable');
};
const { child: server, port, exited } = await startServe();
try {
for (const mode of ['bm25', 'hybrid']) {
const response = await fetch(`http://127.0.0.1:${port}/api/search`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: 'createHandler', mode, repo: options.registryName }),
signal: AbortSignal.timeout(30_000),
});
expect(response.status).toBe(200);
const body = await response.json();
expect(body.warning).toContain('FTS disabled for this index');
expect(body.warning).not.toMatch(/failed to load|indexes missing|repair-fts/);
}
} finally {
if (server.exitCode === null) server.kill('SIGTERM');
const force = setTimeout(() => server.kill('SIGKILL'), 3_000);
await exited;
clearTimeout(force);
}
}, 240_000);
});

14
gitnexus/test/unit/analyze-api.test.ts
View file

@ -605,9 +605,13 @@ describe('POST /api/embed route wiring (#2790)', () => {
* a character-distance regex so a comment edit cannot silently un-assert it.
*/
const insideWithLbugDb = (source: string): string => {
const start = source.indexOf('await withLbugDb(lbugPath, async () => {');
const end = source.indexOf('\n });', start);
expect(start).toBeGreaterThan(-1);
// The open is a multi-line `withLbugDb(lbugPath, async () => {…}, opts)` call
// since #3091 added the FTS-mode options argument, so anchor on the call head
// and close on that options argument rather than a fixed-indent literal.
const head = source.match(/await withLbugDb\(\s*lbugPath,\s*async \(\) => \{/);
expect(head).not.toBeNull();
const start = head!.index!;
const end = source.indexOf('{ ...(ftsSession.skipFts', start);
expect(end).toBeGreaterThan(start);
return source.slice(start, end);
};
@ -653,8 +657,8 @@ describe('POST /api/embed route wiring (#2790)', () => {
it('measures in the post-flush checkpoint callback and nowhere else in the pipeline options', async () => {
const source = await readSource();
expect(source).toContain(
'await saveEmbeddingCheckpoint(checkpoint, [], await countPersistedEmbeddings());',
expect(source).toMatch(
/await saveEmbeddingCheckpoint\(\s*checkpoint,\s*\[\],\s*await countPersistedEmbeddings\(\),?\s*\)/,
);
// The window-start callback fires before any row exists — it must pass no
// count rather than restate a stale one.

242
gitnexus/test/unit/api-fts-mode.test.ts Normal file
View file

@ -0,0 +1,242 @@
import express from 'express';
import { EventEmitter } from 'node:events';
import path from 'node:path';
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
const mocks = vi.hoisted(() => ({
loadMeta: vi.fn(),
listRegisteredRepos: vi.fn(),
withLbugDb: vi.fn(),
search: vi.fn(),
updateJob: vi.fn(),
}));
vi.mock('../../src/storage/repo-manager.js', async (importOriginal) => ({
...(await importOriginal<typeof import('../../src/storage/repo-manager.js')>()),
loadMeta: mocks.loadMeta,
listRegisteredRepos: mocks.listRegisteredRepos,
}));
vi.mock('../../src/core/lbug/lbug-adapter.js', () => ({
withLbugDb: mocks.withLbugDb,
executeQuery: vi.fn(async () => []),
executePrepared: vi.fn(async () => [{ value: 1 }]),
executeWithReusedStatement: vi.fn(async () => []),
streamQuery: vi.fn(async () => 0),
flushWAL: vi.fn(),
closeLbug: vi.fn(),
isReadOnlyDbError: vi.fn(() => false),
}));
vi.mock('../../src/core/search/bm25-index.js', () => ({ searchFTSFromLbug: mocks.search }));
vi.mock('../../src/mcp/local/local-backend.js', () => ({
LocalBackend: class {
async init() {
return true;
}
},
}));
vi.mock('../../src/server/mcp-http.js', () => ({
installServeMcpAuth: vi.fn(),
mountMCPEndpoints: vi.fn(async () => vi.fn()),
}));
vi.mock('../../src/server/upload-sweep.js', () => ({ sweepStaleUploads: vi.fn(async () => {}) }));
vi.mock('../../src/server/update-controller.js', () => ({
createServeUpdateController: vi.fn(() => ({ stop: vi.fn() })),
bindServeUpdateControllerLifecycle: vi.fn(),
buildServerInfo: vi.fn(),
}));
vi.mock('../../src/server/grep-scan.js', () => ({
runGrepScanInWorker: vi.fn(async () => ({ results: [], timedOut: false })),
}));
vi.mock('../../src/server/sse-progress.js', () => ({ mountSSEProgress: vi.fn() }));
vi.mock('../../src/server/analyze-job.js', () => ({
isTerminalJobStatus: vi.fn(() => true),
JobManager: class {
createJob() {
return { id: 'embed-job', status: 'queued' };
}
updateJob = mocks.updateJob;
registerAbortController() {}
getJob() {
return { status: 'complete' };
}
listJobs() {
return [];
}
},
}));
import { createServer } from '../../src/server/api.js';
import { FTS_DISABLED_MESSAGE } from '../../src/core/search/fts-policy.js';
const entry = {
name: 'fts-mode-fixture',
path: path.resolve('fts-mode-fixture'),
storagePath: path.resolve('fts-mode-fixture/.gitnexus'),
};
let app: express.Express;
const events = ['SIGINT', 'SIGTERM', 'uncaughtException', 'unhandledRejection'] as const;
const originalListeners = new Map(events.map((event) => [event, process.listeners(event)]));
beforeAll(async () => {
// Capture the real registered handlers without binding a socket or starting MCP/native work.
const listen = vi.spyOn(express.application, 'listen').mockImplementation(function (
this: express.Express,
...args: any[]
) {
app = this;
queueMicrotask(args.at(-1));
return new EventEmitter() as any;
});
try {
await createServer(0);
} finally {
listen.mockRestore();
}
});
afterAll(() => {
for (const event of events) {
for (const listener of process.listeners(event)) {
if (!originalListeners.get(event)!.includes(listener))
process.removeListener(event, listener);
}
}
vi.unstubAllEnvs();
});
beforeEach(() => {
vi.clearAllMocks();
mocks.listRegisteredRepos.mockResolvedValue([entry]);
mocks.withLbugDb.mockImplementation(async (_path, callback) => callback());
mocks.search.mockImplementation(async (_query, _limit, _exec, reason) => ({
results: [],
ftsAvailable: !reason,
}));
});
async function invoke(route: string, query: Record<string, unknown> = {}) {
const layer = app.router.stack.find((item: any) => item.route?.path === route);
expect(layer, route).toBeDefined();
const handler = layer.route.stack.at(-1).handle;
const req = Object.assign(new EventEmitter(), {
query,
body: { cypher: 'RETURN 1 AS value', query: 'handler', mode: 'bm25', enrich: false },
});
const res = Object.assign(new EventEmitter(), {
statusCode: 200,
body: undefined as any,
writableEnded: false,
destroyed: false,
status(code: number) {
this.statusCode = code;
return this;
},
json(body: unknown) {
this.body = body;
return this;
},
set() {
return this;
},
setHeader() {
return this;
},
flushHeaders() {},
write() {
return true;
},
end() {
this.writableEnded = true;
this.emit('finish');
},
});
await handler(req, res);
expect(res.statusCode, JSON.stringify(res.body)).toBe(route === '/api/embed' ? 202 : 200);
return res;
}
const cases = [
{
name: 'flag-disabled',
fts: { provider: 'ladybugdb-fts', status: 'unavailable', skipReason: 'disabled-by-flag' },
skip: true,
},
{
name: 'env-disabled',
fts: { provider: 'ladybugdb-fts', status: 'unavailable', skipReason: 'disabled-by-env' },
skip: true,
},
{ name: 'normal', fts: { provider: 'ladybugdb-fts', status: 'available' }, skip: false },
{ name: 'legacy', fts: undefined, skip: false },
{
name: 'degraded',
fts: { provider: 'ladybugdb-fts', status: 'degraded', skipReason: 'build-failed' },
skip: false,
},
] as const;
describe('serve uses one metadata-derived FTS mode on every DB-open path', () => {
it.each(cases)(
'keeps mixed read requests consistent for $name indexes',
async ({ fts, skip }) => {
// The server process environment must not override persisted per-index intent.
vi.stubEnv('GITNEXUS_SKIP_FTS', skip ? undefined : '1');
mocks.loadMeta.mockResolvedValue({ capabilities: { fts } });
const sequence = [
['/api/search', {}],
['/api/query', {}],
['/api/graph', {}],
['/api/search', {}],
['/api/graph', { stream: 'true' }],
['/api/grep', { pattern: 'handler' }],
['/api/query', {}],
['/api/search', {}],
] as const;
for (const [route, query] of sequence) {
const response = await invoke(route, query);
if (route === '/api/search') {
expect(response.body.warning).toBe(skip ? FTS_DISABLED_MESSAGE : undefined);
}
}
expect(mocks.withLbugDb).toHaveBeenCalledTimes(sequence.length);
expect(mocks.loadMeta).toHaveBeenCalledTimes(sequence.length);
for (const [dbPath, , options] of mocks.withLbugDb.mock.calls) {
expect(dbPath).toBe(path.join(entry.storagePath, 'lbug'));
expect(options).toEqual({ readOnly: true, ...(skip ? { skipFts: true } : {}) });
}
},
);
it('reads mode changes between requests instead of caching stale metadata', async () => {
for (const mode of [cases[0], cases[2], cases[1]]) {
mocks.loadMeta.mockResolvedValue({ capabilities: { fts: mode.fts } });
await invoke('/api/query');
expect(mocks.withLbugDb.mock.lastCall?.[2]).toEqual({
readOnly: true,
...(mode.skip ? { skipFts: true } : {}),
});
}
});
it.each(cases)(
'preserves write mode while honoring $name metadata for embed',
async ({ fts, skip }) => {
mocks.loadMeta.mockResolvedValue({ capabilities: { fts } });
// This test stops at the DB boundary; it must not generate vectors or write an index.
mocks.withLbugDb.mockResolvedValue(undefined);
await invoke('/api/embed');
await vi.waitFor(() =>
expect(mocks.updateJob).toHaveBeenCalledWith(
'embed-job',
expect.objectContaining({ status: 'complete' }),
),
);
expect(mocks.withLbugDb).toHaveBeenCalledExactlyOnceWith(
path.join(entry.storagePath, 'lbug'),
expect.any(Function),
skip ? { skipFts: true } : {},
);
expect(mocks.loadMeta).toHaveBeenCalledExactlyOnceWith(entry.storagePath);
},
);
});

7
gitnexus/test/unit/api-query-readonly-wiring.test.ts
View file

@ -8,7 +8,12 @@ describe('api query read-only wiring', () => {
path.join(__dirname, '..', '..', 'src', 'server', 'api.ts'),
'utf-8',
);
expect(source).toMatch(/handleQueryRequest[\s\S]*withLbugDb\([\s\S]*readOnly:\s*true/);
// The open now routes through readOnlyFtsOptions(skipFts) so the persisted
// FTS mode rides along; api-readonly-wiring.test.ts pins that the helper
// always sets `readOnly: true`.
expect(source).toMatch(
/handleQueryRequest[\s\S]*withLbugDb\([\s\S]*(?:readOnly:\s*true|readOnlyFtsOptions\()/,
);
});
it('routes /api/query through handleQueryRequest', async () => {

28
gitnexus/test/unit/api-readonly-wiring.test.ts
View file

@ -24,13 +24,15 @@ describe('api read-only endpoint wiring', () => {
it('/api/graph stream path opens read-only', async () => {
const source = await readSource();
expect(source).toMatch(
/streamGraphNdjson\(res, includeContent, abortController\.signal\)[\s\S]{0,200}readOnly:\s*true/,
/streamGraphNdjson\(res, includeContent, abortController\.signal\)[\s\S]{0,200}(?:readOnly:\s*true|readOnlyFtsOptions\()/,
);
});
it('/api/graph non-stream path opens read-only', async () => {
const source = await readSource();
expect(source).toMatch(/buildGraph\(includeContent\)[\s\S]{0,80}readOnly:\s*true/);
expect(source).toMatch(
/buildGraph\(includeContent\)[\s\S]{0,80}(?:readOnly:\s*true|readOnlyFtsOptions\()/,
);
});
it('/api/search opens read-only', async () => {
@ -39,12 +41,30 @@ describe('api read-only endpoint wiring', () => {
// `return { searchResults: enriched, ftsAvailable };` immediately before
// the closing brace + options object. Match that suffix to confirm the
// search call site, not /api/query.
expect(source).toMatch(/searchResults: enriched, ftsAvailable[\s\S]{0,80}readOnly:\s*true/);
expect(source).toMatch(
/searchResults: enriched, ftsAvailable[\s\S]{0,80}(?:readOnly:\s*true|readOnlyFtsOptions\()/,
);
});
it('/api/grep opens read-only', async () => {
const source = await readSource();
expect(source).toMatch(/MATCH \(n:File\)[\s\S]{0,300}readOnly:\s*true/);
expect(source).toMatch(
/MATCH \(n:File\)[\s\S]{0,300}(?:readOnly:\s*true|readOnlyFtsOptions\()/,
);
});
it('readOnlyFtsOptions carries readOnly: true on both branches', async () => {
const source = await readSource();
// The read-only routes above open through this helper rather than an inline
// `{ readOnly: true }` (#3091 threads the persisted FTS mode through the same
// option object). That indirection is why those assertions accept the helper
// by name — so the read-only half of the contract is pinned here instead, or
// a skip-fts index could open write-mode and re-trip the `.shadow` failure.
const helper = source.match(/function readOnlyFtsOptions\([\s\S]*?\n\}/);
expect(helper).not.toBeNull();
expect(helper![0]).toMatch(
/skipFts\s*\?\s*\{ readOnly: true, skipFts: true \}\s*:\s*\{ readOnly: true \}/,
);
});
it('/api/embed remains write-mode (writes embeddings — must not be flipped to readOnly)', async () => {

152
gitnexus/test/unit/fts-policy.test.ts Normal file
View file

@ -0,0 +1,152 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import {
FTS_DISABLED_MESSAGE,
getFtsDisabledReason,
isExplicitFtsDisablement,
resolveFtsDisableReason,
withExplicitFtsDisablement,
} from '../../src/core/search/fts-policy.js';
import type { RepoMeta } from '../../src/storage/repo-meta.js';
import { ftsDegradedWarning } from '../../src/core/search/fts-indexes.js';
import { searchFTSFromLbug } from '../../src/core/search/bm25-index.js';
import { hybridSearch } from '../../src/core/search/hybrid-search.js';
import { extensionManager, resetExtensionState } from '../../src/core/lbug/extension-loader.js';
afterEach(() => {
vi.unstubAllEnvs();
resetExtensionState();
});
describe('explicit FTS opt-out', () => {
it('is off by default and accepts only the exact environment value 1', () => {
vi.stubEnv('GITNEXUS_SKIP_FTS', undefined);
expect(resolveFtsDisableReason()).toBeUndefined();
for (const value of ['', '0', 'true', 'yes', ' 1', '1 ']) {
expect(resolveFtsDisableReason(false, value)).toBeUndefined();
}
expect(resolveFtsDisableReason(false, '1')).toBe('disabled-by-env');
expect(resolveFtsDisableReason(true, '1')).toBe('disabled-by-flag');
expect(resolveFtsDisableReason(true, '0')).toBe('disabled-by-flag');
expect(isExplicitFtsDisablement('disabled-by-flag')).toBe(true);
expect(isExplicitFtsDisablement('disabled-by-env')).toBe(true);
expect(isExplicitFtsDisablement('build-failed')).toBe(false);
});
it('does not infer intent from a failed or legacy index', () => {
expect(getFtsDisabledReason(undefined)).toBeUndefined();
for (const skipReason of [undefined, 'build-failed', 'extension-unavailable'] as const) {
expect(
getFtsDisabledReason({ provider: 'ladybugdb-fts', status: 'unavailable', skipReason }),
).toBeUndefined();
}
expect(
getFtsDisabledReason({
provider: 'ladybugdb-fts',
status: 'available',
skipReason: 'disabled-by-flag',
}),
).toBeUndefined();
expect(
getFtsDisabledReason({
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-env',
}),
).toBe('disabled-by-env');
});
it('stamps explicit disablement without rewriting freshness or sibling capabilities', () => {
const indexedAt = '2026-01-01T00:00:00.000Z';
const meta = {
indexedAt,
lastCommit: 'abc',
capabilities: {
graph: { provider: 'ladybugdb', status: 'available' },
fts: { provider: 'ladybugdb-fts', status: 'available' },
vectorSearch: {
provider: 'ladybugdb-vector',
status: 'vector-index',
exactScanLimit: 10,
},
},
} as RepoMeta;
const stamped = withExplicitFtsDisablement(meta, 'disabled-by-flag');
expect(stamped.indexedAt).toBe(indexedAt);
expect(stamped.lastCommit).toBe('abc');
expect(stamped.capabilities?.graph).toEqual(meta.capabilities?.graph);
expect(stamped.capabilities?.vectorSearch).toEqual(meta.capabilities?.vectorSearch);
expect(stamped.capabilities?.fts).toEqual({
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-flag',
});
expect(withExplicitFtsDisablement(meta, undefined)).toBe(meta);
expect(withExplicitFtsDisablement(stamped, 'disabled-by-flag')).toBe(stamped);
expect(
withExplicitFtsDisablement(stamped, 'disabled-by-env').capabilities?.fts?.skipReason,
).toBe('disabled-by-env');
});
it('reports intent even when another database has an extension failure', async () => {
await extensionManager.ensure(
vi.fn().mockRejectedValue(new Error('invalid ELF header')),
'fts',
'FTS',
{ policy: 'load-only' },
);
expect(ftsDegradedWarning(undefined, 'disabled-by-flag')).toBe(FTS_DISABLED_MESSAGE);
expect(ftsDegradedWarning()).toContain('FTS extension failed to load');
});
it('returns no keyword results without a database or extension load', async () => {
await expect(
searchFTSFromLbug('createHandler', 10, undefined, 'disabled-by-env'),
).resolves.toEqual({ results: [], ftsAvailable: false });
});
it.each([
{ skipFts: true, env: undefined },
{ skipFts: false, env: '1' },
{ skipFts: true, env: '1' },
])('rejects FTS repair while explicitly disabled (%j)', async ({ skipFts, env }) => {
const home = await fs.mkdtemp(path.join(os.tmpdir(), 'gitnexus-fts-policy-'));
vi.stubEnv('GITNEXUS_HOME', home);
vi.stubEnv('GITNEXUS_SKIP_FTS', env);
try {
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
await expect(
runFullAnalysis(path.join(home, 'repo'), { skipFts, repairFts: true }, { onProgress() {} }),
).rejects.toThrow('--repair-fts cannot be used with --skip-fts or GITNEXUS_SKIP_FTS=1');
expect(await fs.readdir(home)).toEqual([]);
} finally {
await fs.rm(home, { recursive: true, force: true });
}
});
it('keeps semantic results when keyword search is explicitly disabled', async () => {
const executeQuery = vi.fn();
const semantic = vi.fn().mockResolvedValue([
{
nodeId: 'Function:handler',
filePath: 'src/handler.ts',
name: 'handler',
label: 'Function',
startLine: 1,
endLine: 3,
distance: 0.1,
},
]);
const result = await hybridSearch('handler', 10, executeQuery, semantic, 'disabled-by-flag');
expect(result).toHaveLength(1);
expect(result[0]).toMatchObject({
name: 'handler',
sources: ['semantic'],
filePath: 'src/handler.ts',
});
expect(semantic).toHaveBeenCalledWith(executeQuery, 'handler', 10);
expect(executeQuery).not.toHaveBeenCalled();
});
});

4
gitnexus/test/unit/grep-params.test.ts
View file

@ -155,6 +155,8 @@ describe('/api/grep handler wiring (source-level, api-readonly-wiring.test.ts st
expect(section).toContain('runGrepScanInWorker(');
expect(section).toMatch(/filePath\.toLowerCase\(\)\.includes\(fileFilter\)/);
expect(section).toContain('timedOut: true');
expect(section).toContain('readOnly: true'); // unchanged read-only DB open
// Unchanged read-only DB open — now expressed via readOnlyFtsOptions(skipFts),
// whose read-only guarantee is pinned in api-readonly-wiring.test.ts.
expect(section).toMatch(/readOnly: true|readOnlyFtsOptions\(/);
});
});

2
gitnexus/test/unit/run-analyze-fts-repair.test.ts
View file

@ -827,6 +827,8 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => {
});
it('ABORTS (throws before publish, leaves the previous index intact) on an FTS integrity failure on the atomic-swap path (#2658 review M1)', async () => {
// Select the atomic path on Windows too; the native adapter is mocked below.
vi.stubEnv('GITNEXUS_ATOMIC_WINDOWS_SWAP', '1');
// The single-writer lock rules out a concurrent-writer race, so an
// integrity-class FTS failure on the atomic-swap (--force) path is a real
// broken build: run-analyze must throw BEFORE swapping the staging DB in,

59
gitnexus/test/unit/run-analyze.test.ts
View file

@ -92,6 +92,65 @@ describe('run-analyze module', () => {
}
});
it('restamps FTS skipReason on the already-up-to-date path when only the discriminator changes', async () => {
const tmpRepo = await createTempDir('gitnexus-run-analyze-fts-restamp-');
try {
execSync('git init', { cwd: tmpRepo.dbPath, stdio: 'pipe' });
execSync('git -c user.name=test -c user.email=test@test commit --allow-empty -m init', {
cwd: tmpRepo.dbPath,
stdio: 'pipe',
});
const currentCommit = execSync('git rev-parse HEAD', {
cwd: tmpRepo.dbPath,
encoding: 'utf-8',
}).trim();
const indexedAt = '2026-01-01T00:00:00.000Z';
const { storagePath } = getStoragePaths(tmpRepo.dbPath);
const meta: RepoMeta = {
repoPath: tmpRepo.dbPath,
lastCommit: currentCommit,
indexedAt,
schemaFingerprint: SCHEMA_FINGERPRINT,
analysisFeatures: CURRENT_ANALYSIS_FEATURES,
runnerIdentity: currentRunnerIdentity(),
capabilities: {
graph: { provider: 'ladybugdb', status: 'available' },
fts: {
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-env',
},
vectorSearch: { provider: 'exact-scan', status: 'unavailable', exactScanLimit: 0 },
},
};
await saveMeta(storagePath, meta);
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const result = await runFullAnalysis(
tmpRepo.dbPath,
{ skipFts: true },
{ onProgress: () => {} },
);
expect(result.alreadyUpToDate).toBe(true);
expect(result.ftsSkipped).toBe(true);
expect(result.ftsSkipReason).toBe('disabled-by-flag');
const restamped = await loadMeta(storagePath);
expect(restamped?.indexedAt).toBe(indexedAt);
expect(restamped?.lastCommit).toBe(currentCommit);
expect(restamped?.incrementalInProgress).toBeUndefined();
expect(restamped?.capabilities?.fts).toEqual({
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'disabled-by-flag',
});
expect(restamped?.capabilities?.graph).toEqual(meta.capabilities?.graph);
expect(restamped?.capabilities?.vectorSearch).toEqual(meta.capabilities?.vectorSearch);
} finally {
await tmpRepo.cleanup();
}
});
it('applies analyze --name on the already-up-to-date path without --force', async () => {
const tmpRepo = await createTempDir('gitnexus-run-analyze-fast-name-');
const tmpHome = await createTempDir('gitnexus-run-analyze-fast-name-home-');

15
gitnexus/test/unit/watch-paths.test.ts
View file

@ -24,6 +24,21 @@ afterEach(async () => {
});
describe('watch path selection', () => {
it('forwards explicit FTS opt-out without changing the default', async () => {
const names = [
'GITNEXUS_MAX_FILE_SIZE',
'GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS',
'GITNEXUS_VERBOSE',
] as const;
for (const name of names) vi.stubEnv(name, process.env[name]);
const baseline = { maxFileSize: undefined, workerTimeout: undefined, verbose: undefined };
try {
expect((await resolveWatchOptions(repoPath, { skipFts: true }, baseline)).skipFts).toBe(true);
expect((await resolveWatchOptions(repoPath, {}, baseline)).skipFts).toBeUndefined();
} finally {
vi.unstubAllEnvs();
}
});
it('accepts every scanner-admitted file instead of maintaining a second allow-list', () => {
expect(isRelevantWatchPath('src/service.ts')).toBe(true);
expect(isRelevantWatchPath('server/app.py')).toBe(true);

2
gitnexus/vitest.config.ts
View file

@ -64,6 +64,7 @@ export default defineConfig({
test: {
name: 'lbug-db',
include: [
'test/integration/skip-fts.test.ts',
'test/integration/impact-callable-value-references.test.ts',
'test/integration/impact-epistemic-lower-bound.test.ts',
'test/integration/impact-scope-omission-persistence.test.ts',
@ -144,6 +145,7 @@ export default defineConfig({
sequence: { groupOrder: 3 },
include: ['test/**/*.test.ts'],
exclude: [
'test/integration/skip-fts.test.ts',
'test/integration/impact-callable-value-references.test.ts',
'test/integration/impact-epistemic-lower-bound.test.ts',
'test/integration/impact-scope-omission-persistence.test.ts',