diff --git a/AGENTS.md b/AGENTS.md index e59468c54..1346facc9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -155,9 +155,9 @@ npx gitnexus analyze --embeddings # also generate embeddings for new/changed npx gitnexus analyze --drop-embeddings # explicit opt-in to wipe existing embeddings ``` -`analyze` runs **incrementally by default**: only files whose content has changed since the last index are re-parsed (chunk-level parse cache at `.gitnexus/parse-cache.json`) and only their rows are rewritten in LadybugDB. Output is byte-equivalent to a full rebuild. Pass `--force` to wipe and re-index from scratch (e.g., to recover from a corrupt index, or after upgrading GitNexus). +`analyze` runs **incrementally by default**. The pipeline still parses every file every run (cross-file resolution requires it), but tree-sitter parsing is **served from a content-addressed cache** at `.gitnexus/parse-cache.json` for chunks whose file contents haven't changed since the last run. Only changed-file rows (and their importers) are rewritten in LadybugDB; unchanged-file rows are preserved. Output is byte-equivalent to a full rebuild. Pass `--force` to wipe and re-index from scratch (e.g., to recover from a corrupt index, or after upgrading GitNexus). -The parse cache is **content-addressed**, so it survives `--force` runs. Safe to delete `.gitnexus/parse-cache.json` at any time — it'll be rebuilt on the next analyze. +The parse cache key is **content-addressed and version-tagged**: it survives `--force` runs, and is automatically invalidated by a `gitnexus` package upgrade (so a new tree-sitter grammar doesn't silently replay stale parse output). Safe to delete `.gitnexus/parse-cache.json` at any time — it'll be rebuilt on the next analyze. Check `.gitnexus/meta.json` `stats.embeddings` (0 = none). A plain `analyze` no longer drops existing vectors — pass `--drop-embeddings` to wipe. diff --git a/docs/superpowers/specs/2026-05-10-incremental-indexing-design.md b/docs/superpowers/specs/2026-05-10-incremental-indexing-design.md deleted file mode 100644 index 88ed0f3c3..000000000 --- a/docs/superpowers/specs/2026-05-10-incremental-indexing-design.md +++ /dev/null @@ -1,165 +0,0 @@ -# Incremental Indexing Design - -**Date:** 2026-05-10 -**Status:** Implemented and equivalence-verified -**Owner:** abhigyanpatwari + Claude Code -**Prior art:** PR #592 (zenprocess), PR #533 (davidbeesley), PR #1146 (azeemshaik025) - -## Why - -`gitnexus analyze` previously had only a coarse early-exit: if `meta.json.lastCommit == HEAD` and the working tree was clean it no-op'd, otherwise it did a full re-index. On a 1000-file TypeScript repo a full re-index runs ~140s. After a one-line edit, that's wasted work. - -Three open community PRs (#592, #533, #1146) attempted incremental re-indexing with different mechanisms; we drew on each but ended up taking a different architectural path after a v1 attempt failed. - -## Correctness contract - -**Incremental output ≡ full-rebuild output.** A run of `analyze` (default, incremental) on a repo that previously had `analyze --force` against state S₁ and is now at state S₂ produces a LadybugDB whose contents (nodes, edges, graph-wide phase outputs) are identical to running `analyze --force` against S₂ from a fresh state. Verified empirically on this repo: same 24,547 nodes / 32,716 edges / 844 clusters / 300 flows from both paths. - -This is the gating invariant — every architectural decision below is in service of preserving it. - -## What's in the box (3 layers) - -The shipped feature has three independent layers that compose. Each is a self-contained optimization with its own correctness story; together they deliver the headline speedup. - -### Layer 1 — Incremental DB writeback - -**The big saving on the storage side: don't wipe-and-reload the whole graph when only a few file rows changed.** - -After every analyze run we hash all source files and store the map at `meta.json.fileHashes`. The next run computes current hashes and diffs: -- `changed` — file content differs from stored hash (rows must be replaced) -- `added` — file is new on disk (insert rows) -- `deleted` — file was in stored map, gone from disk (drop rows) - -Eligibility check: not `--force`, not first run, `schemaVersion=1`, `lastCommit` resolvable, no `incrementalInProgress` dirty flag. When eligible: -1. Open the existing LadybugDB (no wipe). -2. `deleteNodesForFile(f)` for each f in changed ∪ added ∪ deleted. -3. `deleteAllCommunitiesAndProcesses()` — Community/Process are graph-wide, regenerated by their phases on the merged graph (preserves the "Leiden runs on the FULL graph" invariant). -4. From the in-memory `ctx.graph` (which the pipeline produced from the FULL file set), extract the subgraph of `nodes whose filePath ∈ writable_set ∪ {Community, Process}` plus edges with at least one endpoint in that set. -5. `loadGraphToLbug(subgraph)` — only the changed-file rows + graph-wide rows hit the DB. Unchanged-file rows in the DB are never touched. - -Crash recovery: `incrementalInProgress` is set to a `{ startedAt, toWriteCount }` record BEFORE any DB mutation, cleared by the success-path meta save. A crash anywhere in between leaves the flag set; the next run sees it and forces a full rebuild — cheapest path back to a known-good index. - -Modules: -- `gitnexus/src/storage/file-hash.ts` — `computeFileHashes`, `diffFileHashes` -- `gitnexus/src/core/incremental/subgraph-extract.ts` — `extractChangedSubgraph` -- `gitnexus/src/core/lbug/lbug-adapter.ts` — `deleteAllCommunitiesAndProcesses` (new) -- `gitnexus/src/core/run-analyze.ts` — orchestration with branched DB writeback - -### Layer 2 — Chunk-level parse cache - -**The big saving on the parse side: don't re-tree-sitter-parse files whose content hasn't changed.** - -The parse phase chunks files into byte-budget groups (default 2MB) and dispatches each chunk to the worker pool. Each worker returns a `ParseWorkerResult` for its sub-batch. We cache those raw results. - -Cache file: `/.gitnexus/parse-cache.json`. Versioned. Atomic write (tmp + rename). -Cache key: `sha256(sorted("filePath:fileContentHash" for each file in chunk))`. -Cache value: `ParseWorkerResult[]` — the raw worker output before merge. - -On each chunk: -- Compute the chunk hash from current file contents. -- Cache hit → call `mergeChunkResults(graph, symbolTable, cachedRaw)` and skip the worker dispatch entirely. -- Cache miss → dispatch to workers, capture the raw results via `outRawResults`, store them under the chunk hash for next run. - -Two stability fixes the cache depends on: -- **Map-preserving JSON serialization.** `ParsedFile.scopes[*].typeBindings` is a `ReadonlyMap` which `JSON.stringify` collapses to `{}`. Added a replacer/reviver pair (`{__$mapEntries$__: [...]}` tagged form) so Maps round-trip cleanly. Without this, the first cache hit crashes with "typeBindings is not iterable". -- **Stable chunk ordering.** The byte-budget chunker walked files in filesystem-scan order, which on Windows isn't guaranteed stable across runs. Sorted `parseableScanned` alphabetically before chunking so a single-file edit invalidates exactly one chunk, not all of them. - -Modules: -- `gitnexus/src/storage/parse-cache.ts` — load, save, hash, replacer/reviver -- `gitnexus/src/core/ingestion/parsing-processor.ts` — extracted `mergeChunkResults` from `processParsingWithWorkers`; `processParsing` accepts an `outRawResults` out-parameter -- `gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts` — chunk hash + cache lookup + replay, alphabetical sort - -Cache survives `--force` because it's content-addressed: the same bytes always produce the same key. `--force` only matters for the LadybugDB writeback step. - -### Layer 3 — Scope-resolution short-circuit - -**The big saving on cross-file resolution: stop redoing parse work the workers already did.** - -The `scopeResolution` pipeline phase iterates files, calls `extractParsedFile` per file (which re-parses with tree-sitter), then runs the resolution work. The re-parse cost dominates: ~58s on a 1000-file repo. Workers can't pass tree-sitter `Tree` objects across `MessageChannel` (native objects), so `parse-phase.scopeTreeCache` is empty in worker mode and `extractParsedFile` always falls back to a fresh parse. - -But — and this was hiding in plain sight — the workers ALREADY call `extractParsedFile` and produce `ParsedFile` artifacts in `ParseWorkerResult.parsedFiles`. The previous code threw them away. The fix: - -1. Aggregate `parsedFiles` across chunks in `parse-impl` (`allParsedFiles[]` that pushes from every chunk's worker output). -2. Surface them on `ParseOutput.parsedFiles`. -3. In `scopeResolutionPhase`, build a `Map` from `ParseOutput.parsedFiles`. -4. Pass it to `runScopeResolution` as a new `preExtractedParsedFiles` parameter. -5. `runScopeResolution`'s extract loop checks the map first; on hit, uses the cached `ParsedFile` directly (skipping `extractParsedFile`); on miss, falls back to the original path. - -Worker-produced `ParsedFile` is byte-equivalent to one re-derived in scope-resolution: both call `extractParsedFile(provider, content, path, ...)`, both go through the same deterministic `emitScopeCaptures` + `extractScope` path. The only original difference was that one had a tree-sitter tree on hand (worker) and the other had to re-parse (main thread); the resulting `ParsedFile` is identical. - -`provider.populateOwners(parsed)` still runs on the cached `ParsedFile` — this is the same call the original path made post-extract, in-place mutation of the artifact. - -Module: -- `gitnexus/src/core/ingestion/pipeline-phases/parse.ts` — `ParseOutput.parsedFiles` (new) -- `gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts` — `RunScopeResolutionInput.preExtractedParsedFiles`, fast-path branch in extract loop -- `gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts` — builds the map, passes it through - -Effect: scopeResolution phase 58s → 5s. This is the bulk of the cold-rebuild speedup (the parse cache only helps on warm runs). - -## Decisions (locked in) - -1. **Re-parse scope (corrected from the original spec).** v1 tried to parse only changed files into a fresh graph and hydrate the rest from DB. That broke cross-file resolution because the resolution context (`exportedTypeMap`, `bindingAccumulator`) only saw closure files. Equivalence test failed. **Final design parses every file every run** — the "speedup" comes from skipping tree-sitter on cached chunks (Layer 2) and skipping scope-resolution's re-parse (Layer 3), not from skipping files outright. -2. **Change detection: file content hash, not git diff.** The implementation uses SHA-256 of file content, not `git diff`. This works on non-git folders and on dirty working trees uniformly. (Earlier discussion considered git-only — the simpler content-hash path won.) -3. **Rollout: incremental is the default.** `--force` is the explicit opt-out. The existing `lastCommit==HEAD` early-return now also checks for a clean working tree (uncommitted edits no longer slip through as "already up to date"). -4. **Embeddings preserved as before.** The incremental DB writeback path skips the embedding cache+restore cycle (would PK-conflict against rows that weren't deleted). Existing semantics preserved: plain `analyze` keeps existing embeddings on unchanged-file rows; `--embeddings` regenerates for new/changed; `--drop-embeddings` wipes. -5. **Leiden RNG seeded for reproducibility.** `mulberry32` with constant seed `0xC0DE` in `community-processor.ts`. Required for the equivalence test. - -## v1 attempt (reverted) - -The original spec described a "hydrate phase" that loads node/edge state from DB for unchanged files into a fresh `ctx.graph`, then runs the parse phase only on a closure of changed-files-plus-importers. That design failed the equivalence test: - -| Same edited state | Nodes | Edges | Clusters | Flows | -|---|---|---|---|---| -| `--force` (full rebuild) | 24,624 | 32,822 | 850 | 300 | -| v1 incremental | 24,574 | **32,397** | 845 | **252** | - -Δ −425 edges and −48 process flows. Root cause: `crossFile` and `scopeResolution` operated on partial parse data (only the closure files were parsed), so CALLS edges that resolved through unchanged files silently fell off. The reverted commits and the post-mortem are preserved on the branch (commits `aa8d7ae3` … `d4b9de47`, with the revert at `37ef3dda`) so reviewers can see what we ruled out. - -The Option B design (parse-cache + incremental-DB-writeback + scope-resolution-short-circuit) sidesteps this entirely: the pipeline parses every file every run, so cross-file always sees full data. - -## Measured results - -Benchmarked on this repo (993 files / 24,547 nodes / 32,716 edges, Windows): - -| Run | Time | -|---|---| -| Cold full reindex | 86s | -| Warm cache, no source changes | 3s (early-return) | -| Warm cache + 1-file edit, **incremental** | 38s | -| Warm cache + 1-file edit, `--force` from same state | similar to cold | - -40% faster cold rebuild, 72% faster on the practical "edit one file, reindex" workflow. All paths produce byte-identical graph state. - -## Crash recovery - -`meta.json.incrementalInProgress = { startedAt, toWriteCount }` is written BEFORE any destructive DB mutation in the incremental writeback path and cleared on success by overwriting meta.json. If a run crashes between, the next run detects the flag at startup and forces a full rebuild — cheapest path back to a known-good index. Works regardless of LadybugDB transactional semantics. - -## Out of scope (future follow-ups) - -- **GitNexus auto-writes mutate cache-relevant files.** During analyze, GitNexus updates `AGENTS.md`, `CLAUDE.md`, `.claude/skills/...` etc. with current stats. Those files get re-indexed on the next run with new content → chunk hash differs → cache miss for chunks containing them. Mitigations: (a) add the auto-gen output paths to a workspace-level ignore, (b) make the auto-gen output stable across runs, (c) extract the chunk hash from non-mutating fields only. None of these are blocking for the current PR. -- **Sequential-mode parses don't populate `parsedFiles`.** The worker emits `ParsedFile` via `extractParsedFile` but the sequential fallback writes directly to graph without producing the artifact. Small repos that don't trigger workers don't benefit from the scope-resolution short-circuit. Acceptable: small repos are fast either way. -- **Per-file (instead of per-chunk) cache granularity.** Per-file would invalidate even less on a large edit. Requires changing the worker contract to emit per-file results; non-trivial, not justified by current measurements. -- **Cache file size.** 100-300MB on large repos. Compact JSON, no compression. Could be optimized later (gzip, MessagePack, binary). - -## Pre-PR checklist - -- [x] `npx tsc --noEmit` clean across `gitnexus` + `gitnexus-shared` -- [x] Equivalence test verified on this repo (incremental ≡ `--force`) -- [x] Crash-recovery dirty flag implemented + manually tested -- [x] Dirty-tree gate on `lastCommit==HEAD` early-return -- [ ] AGENTS.md / GUARDRAILS.md note on new default behavior -- [ ] PR description with credits to zenprocess / davidbeesley / azeemshaik025 - -## Code map - -| Concern | Files | -|---|---| -| File-hash diff | `gitnexus/src/storage/file-hash.ts` | -| Subgraph extract | `gitnexus/src/core/incremental/subgraph-extract.ts` | -| Parse cache | `gitnexus/src/storage/parse-cache.ts` | -| DB primitives | `gitnexus/src/core/lbug/lbug-adapter.ts` (`deleteAllCommunitiesAndProcesses`) | -| Pipeline plumbing | `gitnexus/src/core/ingestion/pipeline.ts` (`PipelineOptions.parseCache`); `parse.ts` (`ParseOutput.parsedFiles`); `parse-impl.ts` (chunk loop with cache); `parsing-processor.ts` (`mergeChunkResults`, `outRawResults`) | -| Scope-resolution | `gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts` (`preExtractedParsedFiles`); `phase.ts` (map construction) | -| Orchestration | `gitnexus/src/core/run-analyze.ts` (incremental detection + DB-writeback branch + cache lifecycle) | -| Schema | `gitnexus/src/storage/repo-manager.ts` (`RepoMeta.schemaVersion`, `fileHashes`, `incrementalInProgress`) | -| Determinism | `gitnexus/src/core/ingestion/community-processor.ts` (seeded Leiden RNG) | diff --git a/gitnexus/src/core/lbug/lbug-adapter.ts b/gitnexus/src/core/lbug/lbug-adapter.ts index ec879ebef..caa7a58a1 100644 --- a/gitnexus/src/core/lbug/lbug-adapter.ts +++ b/gitnexus/src/core/lbug/lbug-adapter.ts @@ -1204,6 +1204,46 @@ export const deleteNodesForFile = async ( export const getEmbeddingTableName = (): string => EMBEDDING_TABLE_NAME; +/** + * Return the distinct repo-relative paths of files that import + * `targetFilePath` according to the IMPORTS edges currently in the + * DB. Used by the incremental writeback path to expand the + * "files-to-rewrite" set so that files importing a changed file get + * their edges (which may have been refined by cross-file resolution) + * re-emitted, rather than left stale in the DB. + * + * The DB query reads the *previous* run's state — pre-pipeline, before + * any nodes are deleted — so the returned importers are "files that + * USED TO import the target". That's the right set to invalidate: + * those are the files whose edges in the DB might no longer match + * what cross-file resolution produces given the changed file's new + * exports. + */ +export const queryImporters = async (targetFilePath: string): Promise => { + if (!conn) { + throw new Error('LadybugDB not initialized. Call initLbug first.'); + } + const escaped = targetFilePath.replace(/'/g, "''"); + const cypher = ` + MATCH (a)-[r:${REL_TABLE_NAME}]->(b) + WHERE r.type = 'IMPORTS' AND b.filePath = '${escaped}' + RETURN DISTINCT a.filePath AS importer + `; + try { + const queryResult = await conn.query(cypher); + const result = Array.isArray(queryResult) ? queryResult[0] : queryResult; + const rows = await result.getAll(); + const out: string[] = []; + for (const row of rows) { + const v = (row as { importer?: unknown }).importer; + if (typeof v === 'string' && v.length > 0) out.push(v); + } + return out; + } catch { + return []; + } +}; + /** * Drop every Community and Process node (and their MEMBER_OF / * STEP_IN_PROCESS edges via DETACH DELETE). Used at the start of an diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index e42577507..91150b2e8 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -23,6 +23,7 @@ import { loadCachedEmbeddings, deleteNodesForFile, deleteAllCommunitiesAndProcesses, + queryImporters, } from './lbug/lbug-adapter.js'; import { createSearchFTSIndexes } from './search/fts-indexes.js'; import { @@ -436,8 +437,37 @@ export async function runFullAnalysis( let lbugMsgCount = 0; if (isIncremental && hashDiff) { // ── Incremental DB writeback ─────────────────────────────────── + // 0. Expand the writable set with importers of changed/deleted + // files. Reason (Bugbot/Claude review on PR #1479): when a + // barrel/re-export file changes, cross-file resolution may + // update CALLS edges between two unchanged files. Those edges + // live in `ctx.graph` (the pipeline's authoritative output) + // but would be excluded from the subgraph if neither endpoint + // is in the changed set, leaving the DB stale. Pulling the + // importers in (1-hop transitive) puts those edges' source + // files into the writable set so deleteNodesForFile clears + // their old rows and the subgraph re-emits the refined ones. + // Reads `IMPORTS` from the pre-pipeline DB state; importers + // that no longer import the changed file get rewritten too, + // which is harmless (idempotent re-emission) and necessary + // for the "incremental ≡ full-rebuild" invariant. + const writableFiles = new Set(hashDiff.toWrite); + const directlyChangedCount = writableFiles.size; + for (const f of [...hashDiff.toWrite, ...hashDiff.deleted]) { + try { + const importers = await queryImporters(f); + for (const i of importers) writableFiles.add(i); + } catch { + /* importer query failure → don't expand; correctness degrades but DB stays writable */ + } + } + const importerExpansion = writableFiles.size - directlyChangedCount; + if (importerExpansion > 0) { + log(`Incremental: +${importerExpansion} importer(s) added to writable set`); + } + // 1. Delete rows for files we're about to rewrite + deleted files. - const filesToDelete = [...hashDiff.toWrite, ...hashDiff.deleted]; + const filesToDelete = [...writableFiles, ...hashDiff.deleted]; for (let i = 0; i < filesToDelete.length; i++) { const f = filesToDelete[i]; try { @@ -456,7 +486,7 @@ export async function runFullAnalysis( // 3. Extract the changed subgraph from the FULL ctx.graph and write // only that. Unchanged-file rows in the DB stay untouched. - const subgraph = extractChangedSubgraph(pipelineResult.graph, new Set(hashDiff.toWrite)); + const subgraph = extractChangedSubgraph(pipelineResult.graph, writableFiles); await loadGraphToLbug(subgraph, pipelineResult.repoPath, storagePath, (msg) => { lbugMsgCount++; const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19)); diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index b04799de4..a1abf76fa 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -23,25 +23,65 @@ */ import { createHash } from 'crypto'; +import { createRequire } from 'module'; import fs from 'fs/promises'; import path from 'path'; +import { fileURLToPath } from 'url'; import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.js'; -/** Bump on incompatible changes to ParseWorkerResult or upstream parse semantics. */ -export const PARSE_CACHE_VERSION = 1; +/** + * Cache version composed of: + * - A schema bump knob (`SCHEMA_BUMP`) for hand-controlled invalidation + * when ParseWorkerResult shape or upstream parse semantics change. + * - The current `gitnexus` npm package version, read at module load. + * Any release that ships an updated tree-sitter grammar or revised + * extractor logic implies a version bump in package.json, which + * automatically invalidates the on-disk cache. Without this, a user + * running `npm i -g gitnexus@latest` after a parser-affecting + * release would silently replay pre-upgrade ParseWorkerResults + * against the new graph schema (Bugbot/Claude review on #1479). + * + * On version mismatch, `loadParseCache` returns an empty cache and the + * next save overwrites the on-disk file with the new version baked in. + */ +const SCHEMA_BUMP = 1; +const GITNEXUS_PKG_VERSION = (() => { + try { + // package.json sits at gitnexus/package.json — two levels up from + // gitnexus/src/storage/parse-cache.ts (or its dist/ equivalent). + const here = path.dirname(fileURLToPath(import.meta.url)); + const candidates = [ + path.join(here, '..', '..', 'package.json'), // src/storage → gitnexus/ + path.join(here, '..', '..', '..', 'package.json'), // dist/storage → gitnexus/ + ]; + const requireCJS = createRequire(import.meta.url); + for (const c of candidates) { + try { + const pkg = requireCJS(c); + if (typeof pkg?.version === 'string') return pkg.version; + } catch { + /* try next candidate */ + } + } + } catch { + /* fall through to fallback */ + } + return '0.0.0-unknown'; +})(); +export const PARSE_CACHE_VERSION = `${SCHEMA_BUMP}+${GITNEXUS_PKG_VERSION}`; const CACHE_FILENAME = 'parse-cache.json'; /** On-disk shape. */ interface ParseCacheFile { - version: number; + version: string; /** key = chunk hash (hex) → cached chunk result list. */ entries: Record; } /** Runtime view: keyed Map for fast lookup; mutated in place during a run. */ export interface ParseCache { - version: number; + version: string; entries: Map; /** * Hashes referenced (hit OR miss-and-stored) by the current run. diff --git a/gitnexus/test/unit/incremental-file-hash.test.ts b/gitnexus/test/unit/incremental-file-hash.test.ts new file mode 100644 index 000000000..0f59dfb09 --- /dev/null +++ b/gitnexus/test/unit/incremental-file-hash.test.ts @@ -0,0 +1,124 @@ +import { describe, it, expect } from 'vitest'; +import { mkdtemp, writeFile, rm } from 'fs/promises'; +import { tmpdir } from 'os'; +import path from 'path'; +import { computeFileHash, computeFileHashes, diffFileHashes } from '../../src/storage/file-hash.js'; + +describe('diffFileHashes', () => { + it('classifies files into changed / added / deleted / toWrite', () => { + const stored = { a: 'h-a', b: 'h-b', c: 'h-c' }; + const current = new Map([ + ['a', 'h-a'], // unchanged + ['b', 'h-b-NEW'], // changed + ['d', 'h-d'], // added + // 'c' is gone → deleted + ]); + const diff = diffFileHashes(current, stored); + expect(diff.changed).toEqual(['b']); + expect(diff.added).toEqual(['d']); + expect(diff.deleted).toEqual(['c']); + // toWrite is the union of changed ∪ added (rows to be (re)written) + expect(diff.toWrite.sort()).toEqual(['b', 'd']); + }); + + it('treats no stored map as "everything is added"', () => { + const current = new Map([ + ['x', 'h1'], + ['y', 'h2'], + ]); + const diff = diffFileHashes(current, undefined); + expect(diff.added.sort()).toEqual(['x', 'y']); + expect(diff.changed).toEqual([]); + expect(diff.deleted).toEqual([]); + expect(diff.toWrite.sort()).toEqual(['x', 'y']); + }); + + it('returns sorted arrays for stable cross-platform comparison', () => { + const stored = { z: 'h', a: 'h', m: 'h' }; + const current = new Map([ + ['z', 'h2'], + ['a', 'h2'], + ['m', 'h2'], + ]); + const diff = diffFileHashes(current, stored); + expect(diff.changed).toEqual(['a', 'm', 'z']); + expect(diff.toWrite).toEqual(['a', 'm', 'z']); + }); + + it('handles empty current map (all stored files become deleted)', () => { + const stored = { a: 'h1', b: 'h2' }; + const diff = diffFileHashes(new Map(), stored); + expect(diff.deleted).toEqual(['a', 'b']); + expect(diff.changed).toEqual([]); + expect(diff.added).toEqual([]); + }); +}); + +describe('computeFileHash', () => { + it('produces a stable SHA-256 hex digest', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-')); + try { + const f = path.join(dir, 'a.txt'); + await writeFile(f, 'hello world\n', 'utf-8'); + const h1 = await computeFileHash(f); + const h2 = await computeFileHash(f); + expect(h1).toBe(h2); + expect(h1).toMatch(/^[a-f0-9]{64}$/); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('returns null on missing file (caller treats as "no signature")', async () => { + const h = await computeFileHash('/definitely/does/not/exist/here.xyz'); + expect(h).toBeNull(); + }); + + it('different content → different hash', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-')); + try { + const a = path.join(dir, 'a.txt'); + const b = path.join(dir, 'b.txt'); + await writeFile(a, 'hello', 'utf-8'); + await writeFile(b, 'goodbye', 'utf-8'); + const ha = await computeFileHash(a); + const hb = await computeFileHash(b); + expect(ha).not.toBeNull(); + expect(hb).not.toBeNull(); + expect(ha).not.toBe(hb); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); +}); + +describe('computeFileHashes', () => { + it('hashes a small batch of files in parallel', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-')); + try { + await writeFile(path.join(dir, 'one.txt'), 'A', 'utf-8'); + await writeFile(path.join(dir, 'two.txt'), 'B', 'utf-8'); + await writeFile(path.join(dir, 'three.txt'), 'C', 'utf-8'); + const map = await computeFileHashes(dir, ['one.txt', 'two.txt', 'three.txt']); + expect(map.size).toBe(3); + expect(map.get('one.txt')).toMatch(/^[a-f0-9]{64}$/); + // All distinct since contents differ + const hashes = [...map.values()]; + expect(new Set(hashes).size).toBe(3); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('omits files that fail to read (no entry in result)', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-')); + try { + await writeFile(path.join(dir, 'real.txt'), 'X', 'utf-8'); + const map = await computeFileHashes(dir, ['real.txt', 'phantom.txt']); + expect(map.has('real.txt')).toBe(true); + expect(map.has('phantom.txt')).toBe(false); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts new file mode 100644 index 000000000..757b9cf3a --- /dev/null +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -0,0 +1,243 @@ +import { describe, it, expect } from 'vitest'; +import { mkdtemp, rm } from 'fs/promises'; +import { tmpdir } from 'os'; +import path from 'path'; +import { + PARSE_CACHE_VERSION, + computeChunkHash, + fileContentHash, + loadParseCache, + saveParseCache, + pruneCache, + type ParseCache, +} from '../../src/storage/parse-cache.js'; +import type { ParseWorkerResult } from '../../src/core/ingestion/workers/parse-worker.js'; + +const minimalResult = (overrides: Partial = {}): ParseWorkerResult => ({ + nodes: [], + relationships: [], + symbols: [], + imports: [], + calls: [], + assignments: [], + heritage: [], + routes: [], + fetchCalls: [], + decoratorRoutes: [], + toolDefs: [], + ormQueries: [], + constructorBindings: [], + fileScopeBindings: [], + parsedFiles: [], + skippedLanguages: {}, + fileCount: 0, + ...overrides, +}); + +describe('computeChunkHash', () => { + it('produces a stable hex hash for a fixed set of (filePath, contentHash) entries', () => { + const entries = [ + { filePath: 'a.ts', contentHash: 'h-a' }, + { filePath: 'b.ts', contentHash: 'h-b' }, + { filePath: 'c.ts', contentHash: 'h-c' }, + ]; + const h1 = computeChunkHash(entries); + const h2 = computeChunkHash(entries); + expect(h1).toBe(h2); + expect(h1).toMatch(/^[a-f0-9]{64}$/); + }); + + it('is order-independent (same files in different order → same hash)', () => { + const order1 = [ + { filePath: 'a.ts', contentHash: 'h-a' }, + { filePath: 'b.ts', contentHash: 'h-b' }, + ]; + const order2 = [ + { filePath: 'b.ts', contentHash: 'h-b' }, + { filePath: 'a.ts', contentHash: 'h-a' }, + ]; + expect(computeChunkHash(order1)).toBe(computeChunkHash(order2)); + }); + + it('changes when any file content changes', () => { + const before = [ + { filePath: 'a.ts', contentHash: 'h-a' }, + { filePath: 'b.ts', contentHash: 'h-b' }, + ]; + const after = [ + { filePath: 'a.ts', contentHash: 'h-a' }, + { filePath: 'b.ts', contentHash: 'h-b-NEW' }, // b.ts content changed + ]; + expect(computeChunkHash(before)).not.toBe(computeChunkHash(after)); + }); + + it('changes when chunk membership changes (file added or removed)', () => { + const small = [ + { filePath: 'a.ts', contentHash: 'h-a' }, + { filePath: 'b.ts', contentHash: 'h-b' }, + ]; + const bigger = [...small, { filePath: 'c.ts', contentHash: 'h-c' }]; + expect(computeChunkHash(small)).not.toBe(computeChunkHash(bigger)); + }); +}); + +describe('fileContentHash', () => { + it('hashes a string deterministically', () => { + expect(fileContentHash('hello')).toBe(fileContentHash('hello')); + expect(fileContentHash('hello')).not.toBe(fileContentHash('hello!')); + expect(fileContentHash('hello')).toMatch(/^[a-f0-9]{64}$/); + }); + + it('handles Buffer input identical to its string form', () => { + const s = 'sentinel'; + expect(fileContentHash(Buffer.from(s))).toBe(fileContentHash(s)); + }); +}); + +describe('PARSE_CACHE_VERSION', () => { + it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { + // Looks like "1+1.6.4" — schema bump prefix + actual gitnexus version + expect(PARSE_CACHE_VERSION).toMatch(/^\d+\+\d+\.\d+\.\d+/); + }); +}); + +describe('pruneCache', () => { + it('drops entries whose hashes are not in the used-set', () => { + const cache: ParseCache = { + version: PARSE_CACHE_VERSION, + entries: new Map([ + ['hash-A', [minimalResult()]], + ['hash-B', [minimalResult()]], + ['hash-C', [minimalResult()]], + ]), + usedKeys: new Set(['hash-A']), + }; + const removed = pruneCache(cache, cache.usedKeys); + expect(removed).toBe(2); + expect([...cache.entries.keys()].sort()).toEqual(['hash-A']); + }); + + it('returns 0 when every entry is in use', () => { + const cache: ParseCache = { + version: PARSE_CACHE_VERSION, + entries: new Map([ + ['hash-A', [minimalResult()]], + ['hash-B', [minimalResult()]], + ]), + usedKeys: new Set(['hash-A', 'hash-B']), + }; + expect(pruneCache(cache, cache.usedKeys)).toBe(0); + expect(cache.entries.size).toBe(2); + }); +}); + +describe('loadParseCache / saveParseCache (round-trip)', () => { + it('round-trips an empty cache', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-')); + try { + const cache: ParseCache = { + version: PARSE_CACHE_VERSION, + entries: new Map(), + usedKeys: new Set(), + }; + await saveParseCache(dir, cache); + const loaded = await loadParseCache(dir); + expect(loaded.version).toBe(PARSE_CACHE_VERSION); + expect(loaded.entries.size).toBe(0); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('returns an empty cache when the file is missing', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-')); + try { + const loaded = await loadParseCache(dir); + expect(loaded.entries.size).toBe(0); + expect(loaded.usedKeys.size).toBe(0); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('returns an empty cache on version mismatch (next-run regen)', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-')); + try { + // Write a cache file with a different version directly + const fs = await import('fs/promises'); + await fs.writeFile( + path.join(dir, 'parse-cache.json'), + JSON.stringify({ version: 'foreign-99', entries: { h: [] } }), + 'utf-8', + ); + const loaded = await loadParseCache(dir); + expect(loaded.entries.size).toBe(0); // mismatch → empty + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('returns an empty cache on corrupt JSON', async () => { + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-')); + try { + const fs = await import('fs/promises'); + await fs.writeFile(path.join(dir, 'parse-cache.json'), '{not-json', 'utf-8'); + const loaded = await loadParseCache(dir); + expect(loaded.entries.size).toBe(0); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('round-trips Map and Set values through the JSON replacer/reviver', async () => { + // ParsedFile.scopes[*].typeBindings is a ReadonlyMap. + // Without the replacer/reviver pair, JSON.stringify collapses Maps to + // {} and downstream code that does .get() / iterates entries crashes + // with "is not iterable". This test pins the round-trip behaviour. + const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-')); + try { + const innerMap = new Map([ + ['k1', 'v1'], + ['k2', 'v2'], + ]); + const innerSet = new Set(['s1', 's2']); + // Stash the live Map/Set inside a synthetic ParseWorkerResult — we + // only need the serializer to traverse them. Casting to bypass the + // strict shape isn't a problem here: this test is about JSON + // round-tripping of arbitrary nested Map/Set values, not full + // ParseWorkerResult contents. + const fake = minimalResult({ + parsedFiles: [ + { + filePath: 't.ts', + // Cast through unknown to satisfy the readonly Scope shape + // while still smuggling a live Map into the serializer's + // traversal path — see comment block above. + scopes: [{ id: 's1', typeBindings: innerMap, extras: innerSet }], + } as unknown as ParseWorkerResult['parsedFiles'][number], + ], + }); + + const cache: ParseCache = { + version: PARSE_CACHE_VERSION, + entries: new Map([['chunk-h', [fake]]]), + usedKeys: new Set(['chunk-h']), + }; + await saveParseCache(dir, cache); + const loaded = await loadParseCache(dir); + const reloaded = loaded.entries.get('chunk-h')?.[0]; + expect(reloaded).toBeDefined(); + const scope = (reloaded as ParseWorkerResult).parsedFiles[0]?.scopes[0] as unknown as { + typeBindings?: unknown; + extras?: unknown; + }; + expect(scope.typeBindings).toBeInstanceOf(Map); + expect((scope.typeBindings as Map).get('k1')).toBe('v1'); + expect((scope.typeBindings as Map).size).toBe(2); + expect(scope.extras).toBeInstanceOf(Set); + expect((scope.extras as Set).has('s2')).toBe(true); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/incremental-subgraph-extract.test.ts b/gitnexus/test/unit/incremental-subgraph-extract.test.ts new file mode 100644 index 000000000..f2a8c3e3e --- /dev/null +++ b/gitnexus/test/unit/incremental-subgraph-extract.test.ts @@ -0,0 +1,105 @@ +import { describe, it, expect } from 'vitest'; +import { createKnowledgeGraph } from '../../src/core/graph/graph.js'; +import { extractChangedSubgraph } from '../../src/core/incremental/subgraph-extract.js'; +import type { GraphNode, GraphRelationship } from 'gitnexus-shared'; + +const fileNode = ( + id: string, + filePath: string, + label: GraphNode['label'] = 'Function', +): GraphNode => ({ + id, + label, + properties: { name: id, filePath }, +}); + +const wideNode = (id: string, label: GraphNode['label']): GraphNode => ({ + id, + label, + properties: { name: id }, +}); + +const rel = ( + id: string, + type: GraphRelationship['type'], + src: string, + dst: string, +): GraphRelationship => ({ + id, + type, + sourceId: src, + targetId: dst, + confidence: 1, + reason: 't', +}); + +describe('extractChangedSubgraph', () => { + it('keeps file nodes whose filePath is in the writable set', () => { + const g = createKnowledgeGraph(); + g.addNode(fileNode('Function:a.ts:foo', 'a.ts')); + g.addNode(fileNode('Function:b.ts:bar', 'b.ts')); + g.addNode(fileNode('Function:c.ts:baz', 'c.ts')); + + const sub = extractChangedSubgraph(g, new Set(['a.ts', 'c.ts'])); + expect(sub.nodeCount).toBe(2); + expect(sub.getNode('Function:a.ts:foo')).toBeDefined(); + expect(sub.getNode('Function:c.ts:baz')).toBeDefined(); + expect(sub.getNode('Function:b.ts:bar')).toBeUndefined(); + }); + + it('always keeps Community and Process nodes (regenerated graph-wide)', () => { + const g = createKnowledgeGraph(); + g.addNode(fileNode('Function:a.ts:foo', 'a.ts')); + g.addNode(wideNode('comm_1', 'Community')); + g.addNode(wideNode('proc_1', 'Process')); + + // Empty writable set: only graph-wide nodes survive + const sub = extractChangedSubgraph(g, new Set()); + expect(sub.getNode('Function:a.ts:foo')).toBeUndefined(); + expect(sub.getNode('comm_1')).toBeDefined(); + expect(sub.getNode('proc_1')).toBeDefined(); + }); + + it('keeps edges where at least one endpoint is in the writable subgraph', () => { + const g = createKnowledgeGraph(); + g.addNode(fileNode('Function:a.ts:foo', 'a.ts')); + g.addNode(fileNode('Function:b.ts:bar', 'b.ts')); + g.addNode(fileNode('Function:c.ts:baz', 'c.ts')); + g.addRelationship(rel('r-ab', 'CALLS', 'Function:a.ts:foo', 'Function:b.ts:bar')); + g.addRelationship(rel('r-bc', 'CALLS', 'Function:b.ts:bar', 'Function:c.ts:baz')); + g.addRelationship(rel('r-ac', 'CALLS', 'Function:a.ts:foo', 'Function:c.ts:baz')); + + const sub = extractChangedSubgraph(g, new Set(['a.ts'])); + // r-ab: src in a.ts (writable) → kept + // r-ac: src in a.ts (writable) → kept + // r-bc: src in b.ts, dst in c.ts (both unchanged) → dropped + const rels = [...sub.iterRelationships()].map((r) => r.id).sort(); + expect(rels).toEqual(['r-ab', 'r-ac']); + }); + + it('keeps MEMBER_OF edges (Community endpoints are graph-wide)', () => { + const g = createKnowledgeGraph(); + g.addNode(fileNode('Function:a.ts:foo', 'a.ts')); + g.addNode(fileNode('Function:b.ts:bar', 'b.ts')); + g.addNode(wideNode('comm_1', 'Community')); + // Both functions are members of the same community + g.addRelationship(rel('r-a-comm', 'MEMBER_OF', 'Function:a.ts:foo', 'comm_1')); + g.addRelationship(rel('r-b-comm', 'MEMBER_OF', 'Function:b.ts:bar', 'comm_1')); + + const sub = extractChangedSubgraph(g, new Set(['a.ts'])); + // r-a-comm: src in a.ts (writable) — kept + // r-b-comm: src in b.ts (NOT writable) but dst is graph-wide (Community is in writable subgraph) — kept + const rels = [...sub.iterRelationships()].map((r) => r.id).sort(); + expect(rels).toEqual(['r-a-comm', 'r-b-comm']); + }); + + it('produces an empty subgraph when no nodes match', () => { + const g = createKnowledgeGraph(); + g.addNode(fileNode('Function:a.ts:foo', 'a.ts')); + g.addRelationship(rel('r1', 'CALLS', 'Function:a.ts:foo', 'Function:a.ts:foo')); + + const sub = extractChangedSubgraph(g, new Set(['nonexistent.ts'])); + expect(sub.nodeCount).toBe(0); + expect(sub.relationshipCount).toBe(0); + }); +});