fix(incremental): address Claude+Bugbot review findings + remove design doc

Addresses CHANGES_REQUESTED review on PR #1479:

1. Remove docs/superpowers/specs/2026-05-10-incremental-indexing-design.md
   per maintainer request.

2. BLOCKER (Claude Finding 1, Bugbot Round 3): Stale cross-file edges
   between unchanged files. extractChangedSubgraph excluded edges where
   both endpoints were unchanged-file nodes — when a barrel/re-export
   file changes, cross-file resolution may update CALLS edges between
   two unchanged files that would then be silently lost.

   Fix: 1-hop importer-closure expansion of the writable set in
   run-analyze.ts. Before deleting/rewriting rows, query DB for
   importers of every changed/deleted file and add them to the writable
   set. Their nodes get deleted+rewritten too, so cross-file's refined
   edges land in the DB. Re-added queryImporters to lbug-adapter.ts.

3. BLOCKER (Claude Finding 3): Parse cache key omitted parser version.
   After a GitNexus upgrade, the cache silently replays pre-upgrade
   ParseWorkerResults against the new schema → wrong CALLS/IMPORTS/
   scope edges with no visible signal.

   Fix: PARSE_CACHE_VERSION now embeds the gitnexus npm package
   version (read at module load via createRequire on package.json).
   Format: `${SCHEMA_BUMP}+${PKG_VERSION}` e.g. "1+1.6.4". Any release
   that bumps package.json automatically invalidates the on-disk cache.
   Mismatched versions fall through to an empty cache (next save
   overwrites with the new version baked in).

4. BLOCKER (Claude Finding 2): No automated tests for incremental
   behavior. Added 28 unit tests across 3 files:

     - incremental-file-hash.test.ts (10 tests)
       diffFileHashes classification, computeFileHash determinism,
       computeFileHashes batch / missing-file tolerance, sorted output.

     - incremental-parse-cache.test.ts (12 tests)
       computeChunkHash stability and order-independence, version
       prefix format, pruneCache, load/save round-trip on empty /
       missing / corrupt / version-mismatched files, AND a Map/Set
       round-trip test that pins the JSON replacer/reviver behaviour
       (without it, ParsedFile.scopes[*].typeBindings collapses to
       {} and downstream `.get()` / iteration throws).

     - incremental-subgraph-extract.test.ts (6 tests)
       writable-set node inclusion, Community/Process always kept,
       edge inclusion when at least one endpoint is writable, MEMBER_OF
       edges via graph-wide endpoints, empty subgraph case.

5. Medium (Claude Finding 6): AGENTS.md "Keeping the Index Fresh"
   said "only changed files are re-parsed." Imprecise — the pipeline
   parses every file every run; the cache skips tree-sitter for chunks
   whose contents haven't changed. Reworded to match the design doc.

Test plan still expects:
  [x] Typecheck clean
  [x] All 28 new unit tests pass
  [x] All previously-failing tests still pass on the rebased branch
  [x] Equivalence verified locally (incremental ≡ --force, byte-identical
      stats on this repo)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
abhigyanpatwari 2026-05-10 21:19:16 +05:30
parent d8b6971714
commit ad7bd311a9
8 changed files with 590 additions and 173 deletions

View file

@ -155,9 +155,9 @@ npx gitnexus analyze --embeddings # also generate embeddings for new/changed
npx gitnexus analyze --drop-embeddings # explicit opt-in to wipe existing embeddings
```
`analyze` runs **incrementally by default**: only files whose content has changed since the last index are re-parsed (chunk-level parse cache at `.gitnexus/parse-cache.json`) and only their rows are rewritten in LadybugDB. Output is byte-equivalent to a full rebuild. Pass `--force` to wipe and re-index from scratch (e.g., to recover from a corrupt index, or after upgrading GitNexus).
`analyze` runs **incrementally by default**. The pipeline still parses every file every run (cross-file resolution requires it), but tree-sitter parsing is **served from a content-addressed cache** at `.gitnexus/parse-cache.json` for chunks whose file contents haven't changed since the last run. Only changed-file rows (and their importers) are rewritten in LadybugDB; unchanged-file rows are preserved. Output is byte-equivalent to a full rebuild. Pass `--force` to wipe and re-index from scratch (e.g., to recover from a corrupt index, or after upgrading GitNexus).
The parse cache is **content-addressed**, so it survives `--force` runs. Safe to delete `.gitnexus/parse-cache.json` at any time — it'll be rebuilt on the next analyze.
The parse cache key is **content-addressed and version-tagged**: it survives `--force` runs, and is automatically invalidated by a `gitnexus` package upgrade (so a new tree-sitter grammar doesn't silently replay stale parse output). Safe to delete `.gitnexus/parse-cache.json` at any time — it'll be rebuilt on the next analyze.
Check `.gitnexus/meta.json` `stats.embeddings` (0 = none). A plain `analyze` no longer drops existing vectors — pass `--drop-embeddings` to wipe.

View file

@ -1,165 +0,0 @@
# Incremental Indexing Design
**Date:** 2026-05-10
**Status:** Implemented and equivalence-verified
**Owner:** abhigyanpatwari + Claude Code
**Prior art:** PR #592 (zenprocess), PR #533 (davidbeesley), PR #1146 (azeemshaik025)
## Why
`gitnexus analyze` previously had only a coarse early-exit: if `meta.json.lastCommit == HEAD` and the working tree was clean it no-op'd, otherwise it did a full re-index. On a 1000-file TypeScript repo a full re-index runs ~140s. After a one-line edit, that's wasted work.
Three open community PRs (#592, #533, #1146) attempted incremental re-indexing with different mechanisms; we drew on each but ended up taking a different architectural path after a v1 attempt failed.
## Correctness contract
**Incremental output ≡ full-rebuild output.** A run of `analyze` (default, incremental) on a repo that previously had `analyze --force` against state S₁ and is now at state S₂ produces a LadybugDB whose contents (nodes, edges, graph-wide phase outputs) are identical to running `analyze --force` against S₂ from a fresh state. Verified empirically on this repo: same 24,547 nodes / 32,716 edges / 844 clusters / 300 flows from both paths.
This is the gating invariant — every architectural decision below is in service of preserving it.
## What's in the box (3 layers)
The shipped feature has three independent layers that compose. Each is a self-contained optimization with its own correctness story; together they deliver the headline speedup.
### Layer 1 — Incremental DB writeback
**The big saving on the storage side: don't wipe-and-reload the whole graph when only a few file rows changed.**
After every analyze run we hash all source files and store the map at `meta.json.fileHashes`. The next run computes current hashes and diffs:
- `changed` — file content differs from stored hash (rows must be replaced)
- `added` — file is new on disk (insert rows)
- `deleted` — file was in stored map, gone from disk (drop rows)
Eligibility check: not `--force`, not first run, `schemaVersion=1`, `lastCommit` resolvable, no `incrementalInProgress` dirty flag. When eligible:
1. Open the existing LadybugDB (no wipe).
2. `deleteNodesForFile(f)` for each f in changed added deleted.
3. `deleteAllCommunitiesAndProcesses()` — Community/Process are graph-wide, regenerated by their phases on the merged graph (preserves the "Leiden runs on the FULL graph" invariant).
4. From the in-memory `ctx.graph` (which the pipeline produced from the FULL file set), extract the subgraph of `nodes whose filePath ∈ writable_set {Community, Process}` plus edges with at least one endpoint in that set.
5. `loadGraphToLbug(subgraph)` — only the changed-file rows + graph-wide rows hit the DB. Unchanged-file rows in the DB are never touched.
Crash recovery: `incrementalInProgress` is set to a `{ startedAt, toWriteCount }` record BEFORE any DB mutation, cleared by the success-path meta save. A crash anywhere in between leaves the flag set; the next run sees it and forces a full rebuild — cheapest path back to a known-good index.
Modules:
- `gitnexus/src/storage/file-hash.ts``computeFileHashes`, `diffFileHashes`
- `gitnexus/src/core/incremental/subgraph-extract.ts``extractChangedSubgraph`
- `gitnexus/src/core/lbug/lbug-adapter.ts``deleteAllCommunitiesAndProcesses` (new)
- `gitnexus/src/core/run-analyze.ts` — orchestration with branched DB writeback
### Layer 2 — Chunk-level parse cache
**The big saving on the parse side: don't re-tree-sitter-parse files whose content hasn't changed.**
The parse phase chunks files into byte-budget groups (default 2MB) and dispatches each chunk to the worker pool. Each worker returns a `ParseWorkerResult` for its sub-batch. We cache those raw results.
Cache file: `<repo>/.gitnexus/parse-cache.json`. Versioned. Atomic write (tmp + rename).
Cache key: `sha256(sorted("filePath:fileContentHash" for each file in chunk))`.
Cache value: `ParseWorkerResult[]` — the raw worker output before merge.
On each chunk:
- Compute the chunk hash from current file contents.
- Cache hit → call `mergeChunkResults(graph, symbolTable, cachedRaw)` and skip the worker dispatch entirely.
- Cache miss → dispatch to workers, capture the raw results via `outRawResults`, store them under the chunk hash for next run.
Two stability fixes the cache depends on:
- **Map-preserving JSON serialization.** `ParsedFile.scopes[*].typeBindings` is a `ReadonlyMap<string, TypeRef>` which `JSON.stringify` collapses to `{}`. Added a replacer/reviver pair (`{__$mapEntries$__: [...]}` tagged form) so Maps round-trip cleanly. Without this, the first cache hit crashes with "typeBindings is not iterable".
- **Stable chunk ordering.** The byte-budget chunker walked files in filesystem-scan order, which on Windows isn't guaranteed stable across runs. Sorted `parseableScanned` alphabetically before chunking so a single-file edit invalidates exactly one chunk, not all of them.
Modules:
- `gitnexus/src/storage/parse-cache.ts` — load, save, hash, replacer/reviver
- `gitnexus/src/core/ingestion/parsing-processor.ts` — extracted `mergeChunkResults` from `processParsingWithWorkers`; `processParsing` accepts an `outRawResults` out-parameter
- `gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts` — chunk hash + cache lookup + replay, alphabetical sort
Cache survives `--force` because it's content-addressed: the same bytes always produce the same key. `--force` only matters for the LadybugDB writeback step.
### Layer 3 — Scope-resolution short-circuit
**The big saving on cross-file resolution: stop redoing parse work the workers already did.**
The `scopeResolution` pipeline phase iterates files, calls `extractParsedFile` per file (which re-parses with tree-sitter), then runs the resolution work. The re-parse cost dominates: ~58s on a 1000-file repo. Workers can't pass tree-sitter `Tree` objects across `MessageChannel` (native objects), so `parse-phase.scopeTreeCache` is empty in worker mode and `extractParsedFile` always falls back to a fresh parse.
But — and this was hiding in plain sight — the workers ALREADY call `extractParsedFile` and produce `ParsedFile` artifacts in `ParseWorkerResult.parsedFiles`. The previous code threw them away. The fix:
1. Aggregate `parsedFiles` across chunks in `parse-impl` (`allParsedFiles[]` that pushes from every chunk's worker output).
2. Surface them on `ParseOutput.parsedFiles`.
3. In `scopeResolutionPhase`, build a `Map<filePath, ParsedFile>` from `ParseOutput.parsedFiles`.
4. Pass it to `runScopeResolution` as a new `preExtractedParsedFiles` parameter.
5. `runScopeResolution`'s extract loop checks the map first; on hit, uses the cached `ParsedFile` directly (skipping `extractParsedFile`); on miss, falls back to the original path.
Worker-produced `ParsedFile` is byte-equivalent to one re-derived in scope-resolution: both call `extractParsedFile(provider, content, path, ...)`, both go through the same deterministic `emitScopeCaptures` + `extractScope` path. The only original difference was that one had a tree-sitter tree on hand (worker) and the other had to re-parse (main thread); the resulting `ParsedFile` is identical.
`provider.populateOwners(parsed)` still runs on the cached `ParsedFile` — this is the same call the original path made post-extract, in-place mutation of the artifact.
Module:
- `gitnexus/src/core/ingestion/pipeline-phases/parse.ts``ParseOutput.parsedFiles` (new)
- `gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts``RunScopeResolutionInput.preExtractedParsedFiles`, fast-path branch in extract loop
- `gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts` — builds the map, passes it through
Effect: scopeResolution phase 58s → 5s. This is the bulk of the cold-rebuild speedup (the parse cache only helps on warm runs).
## Decisions (locked in)
1. **Re-parse scope (corrected from the original spec).** v1 tried to parse only changed files into a fresh graph and hydrate the rest from DB. That broke cross-file resolution because the resolution context (`exportedTypeMap`, `bindingAccumulator`) only saw closure files. Equivalence test failed. **Final design parses every file every run** — the "speedup" comes from skipping tree-sitter on cached chunks (Layer 2) and skipping scope-resolution's re-parse (Layer 3), not from skipping files outright.
2. **Change detection: file content hash, not git diff.** The implementation uses SHA-256 of file content, not `git diff`. This works on non-git folders and on dirty working trees uniformly. (Earlier discussion considered git-only — the simpler content-hash path won.)
3. **Rollout: incremental is the default.** `--force` is the explicit opt-out. The existing `lastCommit==HEAD` early-return now also checks for a clean working tree (uncommitted edits no longer slip through as "already up to date").
4. **Embeddings preserved as before.** The incremental DB writeback path skips the embedding cache+restore cycle (would PK-conflict against rows that weren't deleted). Existing semantics preserved: plain `analyze` keeps existing embeddings on unchanged-file rows; `--embeddings` regenerates for new/changed; `--drop-embeddings` wipes.
5. **Leiden RNG seeded for reproducibility.** `mulberry32` with constant seed `0xC0DE` in `community-processor.ts`. Required for the equivalence test.
## v1 attempt (reverted)
The original spec described a "hydrate phase" that loads node/edge state from DB for unchanged files into a fresh `ctx.graph`, then runs the parse phase only on a closure of changed-files-plus-importers. That design failed the equivalence test:
| Same edited state | Nodes | Edges | Clusters | Flows |
|---|---|---|---|---|
| `--force` (full rebuild) | 24,624 | 32,822 | 850 | 300 |
| v1 incremental | 24,574 | **32,397** | 845 | **252** |
Δ 425 edges and 48 process flows. Root cause: `crossFile` and `scopeResolution` operated on partial parse data (only the closure files were parsed), so CALLS edges that resolved through unchanged files silently fell off. The reverted commits and the post-mortem are preserved on the branch (commits `aa8d7ae3``d4b9de47`, with the revert at `37ef3dda`) so reviewers can see what we ruled out.
The Option B design (parse-cache + incremental-DB-writeback + scope-resolution-short-circuit) sidesteps this entirely: the pipeline parses every file every run, so cross-file always sees full data.
## Measured results
Benchmarked on this repo (993 files / 24,547 nodes / 32,716 edges, Windows):
| Run | Time |
|---|---|
| Cold full reindex | 86s |
| Warm cache, no source changes | 3s (early-return) |
| Warm cache + 1-file edit, **incremental** | 38s |
| Warm cache + 1-file edit, `--force` from same state | similar to cold |
40% faster cold rebuild, 72% faster on the practical "edit one file, reindex" workflow. All paths produce byte-identical graph state.
## Crash recovery
`meta.json.incrementalInProgress = { startedAt, toWriteCount }` is written BEFORE any destructive DB mutation in the incremental writeback path and cleared on success by overwriting meta.json. If a run crashes between, the next run detects the flag at startup and forces a full rebuild — cheapest path back to a known-good index. Works regardless of LadybugDB transactional semantics.
## Out of scope (future follow-ups)
- **GitNexus auto-writes mutate cache-relevant files.** During analyze, GitNexus updates `AGENTS.md`, `CLAUDE.md`, `.claude/skills/...` etc. with current stats. Those files get re-indexed on the next run with new content → chunk hash differs → cache miss for chunks containing them. Mitigations: (a) add the auto-gen output paths to a workspace-level ignore, (b) make the auto-gen output stable across runs, (c) extract the chunk hash from non-mutating fields only. None of these are blocking for the current PR.
- **Sequential-mode parses don't populate `parsedFiles`.** The worker emits `ParsedFile` via `extractParsedFile` but the sequential fallback writes directly to graph without producing the artifact. Small repos that don't trigger workers don't benefit from the scope-resolution short-circuit. Acceptable: small repos are fast either way.
- **Per-file (instead of per-chunk) cache granularity.** Per-file would invalidate even less on a large edit. Requires changing the worker contract to emit per-file results; non-trivial, not justified by current measurements.
- **Cache file size.** 100-300MB on large repos. Compact JSON, no compression. Could be optimized later (gzip, MessagePack, binary).
## Pre-PR checklist
- [x] `npx tsc --noEmit` clean across `gitnexus` + `gitnexus-shared`
- [x] Equivalence test verified on this repo (incremental ≡ `--force`)
- [x] Crash-recovery dirty flag implemented + manually tested
- [x] Dirty-tree gate on `lastCommit==HEAD` early-return
- [ ] AGENTS.md / GUARDRAILS.md note on new default behavior
- [ ] PR description with credits to zenprocess / davidbeesley / azeemshaik025
## Code map
| Concern | Files |
|---|---|
| File-hash diff | `gitnexus/src/storage/file-hash.ts` |
| Subgraph extract | `gitnexus/src/core/incremental/subgraph-extract.ts` |
| Parse cache | `gitnexus/src/storage/parse-cache.ts` |
| DB primitives | `gitnexus/src/core/lbug/lbug-adapter.ts` (`deleteAllCommunitiesAndProcesses`) |
| Pipeline plumbing | `gitnexus/src/core/ingestion/pipeline.ts` (`PipelineOptions.parseCache`); `parse.ts` (`ParseOutput.parsedFiles`); `parse-impl.ts` (chunk loop with cache); `parsing-processor.ts` (`mergeChunkResults`, `outRawResults`) |
| Scope-resolution | `gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts` (`preExtractedParsedFiles`); `phase.ts` (map construction) |
| Orchestration | `gitnexus/src/core/run-analyze.ts` (incremental detection + DB-writeback branch + cache lifecycle) |
| Schema | `gitnexus/src/storage/repo-manager.ts` (`RepoMeta.schemaVersion`, `fileHashes`, `incrementalInProgress`) |
| Determinism | `gitnexus/src/core/ingestion/community-processor.ts` (seeded Leiden RNG) |

View file

@ -1204,6 +1204,46 @@ export const deleteNodesForFile = async (
export const getEmbeddingTableName = (): string => EMBEDDING_TABLE_NAME;
/**
* Return the distinct repo-relative paths of files that import
* `targetFilePath` according to the IMPORTS edges currently in the
* DB. Used by the incremental writeback path to expand the
* "files-to-rewrite" set so that files importing a changed file get
* their edges (which may have been refined by cross-file resolution)
* re-emitted, rather than left stale in the DB.
*
* The DB query reads the *previous* run's state pre-pipeline, before
* any nodes are deleted so the returned importers are "files that
* USED TO import the target". That's the right set to invalidate:
* those are the files whose edges in the DB might no longer match
* what cross-file resolution produces given the changed file's new
* exports.
*/
export const queryImporters = async (targetFilePath: string): Promise<string[]> => {
if (!conn) {
throw new Error('LadybugDB not initialized. Call initLbug first.');
}
const escaped = targetFilePath.replace(/'/g, "''");
const cypher = `
MATCH (a)-[r:${REL_TABLE_NAME}]->(b)
WHERE r.type = 'IMPORTS' AND b.filePath = '${escaped}'
RETURN DISTINCT a.filePath AS importer
`;
try {
const queryResult = await conn.query(cypher);
const result = Array.isArray(queryResult) ? queryResult[0] : queryResult;
const rows = await result.getAll();
const out: string[] = [];
for (const row of rows) {
const v = (row as { importer?: unknown }).importer;
if (typeof v === 'string' && v.length > 0) out.push(v);
}
return out;
} catch {
return [];
}
};
/**
* Drop every Community and Process node (and their MEMBER_OF /
* STEP_IN_PROCESS edges via DETACH DELETE). Used at the start of an

View file

@ -23,6 +23,7 @@ import {
loadCachedEmbeddings,
deleteNodesForFile,
deleteAllCommunitiesAndProcesses,
queryImporters,
} from './lbug/lbug-adapter.js';
import { createSearchFTSIndexes } from './search/fts-indexes.js';
import {
@ -436,8 +437,37 @@ export async function runFullAnalysis(
let lbugMsgCount = 0;
if (isIncremental && hashDiff) {
// ── Incremental DB writeback ───────────────────────────────────
// 0. Expand the writable set with importers of changed/deleted
// files. Reason (Bugbot/Claude review on PR #1479): when a
// barrel/re-export file changes, cross-file resolution may
// update CALLS edges between two unchanged files. Those edges
// live in `ctx.graph` (the pipeline's authoritative output)
// but would be excluded from the subgraph if neither endpoint
// is in the changed set, leaving the DB stale. Pulling the
// importers in (1-hop transitive) puts those edges' source
// files into the writable set so deleteNodesForFile clears
// their old rows and the subgraph re-emits the refined ones.
// Reads `IMPORTS` from the pre-pipeline DB state; importers
// that no longer import the changed file get rewritten too,
// which is harmless (idempotent re-emission) and necessary
// for the "incremental ≡ full-rebuild" invariant.
const writableFiles = new Set<string>(hashDiff.toWrite);
const directlyChangedCount = writableFiles.size;
for (const f of [...hashDiff.toWrite, ...hashDiff.deleted]) {
try {
const importers = await queryImporters(f);
for (const i of importers) writableFiles.add(i);
} catch {
/* importer query failure → don't expand; correctness degrades but DB stays writable */
}
}
const importerExpansion = writableFiles.size - directlyChangedCount;
if (importerExpansion > 0) {
log(`Incremental: +${importerExpansion} importer(s) added to writable set`);
}
// 1. Delete rows for files we're about to rewrite + deleted files.
const filesToDelete = [...hashDiff.toWrite, ...hashDiff.deleted];
const filesToDelete = [...writableFiles, ...hashDiff.deleted];
for (let i = 0; i < filesToDelete.length; i++) {
const f = filesToDelete[i];
try {
@ -456,7 +486,7 @@ export async function runFullAnalysis(
// 3. Extract the changed subgraph from the FULL ctx.graph and write
// only that. Unchanged-file rows in the DB stay untouched.
const subgraph = extractChangedSubgraph(pipelineResult.graph, new Set(hashDiff.toWrite));
const subgraph = extractChangedSubgraph(pipelineResult.graph, writableFiles);
await loadGraphToLbug(subgraph, pipelineResult.repoPath, storagePath, (msg) => {
lbugMsgCount++;
const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19));

View file

@ -23,25 +23,65 @@
*/
import { createHash } from 'crypto';
import { createRequire } from 'module';
import fs from 'fs/promises';
import path from 'path';
import { fileURLToPath } from 'url';
import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.js';
/** Bump on incompatible changes to ParseWorkerResult or upstream parse semantics. */
export const PARSE_CACHE_VERSION = 1;
/**
* Cache version composed of:
* - A schema bump knob (`SCHEMA_BUMP`) for hand-controlled invalidation
* when ParseWorkerResult shape or upstream parse semantics change.
* - The current `gitnexus` npm package version, read at module load.
* Any release that ships an updated tree-sitter grammar or revised
* extractor logic implies a version bump in package.json, which
* automatically invalidates the on-disk cache. Without this, a user
* running `npm i -g gitnexus@latest` after a parser-affecting
* release would silently replay pre-upgrade ParseWorkerResults
* against the new graph schema (Bugbot/Claude review on #1479).
*
* On version mismatch, `loadParseCache` returns an empty cache and the
* next save overwrites the on-disk file with the new version baked in.
*/
const SCHEMA_BUMP = 1;
const GITNEXUS_PKG_VERSION = (() => {
try {
// package.json sits at gitnexus/package.json — two levels up from
// gitnexus/src/storage/parse-cache.ts (or its dist/ equivalent).
const here = path.dirname(fileURLToPath(import.meta.url));
const candidates = [
path.join(here, '..', '..', 'package.json'), // src/storage → gitnexus/
path.join(here, '..', '..', '..', 'package.json'), // dist/storage → gitnexus/
];
const requireCJS = createRequire(import.meta.url);
for (const c of candidates) {
try {
const pkg = requireCJS(c);
if (typeof pkg?.version === 'string') return pkg.version;
} catch {
/* try next candidate */
}
}
} catch {
/* fall through to fallback */
}
return '0.0.0-unknown';
})();
export const PARSE_CACHE_VERSION = `${SCHEMA_BUMP}+${GITNEXUS_PKG_VERSION}`;
const CACHE_FILENAME = 'parse-cache.json';
/** On-disk shape. */
interface ParseCacheFile {
version: number;
version: string;
/** key = chunk hash (hex) → cached chunk result list. */
entries: Record<string, ParseWorkerResult[]>;
}
/** Runtime view: keyed Map for fast lookup; mutated in place during a run. */
export interface ParseCache {
version: number;
version: string;
entries: Map<string, ParseWorkerResult[]>;
/**
* Hashes referenced (hit OR miss-and-stored) by the current run.

View file

@ -0,0 +1,124 @@
import { describe, it, expect } from 'vitest';
import { mkdtemp, writeFile, rm } from 'fs/promises';
import { tmpdir } from 'os';
import path from 'path';
import { computeFileHash, computeFileHashes, diffFileHashes } from '../../src/storage/file-hash.js';
describe('diffFileHashes', () => {
it('classifies files into changed / added / deleted / toWrite', () => {
const stored = { a: 'h-a', b: 'h-b', c: 'h-c' };
const current = new Map<string, string>([
['a', 'h-a'], // unchanged
['b', 'h-b-NEW'], // changed
['d', 'h-d'], // added
// 'c' is gone → deleted
]);
const diff = diffFileHashes(current, stored);
expect(diff.changed).toEqual(['b']);
expect(diff.added).toEqual(['d']);
expect(diff.deleted).toEqual(['c']);
// toWrite is the union of changed added (rows to be (re)written)
expect(diff.toWrite.sort()).toEqual(['b', 'd']);
});
it('treats no stored map as "everything is added"', () => {
const current = new Map<string, string>([
['x', 'h1'],
['y', 'h2'],
]);
const diff = diffFileHashes(current, undefined);
expect(diff.added.sort()).toEqual(['x', 'y']);
expect(diff.changed).toEqual([]);
expect(diff.deleted).toEqual([]);
expect(diff.toWrite.sort()).toEqual(['x', 'y']);
});
it('returns sorted arrays for stable cross-platform comparison', () => {
const stored = { z: 'h', a: 'h', m: 'h' };
const current = new Map<string, string>([
['z', 'h2'],
['a', 'h2'],
['m', 'h2'],
]);
const diff = diffFileHashes(current, stored);
expect(diff.changed).toEqual(['a', 'm', 'z']);
expect(diff.toWrite).toEqual(['a', 'm', 'z']);
});
it('handles empty current map (all stored files become deleted)', () => {
const stored = { a: 'h1', b: 'h2' };
const diff = diffFileHashes(new Map(), stored);
expect(diff.deleted).toEqual(['a', 'b']);
expect(diff.changed).toEqual([]);
expect(diff.added).toEqual([]);
});
});
describe('computeFileHash', () => {
it('produces a stable SHA-256 hex digest', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
try {
const f = path.join(dir, 'a.txt');
await writeFile(f, 'hello world\n', 'utf-8');
const h1 = await computeFileHash(f);
const h2 = await computeFileHash(f);
expect(h1).toBe(h2);
expect(h1).toMatch(/^[a-f0-9]{64}$/);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('returns null on missing file (caller treats as "no signature")', async () => {
const h = await computeFileHash('/definitely/does/not/exist/here.xyz');
expect(h).toBeNull();
});
it('different content → different hash', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
try {
const a = path.join(dir, 'a.txt');
const b = path.join(dir, 'b.txt');
await writeFile(a, 'hello', 'utf-8');
await writeFile(b, 'goodbye', 'utf-8');
const ha = await computeFileHash(a);
const hb = await computeFileHash(b);
expect(ha).not.toBeNull();
expect(hb).not.toBeNull();
expect(ha).not.toBe(hb);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
});
describe('computeFileHashes', () => {
it('hashes a small batch of files in parallel', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
try {
await writeFile(path.join(dir, 'one.txt'), 'A', 'utf-8');
await writeFile(path.join(dir, 'two.txt'), 'B', 'utf-8');
await writeFile(path.join(dir, 'three.txt'), 'C', 'utf-8');
const map = await computeFileHashes(dir, ['one.txt', 'two.txt', 'three.txt']);
expect(map.size).toBe(3);
expect(map.get('one.txt')).toMatch(/^[a-f0-9]{64}$/);
// All distinct since contents differ
const hashes = [...map.values()];
expect(new Set(hashes).size).toBe(3);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('omits files that fail to read (no entry in result)', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
try {
await writeFile(path.join(dir, 'real.txt'), 'X', 'utf-8');
const map = await computeFileHashes(dir, ['real.txt', 'phantom.txt']);
expect(map.has('real.txt')).toBe(true);
expect(map.has('phantom.txt')).toBe(false);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
});

View file

@ -0,0 +1,243 @@
import { describe, it, expect } from 'vitest';
import { mkdtemp, rm } from 'fs/promises';
import { tmpdir } from 'os';
import path from 'path';
import {
PARSE_CACHE_VERSION,
computeChunkHash,
fileContentHash,
loadParseCache,
saveParseCache,
pruneCache,
type ParseCache,
} from '../../src/storage/parse-cache.js';
import type { ParseWorkerResult } from '../../src/core/ingestion/workers/parse-worker.js';
const minimalResult = (overrides: Partial<ParseWorkerResult> = {}): ParseWorkerResult => ({
nodes: [],
relationships: [],
symbols: [],
imports: [],
calls: [],
assignments: [],
heritage: [],
routes: [],
fetchCalls: [],
decoratorRoutes: [],
toolDefs: [],
ormQueries: [],
constructorBindings: [],
fileScopeBindings: [],
parsedFiles: [],
skippedLanguages: {},
fileCount: 0,
...overrides,
});
describe('computeChunkHash', () => {
it('produces a stable hex hash for a fixed set of (filePath, contentHash) entries', () => {
const entries = [
{ filePath: 'a.ts', contentHash: 'h-a' },
{ filePath: 'b.ts', contentHash: 'h-b' },
{ filePath: 'c.ts', contentHash: 'h-c' },
];
const h1 = computeChunkHash(entries);
const h2 = computeChunkHash(entries);
expect(h1).toBe(h2);
expect(h1).toMatch(/^[a-f0-9]{64}$/);
});
it('is order-independent (same files in different order → same hash)', () => {
const order1 = [
{ filePath: 'a.ts', contentHash: 'h-a' },
{ filePath: 'b.ts', contentHash: 'h-b' },
];
const order2 = [
{ filePath: 'b.ts', contentHash: 'h-b' },
{ filePath: 'a.ts', contentHash: 'h-a' },
];
expect(computeChunkHash(order1)).toBe(computeChunkHash(order2));
});
it('changes when any file content changes', () => {
const before = [
{ filePath: 'a.ts', contentHash: 'h-a' },
{ filePath: 'b.ts', contentHash: 'h-b' },
];
const after = [
{ filePath: 'a.ts', contentHash: 'h-a' },
{ filePath: 'b.ts', contentHash: 'h-b-NEW' }, // b.ts content changed
];
expect(computeChunkHash(before)).not.toBe(computeChunkHash(after));
});
it('changes when chunk membership changes (file added or removed)', () => {
const small = [
{ filePath: 'a.ts', contentHash: 'h-a' },
{ filePath: 'b.ts', contentHash: 'h-b' },
];
const bigger = [...small, { filePath: 'c.ts', contentHash: 'h-c' }];
expect(computeChunkHash(small)).not.toBe(computeChunkHash(bigger));
});
});
describe('fileContentHash', () => {
it('hashes a string deterministically', () => {
expect(fileContentHash('hello')).toBe(fileContentHash('hello'));
expect(fileContentHash('hello')).not.toBe(fileContentHash('hello!'));
expect(fileContentHash('hello')).toMatch(/^[a-f0-9]{64}$/);
});
it('handles Buffer input identical to its string form', () => {
const s = 'sentinel';
expect(fileContentHash(Buffer.from(s))).toBe(fileContentHash(s));
});
});
describe('PARSE_CACHE_VERSION', () => {
it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => {
// Looks like "1+1.6.4" — schema bump prefix + actual gitnexus version
expect(PARSE_CACHE_VERSION).toMatch(/^\d+\+\d+\.\d+\.\d+/);
});
});
describe('pruneCache', () => {
it('drops entries whose hashes are not in the used-set', () => {
const cache: ParseCache = {
version: PARSE_CACHE_VERSION,
entries: new Map<string, ParseWorkerResult[]>([
['hash-A', [minimalResult()]],
['hash-B', [minimalResult()]],
['hash-C', [minimalResult()]],
]),
usedKeys: new Set<string>(['hash-A']),
};
const removed = pruneCache(cache, cache.usedKeys);
expect(removed).toBe(2);
expect([...cache.entries.keys()].sort()).toEqual(['hash-A']);
});
it('returns 0 when every entry is in use', () => {
const cache: ParseCache = {
version: PARSE_CACHE_VERSION,
entries: new Map<string, ParseWorkerResult[]>([
['hash-A', [minimalResult()]],
['hash-B', [minimalResult()]],
]),
usedKeys: new Set<string>(['hash-A', 'hash-B']),
};
expect(pruneCache(cache, cache.usedKeys)).toBe(0);
expect(cache.entries.size).toBe(2);
});
});
describe('loadParseCache / saveParseCache (round-trip)', () => {
it('round-trips an empty cache', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
try {
const cache: ParseCache = {
version: PARSE_CACHE_VERSION,
entries: new Map(),
usedKeys: new Set(),
};
await saveParseCache(dir, cache);
const loaded = await loadParseCache(dir);
expect(loaded.version).toBe(PARSE_CACHE_VERSION);
expect(loaded.entries.size).toBe(0);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('returns an empty cache when the file is missing', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
try {
const loaded = await loadParseCache(dir);
expect(loaded.entries.size).toBe(0);
expect(loaded.usedKeys.size).toBe(0);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('returns an empty cache on version mismatch (next-run regen)', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
try {
// Write a cache file with a different version directly
const fs = await import('fs/promises');
await fs.writeFile(
path.join(dir, 'parse-cache.json'),
JSON.stringify({ version: 'foreign-99', entries: { h: [] } }),
'utf-8',
);
const loaded = await loadParseCache(dir);
expect(loaded.entries.size).toBe(0); // mismatch → empty
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('returns an empty cache on corrupt JSON', async () => {
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
try {
const fs = await import('fs/promises');
await fs.writeFile(path.join(dir, 'parse-cache.json'), '{not-json', 'utf-8');
const loaded = await loadParseCache(dir);
expect(loaded.entries.size).toBe(0);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('round-trips Map and Set values through the JSON replacer/reviver', async () => {
// ParsedFile.scopes[*].typeBindings is a ReadonlyMap<string, TypeRef>.
// Without the replacer/reviver pair, JSON.stringify collapses Maps to
// {} and downstream code that does .get() / iterates entries crashes
// with "is not iterable". This test pins the round-trip behaviour.
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
try {
const innerMap = new Map<string, string>([
['k1', 'v1'],
['k2', 'v2'],
]);
const innerSet = new Set<string>(['s1', 's2']);
// Stash the live Map/Set inside a synthetic ParseWorkerResult — we
// only need the serializer to traverse them. Casting to bypass the
// strict shape isn't a problem here: this test is about JSON
// round-tripping of arbitrary nested Map/Set values, not full
// ParseWorkerResult contents.
const fake = minimalResult({
parsedFiles: [
{
filePath: 't.ts',
// Cast through unknown to satisfy the readonly Scope shape
// while still smuggling a live Map into the serializer's
// traversal path — see comment block above.
scopes: [{ id: 's1', typeBindings: innerMap, extras: innerSet }],
} as unknown as ParseWorkerResult['parsedFiles'][number],
],
});
const cache: ParseCache = {
version: PARSE_CACHE_VERSION,
entries: new Map<string, ParseWorkerResult[]>([['chunk-h', [fake]]]),
usedKeys: new Set(['chunk-h']),
};
await saveParseCache(dir, cache);
const loaded = await loadParseCache(dir);
const reloaded = loaded.entries.get('chunk-h')?.[0];
expect(reloaded).toBeDefined();
const scope = (reloaded as ParseWorkerResult).parsedFiles[0]?.scopes[0] as unknown as {
typeBindings?: unknown;
extras?: unknown;
};
expect(scope.typeBindings).toBeInstanceOf(Map);
expect((scope.typeBindings as Map<string, string>).get('k1')).toBe('v1');
expect((scope.typeBindings as Map<string, string>).size).toBe(2);
expect(scope.extras).toBeInstanceOf(Set);
expect((scope.extras as Set<string>).has('s2')).toBe(true);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
});

View file

@ -0,0 +1,105 @@
import { describe, it, expect } from 'vitest';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
import { extractChangedSubgraph } from '../../src/core/incremental/subgraph-extract.js';
import type { GraphNode, GraphRelationship } from 'gitnexus-shared';
const fileNode = (
id: string,
filePath: string,
label: GraphNode['label'] = 'Function',
): GraphNode => ({
id,
label,
properties: { name: id, filePath },
});
const wideNode = (id: string, label: GraphNode['label']): GraphNode => ({
id,
label,
properties: { name: id },
});
const rel = (
id: string,
type: GraphRelationship['type'],
src: string,
dst: string,
): GraphRelationship => ({
id,
type,
sourceId: src,
targetId: dst,
confidence: 1,
reason: 't',
});
describe('extractChangedSubgraph', () => {
it('keeps file nodes whose filePath is in the writable set', () => {
const g = createKnowledgeGraph();
g.addNode(fileNode('Function:a.ts:foo', 'a.ts'));
g.addNode(fileNode('Function:b.ts:bar', 'b.ts'));
g.addNode(fileNode('Function:c.ts:baz', 'c.ts'));
const sub = extractChangedSubgraph(g, new Set(['a.ts', 'c.ts']));
expect(sub.nodeCount).toBe(2);
expect(sub.getNode('Function:a.ts:foo')).toBeDefined();
expect(sub.getNode('Function:c.ts:baz')).toBeDefined();
expect(sub.getNode('Function:b.ts:bar')).toBeUndefined();
});
it('always keeps Community and Process nodes (regenerated graph-wide)', () => {
const g = createKnowledgeGraph();
g.addNode(fileNode('Function:a.ts:foo', 'a.ts'));
g.addNode(wideNode('comm_1', 'Community'));
g.addNode(wideNode('proc_1', 'Process'));
// Empty writable set: only graph-wide nodes survive
const sub = extractChangedSubgraph(g, new Set());
expect(sub.getNode('Function:a.ts:foo')).toBeUndefined();
expect(sub.getNode('comm_1')).toBeDefined();
expect(sub.getNode('proc_1')).toBeDefined();
});
it('keeps edges where at least one endpoint is in the writable subgraph', () => {
const g = createKnowledgeGraph();
g.addNode(fileNode('Function:a.ts:foo', 'a.ts'));
g.addNode(fileNode('Function:b.ts:bar', 'b.ts'));
g.addNode(fileNode('Function:c.ts:baz', 'c.ts'));
g.addRelationship(rel('r-ab', 'CALLS', 'Function:a.ts:foo', 'Function:b.ts:bar'));
g.addRelationship(rel('r-bc', 'CALLS', 'Function:b.ts:bar', 'Function:c.ts:baz'));
g.addRelationship(rel('r-ac', 'CALLS', 'Function:a.ts:foo', 'Function:c.ts:baz'));
const sub = extractChangedSubgraph(g, new Set(['a.ts']));
// r-ab: src in a.ts (writable) → kept
// r-ac: src in a.ts (writable) → kept
// r-bc: src in b.ts, dst in c.ts (both unchanged) → dropped
const rels = [...sub.iterRelationships()].map((r) => r.id).sort();
expect(rels).toEqual(['r-ab', 'r-ac']);
});
it('keeps MEMBER_OF edges (Community endpoints are graph-wide)', () => {
const g = createKnowledgeGraph();
g.addNode(fileNode('Function:a.ts:foo', 'a.ts'));
g.addNode(fileNode('Function:b.ts:bar', 'b.ts'));
g.addNode(wideNode('comm_1', 'Community'));
// Both functions are members of the same community
g.addRelationship(rel('r-a-comm', 'MEMBER_OF', 'Function:a.ts:foo', 'comm_1'));
g.addRelationship(rel('r-b-comm', 'MEMBER_OF', 'Function:b.ts:bar', 'comm_1'));
const sub = extractChangedSubgraph(g, new Set(['a.ts']));
// r-a-comm: src in a.ts (writable) — kept
// r-b-comm: src in b.ts (NOT writable) but dst is graph-wide (Community is in writable subgraph) — kept
const rels = [...sub.iterRelationships()].map((r) => r.id).sort();
expect(rels).toEqual(['r-a-comm', 'r-b-comm']);
});
it('produces an empty subgraph when no nodes match', () => {
const g = createKnowledgeGraph();
g.addNode(fileNode('Function:a.ts:foo', 'a.ts'));
g.addRelationship(rel('r1', 'CALLS', 'Function:a.ts:foo', 'Function:a.ts:foo'));
const sub = extractChangedSubgraph(g, new Set(['nonexistent.ts']));
expect(sub.nodeCount).toBe(0);
expect(sub.relationshipCount).toBe(0);
});
});