mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-24 00:51:53 +00:00
Merge branch 'main' into feat/Desktop-app
This commit is contained in:
Sparsh
GitHub
commit
b88200a64a
23 changed files with 2409 additions and 187 deletions
|
|
@ -156,11 +156,16 @@ npx gitnexus analyze
|
|||
If the index previously included embeddings, preserve them by adding `--embeddings`:
|
||||
|
||||
```bash
|
||||
npx gitnexus analyze # basic refresh; preserves any existing embeddings
|
||||
npx gitnexus analyze # incremental by default; preserves embeddings
|
||||
npx gitnexus analyze --force # full rebuild from scratch (opt out of incremental)
|
||||
npx gitnexus analyze --embeddings # also generate embeddings for new/changed nodes
|
||||
npx gitnexus analyze --drop-embeddings # explicit opt-in to wipe existing embeddings
|
||||
```
|
||||
|
||||
`analyze` runs **incrementally by default**. The pipeline still parses every file every run (cross-file resolution requires it), but tree-sitter parsing is **served from a content-addressed cache** at `.gitnexus/parse-cache.json` for chunks whose file contents haven't changed since the last run. Only changed-file rows (and their importers) are rewritten in LadybugDB; unchanged-file rows are preserved. Output is byte-equivalent to a full rebuild. Pass `--force` to wipe and re-index from scratch (e.g., to recover from a corrupt index, or after upgrading GitNexus).
|
||||
|
||||
The parse cache key is **content-addressed and version-tagged**: it survives `--force` runs, and is automatically invalidated by a `gitnexus` package upgrade (so a new tree-sitter grammar doesn't silently replay stale parse output). Safe to delete `.gitnexus/parse-cache.json` at any time — it'll be rebuilt on the next analyze.
|
||||
|
||||
Check `.gitnexus/meta.json` `stats.embeddings` (0 = none). A plain `analyze` no longer drops existing vectors — pass `--drop-embeddings` to wipe.
|
||||
|
||||
> Claude Code: PostToolUse hook detects a stale index after `git commit` and `git merge` and prompts the agent to run `analyze`. The hook does not invoke `analyze` itself.
|
||||
|
|
|
|||
|
|
@ -30,9 +30,15 @@ Format: **Trigger → Instruction → Reason**. Append new Signs when the same m
|
|||
### Stale graph after edits
|
||||
|
||||
- **Trigger:** MCP warns index is behind `HEAD`, or search doesn't match latest commit.
|
||||
- **Do:** `npx gitnexus analyze` (plus `--embeddings` if used).
|
||||
- **Do:** `npx gitnexus analyze` (plus `--embeddings` if used). Runs incrementally by default — the pipeline parses every file every run (cross-file resolution requires it), but tree-sitter dispatch is skipped for unchanged file chunks via the content-addressed cache, and only changed-file rows (plus their importers, transitively) are rewritten in LadybugDB.
|
||||
- **Why:** Tools query LadybugDB from last analyze; git changes are invisible until re-indexed.
|
||||
|
||||
### Index seems corrupt or "incremental" is misbehaving
|
||||
|
||||
- **Trigger:** `analyze` produces unexpected results, or `meta.json.incrementalInProgress` is set, or the index is in a half-state after a crash.
|
||||
- **Do:** `npx gitnexus analyze --force` to rebuild from scratch. The dirty-flag check forces this automatically when a previous incremental run didn't complete cleanly, but `--force` is the manual escape hatch. Safe to delete `.gitnexus/parse-cache.json` at any time — content-addressed, will be regenerated.
|
||||
- **Why:** Incremental writeback is selective DB row replacement; if the on-disk state is inconsistent for any reason, a full rebuild is the cheapest path back to a known-good index.
|
||||
|
||||
### Embeddings vanished after analyze
|
||||
|
||||
- **Trigger:** Semantic search quality drops; `stats.embeddings` in `meta.json` is 0 after refresh.
|
||||
|
|
|
|||
76
gitnexus/src/core/incremental/shadow-candidates.ts
Normal file
76
gitnexus/src/core/incremental/shadow-candidates.ts
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
/**
|
||||
* Shadow-candidate path derivation for incremental indexing.
|
||||
*
|
||||
* Background — Bugbot review on PR #1479:
|
||||
* queryImporters() on a NEWLY ADDED file returns 0 importers in the
|
||||
* pre-pipeline DB, because the new file's IMPORTS rows haven't been
|
||||
* written yet. But pre-existing files may have IMPORTS edges that
|
||||
* *resolved to a sibling path*, and the newcomer can now steal that
|
||||
* resolution under standard JS/TS module-resolution rules. Without
|
||||
* pulling those pre-existing files into the writable set, their
|
||||
* stale CALLS edges remain pointing at the OLD resolution target.
|
||||
*
|
||||
* Given an added file path, this helper enumerates the pre-existing
|
||||
* file paths whose import-resolution claim the newcomer can steal.
|
||||
* Caller filters the candidates against the prior-run `fileHashes`
|
||||
* map so we only query importers of paths that actually existed.
|
||||
*
|
||||
* Shadow patterns covered (resolution-priority-aware):
|
||||
*
|
||||
* (a) Same basename, different extension —
|
||||
* added `foo/bar.ts` shadows `foo/bar.{tsx,js,jsx,mjs,cjs,d.ts}`.
|
||||
* (b) Bare-file beats directory-style index —
|
||||
* added `foo/bar.ts` shadows `foo/bar/index.{ts,tsx,...}`.
|
||||
* (c) Directory-index beats bare-file —
|
||||
* added `foo/index.ts` shadows `foo.{ts,tsx,...}` (rare but real,
|
||||
* e.g. converting a single-file module into a directory module).
|
||||
*
|
||||
* Resolution-order priority is conservatively wide: we enumerate ALL
|
||||
* common extensions because we don't know which the importer actually
|
||||
* specified, and over-seeding is harmless (extra BFS work, but the
|
||||
* subgraph extract still gates write-back by file membership).
|
||||
*
|
||||
* Cross-platform path separators: candidates are emitted with both `/`
|
||||
* and `\` for shadow pattern (b), since the caller's prior fileHashes
|
||||
* map may use either depending on the OS that wrote it.
|
||||
*/
|
||||
|
||||
const SHADOW_EXTS = ['.d.ts', '.tsx', '.ts', '.jsx', '.js', '.mjs', '.cjs'];
|
||||
|
||||
/**
|
||||
* Enumerate pre-existing paths whose import-resolution `added` can steal.
|
||||
*
|
||||
* @param added — repo-relative path of a newly-added file
|
||||
* @returns deduplicated list of candidate paths (NOT filtered against
|
||||
* any known-files set — caller does that)
|
||||
*/
|
||||
export const shadowCandidatesFor = (added: string): string[] => {
|
||||
const ext = SHADOW_EXTS.find((e) => added.endsWith(e));
|
||||
if (!ext) return [];
|
||||
|
||||
const noExt = added.slice(0, -ext.length);
|
||||
const out = new Set<string>();
|
||||
|
||||
// (a) Same basename, different extension.
|
||||
for (const alt of SHADOW_EXTS) {
|
||||
if (alt !== ext) out.add(noExt + alt);
|
||||
}
|
||||
|
||||
// (b) Bare file beats sibling directory-style index.
|
||||
for (const idx of SHADOW_EXTS) {
|
||||
out.add(`${noExt}/index${idx}`);
|
||||
out.add(`${noExt}\\index${idx}`);
|
||||
}
|
||||
|
||||
// (c) New `foo/index.ext` shadows old `foo.ext`.
|
||||
const idxSuffixSlash = '/index';
|
||||
const idxSuffixBack = '\\index';
|
||||
let dir: string | null = null;
|
||||
if (noExt.endsWith(idxSuffixSlash)) dir = noExt.slice(0, -idxSuffixSlash.length);
|
||||
else if (noExt.endsWith(idxSuffixBack)) dir = noExt.slice(0, -idxSuffixBack.length);
|
||||
if (dir !== null) {
|
||||
for (const alt of SHADOW_EXTS) out.add(dir + alt);
|
||||
}
|
||||
|
||||
return [...out];
|
||||
};
|
||||
123
gitnexus/src/core/incremental/subgraph-extract.ts
Normal file
123
gitnexus/src/core/incremental/subgraph-extract.ts
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
/**
|
||||
* Subgraph extraction for incremental DB writeback.
|
||||
*
|
||||
* Given the FULL ctx.graph produced by the pipeline (all files parsed,
|
||||
* all phases run) and the set of file paths whose DB rows must be
|
||||
* replaced, produce a smaller KnowledgeGraph that contains:
|
||||
*
|
||||
* - Every node whose `properties.filePath` is in `toWriteSet`.
|
||||
* - Every graph-wide node (Community, Process) — these are regenerated
|
||||
* each run by the communities/processes phases and must be fully
|
||||
* rewritten.
|
||||
* - Every relationship where AT LEAST ONE endpoint is in the writable
|
||||
* set above. Relationships entirely between unchanged-file nodes
|
||||
* are skipped — their rows are still in the DB and re-inserting
|
||||
* them would PK-conflict at COPY time.
|
||||
*
|
||||
* The resulting subgraph is what gets passed to `loadGraphToLbug` after
|
||||
* the orchestrator has deleted the corresponding DB rows. Hydrated
|
||||
* unchanged-file rows are never touched in the DB.
|
||||
*
|
||||
* # Cross-file edge consistency (Finding 1)
|
||||
*
|
||||
* `extractChangedSubgraph` intentionally does NOT expand the set it is
|
||||
* given — expansion is the orchestrator's job, so the SAME expanded set
|
||||
* can be fed to both `deleteNodesForFile` and this function (asymmetry
|
||||
* between the delete set and the write set silently corrupts the DB).
|
||||
* `computeEffectiveWriteSet` below performs the boundary-crossing 1-hop
|
||||
* walk; the orchestrator composes it with its importer-BFS expansion and
|
||||
* passes the result here.
|
||||
*
|
||||
* Why the 1-hop walk is needed: consider a barrel re-export change —
|
||||
* file C (a barrel) shifts `export { foo } from './b'` to
|
||||
* `export { foo } from './d'`. After scope resolution, file A's CALLS
|
||||
* edge to `foo` resolves to D instead of B, even though A's content is
|
||||
* byte-for-byte identical:
|
||||
*
|
||||
* - Old A→B edge survives in DB (neither A nor B is changed → not deleted)
|
||||
* - New A→D edge is missing (neither A nor D in writable set → skipped)
|
||||
*
|
||||
* Pulling the unchanged-side file of every writable-boundary-crossing
|
||||
* edge into the write set fixes both halves: the orchestrator's
|
||||
* `DETACH DELETE` cleans up the stale unchanged-side rows, and the new
|
||||
* cross-file edges land because at least one endpoint is now writable.
|
||||
*
|
||||
* Limitation (documented): if a file X *stopped* importing from a
|
||||
* changed file C, X has no edge to C in the new graph, so this 1-hop
|
||||
* walk doesn't catch it. The orchestrator's importer-BFS (which reads
|
||||
* IMPORTS from the pre-pipeline DB) covers that case instead.
|
||||
*/
|
||||
|
||||
import type { GraphNode, GraphRelationship } from 'gitnexus-shared';
|
||||
import { createKnowledgeGraph } from '../graph/graph.js';
|
||||
import type { KnowledgeGraph } from '../graph/types.js';
|
||||
|
||||
const isGraphWide = (label: string): boolean => label === 'Community' || label === 'Process';
|
||||
|
||||
/**
|
||||
* Build a Map<nodeId, filePath> for every File-bound node in the graph.
|
||||
* Graph-wide nodes (Community/Process) have no filePath and are filtered.
|
||||
*/
|
||||
const indexNodeFilePaths = (fullGraph: KnowledgeGraph): Map<string, string> => {
|
||||
const idx = new Map<string, string>();
|
||||
fullGraph.forEachNode((n: GraphNode) => {
|
||||
const fp = n.properties?.filePath as string | undefined;
|
||||
if (fp) idx.set(n.id, fp);
|
||||
});
|
||||
return idx;
|
||||
};
|
||||
|
||||
export const extractChangedSubgraph = (
|
||||
fullGraph: KnowledgeGraph,
|
||||
toWriteSet: ReadonlySet<string>,
|
||||
): KnowledgeGraph => {
|
||||
const sub = createKnowledgeGraph();
|
||||
const writableNodeIds = new Set<string>();
|
||||
|
||||
fullGraph.forEachNode((n: GraphNode) => {
|
||||
const filePath = n.properties?.filePath as string | undefined;
|
||||
const include = (filePath && toWriteSet.has(filePath)) || isGraphWide(n.label);
|
||||
if (include) {
|
||||
sub.addNode(n);
|
||||
writableNodeIds.add(n.id);
|
||||
}
|
||||
});
|
||||
|
||||
fullGraph.forEachRelationship((r: GraphRelationship) => {
|
||||
if (writableNodeIds.has(r.sourceId) || writableNodeIds.has(r.targetId)) {
|
||||
sub.addRelationship(r);
|
||||
}
|
||||
});
|
||||
|
||||
return sub;
|
||||
};
|
||||
|
||||
/**
|
||||
* Public — derive the EFFECTIVE write-set: `toWriteSet` expanded by one
|
||||
* hop along every edge in the new graph that crosses the writable
|
||||
* boundary (one endpoint in a writable file, the other in an unchanged
|
||||
* file). The unchanged-side file is pulled in so its stale rows are
|
||||
* deleted + rewritten in lockstep with the changed side.
|
||||
*
|
||||
* Single pass over the edge list. Does NOT mutate `toWriteSet`. The
|
||||
* orchestrator MUST feed the returned set to both `deleteNodesForFile`
|
||||
* and `extractChangedSubgraph` — feeding the unexpanded set to either
|
||||
* one leaves stale rows or PK-conflicts at COPY time.
|
||||
*/
|
||||
export const computeEffectiveWriteSet = (
|
||||
fullGraph: KnowledgeGraph,
|
||||
toWriteSet: ReadonlySet<string>,
|
||||
): Set<string> => {
|
||||
const nodeFilePaths = indexNodeFilePaths(fullGraph);
|
||||
const expanded = new Set<string>(toWriteSet);
|
||||
fullGraph.forEachRelationship((r: GraphRelationship) => {
|
||||
const sourcePath = nodeFilePaths.get(r.sourceId);
|
||||
const targetPath = nodeFilePaths.get(r.targetId);
|
||||
if (!sourcePath || !targetPath) return; // skip edges to graph-wide nodes
|
||||
const sourceWritable = toWriteSet.has(sourcePath);
|
||||
const targetWritable = toWriteSet.has(targetPath);
|
||||
if (sourceWritable && !targetWritable) expanded.add(targetPath);
|
||||
else if (targetWritable && !sourceWritable) expanded.add(sourcePath);
|
||||
});
|
||||
return expanded;
|
||||
};
|
||||
|
|
@ -42,12 +42,14 @@ import { isVerboseIngestionEnabled } from './utils/verbose.js';
|
|||
import { yieldToEventLoop } from './utils/event-loop.js';
|
||||
import { parseSourceSafe } from '../tree-sitter/safe-parse.js';
|
||||
import {
|
||||
CLASS_CONTAINER_TYPES,
|
||||
FUNCTION_NODE_TYPES,
|
||||
findEnclosingClassId,
|
||||
findEnclosingClassInfo,
|
||||
genericFuncName,
|
||||
inferFunctionLabel,
|
||||
} from './utils/ast-helpers.js';
|
||||
import type { FieldInfo, FieldExtractorContext } from './field-types.js';
|
||||
import type { LanguageProvider } from './language-provider.js';
|
||||
import { typeTagForId, constTagForId, buildCollisionGroups } from './utils/method-props.js';
|
||||
import type { MethodInfo } from './method-types.js';
|
||||
import {
|
||||
|
|
@ -77,6 +79,62 @@ import type { LiteralTypeInferrer } from './type-extractors/types.js';
|
|||
import type { SyntaxNode } from './utils/ast-helpers.js';
|
||||
|
||||
import { logger } from '../logger.js';
|
||||
|
||||
// ── Property-prepass helpers (parity with parse-worker.ts) ──
|
||||
// These mirror the sequential-path equivalents in parse-worker.ts so the main-
|
||||
// thread `processCalls` pre-pass produces byte-identical Property nodes/symbols
|
||||
// to the worker pool. Drift between the two paths breaks the
|
||||
// `incremental ≡ --force` invariant the moment a repo crosses the worker
|
||||
// threshold between runs.
|
||||
|
||||
/** Walk up to the nearest enclosing class/struct/interface AST node. */
|
||||
const findEnclosingClassNode = (node: SyntaxNode): SyntaxNode | null => {
|
||||
let current = node.parent;
|
||||
while (current) {
|
||||
if (CLASS_CONTAINER_TYPES.has(current.type)) return current;
|
||||
current = current.parent;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
/** No-op SymbolTable stub for FieldExtractorContext — matches parse-worker. */
|
||||
const NOOP_SYMBOL_TABLE: SymbolTableReader = {
|
||||
lookupExact: () => undefined,
|
||||
lookupExactFull: () => undefined,
|
||||
lookupExactAll: () => [],
|
||||
lookupCallableByName: () => [],
|
||||
getFiles: () => [][Symbol.iterator](),
|
||||
getStats: () => ({ fileCount: 0 }),
|
||||
};
|
||||
|
||||
/**
|
||||
* Extract (and cache) field info for a class node. Cache is passed in so it
|
||||
* stays scoped to a single `processCalls` invocation rather than leaking
|
||||
* across analyze runs (worker uses module-level caching because each worker
|
||||
* process is short-lived; the main thread is not).
|
||||
*
|
||||
* Cache key is `${filePath}:${classNode.startIndex}` — startIndex alone is a
|
||||
* per-file byte offset, so almost every Ruby/Python file's leading class lands
|
||||
* at byte 0 and would collide across files in the shared map.
|
||||
*/
|
||||
const getFieldInfo = (
|
||||
classNode: SyntaxNode,
|
||||
provider: LanguageProvider,
|
||||
context: FieldExtractorContext,
|
||||
cache: Map<string, Map<string, FieldInfo>>,
|
||||
): Map<string, FieldInfo> | undefined => {
|
||||
if (!provider.fieldExtractor) return undefined;
|
||||
const cacheKey = `${context.filePath}:${classNode.startIndex}`;
|
||||
const cached = cache.get(cacheKey);
|
||||
if (cached) return cached;
|
||||
const result = provider.fieldExtractor.extract(classNode, context);
|
||||
if (!result?.fields?.length) return undefined;
|
||||
const map = new Map<string, FieldInfo>();
|
||||
for (const field of result.fields) map.set(field.name, field);
|
||||
cache.set(cacheKey, map);
|
||||
return map;
|
||||
};
|
||||
|
||||
/** Per-file resolved type bindings for exported symbols.
|
||||
* Populated during call processing, consumed by Phase 14 re-resolution pass. */
|
||||
export type ExportedTypeMap = Map<string, Map<string, string>>;
|
||||
|
|
@ -860,6 +918,120 @@ export const processCalls = async (
|
|||
prepared.push({ file, language, provider, tree, matches, parentMap, typeEnv });
|
||||
}
|
||||
|
||||
// ── Property-registration pre-pass ──
|
||||
// Register all routed properties (e.g. Ruby attr_accessor) BEFORE the
|
||||
// resolution loop so cross-file field-type lookups (e.g.
|
||||
// `user.address.save → Address#save`) succeed regardless of file
|
||||
// processing order. This MUST stay in lockstep with the equivalent
|
||||
// worker-path block in parse-worker.ts (kind === 'properties') — any
|
||||
// divergence between the two paths breaks the `incremental ≡ --force`
|
||||
// invariant once a repo crosses the worker threshold between runs.
|
||||
const fieldInfoCache = new Map<string, Map<string, FieldInfo>>();
|
||||
for (const { file, language, provider, matches, typeEnv } of prepared) {
|
||||
const callRouter = provider.callRouter;
|
||||
if (!callRouter) continue;
|
||||
matches.forEach((match) => {
|
||||
const captureMap: Record<string, any> = {};
|
||||
match.captures.forEach((c) => (captureMap[c.name] = c.node));
|
||||
if (!captureMap['call']) return;
|
||||
const callNameNode = captureMap['call.name'];
|
||||
if (!callNameNode) return;
|
||||
const routed = callRouter(callNameNode.text, captureMap['call']);
|
||||
if (!routed || routed.kind !== 'properties') return;
|
||||
|
||||
const propEnclosingInfo = findEnclosingClassInfo(
|
||||
captureMap['call'],
|
||||
file.path,
|
||||
provider.resolveEnclosingOwner,
|
||||
);
|
||||
const propEnclosingClassId = propEnclosingInfo?.classId ?? null;
|
||||
|
||||
// Enrich routed properties with FieldExtractor metadata so types
|
||||
// discovered from constructor assignments (e.g. `@address = Address.new`)
|
||||
// are propagated even when the routing payload itself lacks declaredType.
|
||||
let routedFieldMap: Map<string, FieldInfo> | undefined;
|
||||
if (provider.fieldExtractor && typeEnv) {
|
||||
const classNode = findEnclosingClassNode(captureMap['call']);
|
||||
if (classNode) {
|
||||
routedFieldMap = getFieldInfo(
|
||||
classNode,
|
||||
provider,
|
||||
{
|
||||
typeEnv,
|
||||
symbolTable: NOOP_SYMBOL_TABLE,
|
||||
filePath: file.path,
|
||||
language,
|
||||
},
|
||||
fieldInfoCache,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const fileId = generateId('File', file.path);
|
||||
for (const item of routed.items) {
|
||||
const routedFieldInfo = routedFieldMap?.get(item.propName);
|
||||
const propQualifiedName = propEnclosingInfo
|
||||
? `${propEnclosingInfo.className}.${item.propName}`
|
||||
: item.propName;
|
||||
const nodeId = generateId('Property', `${file.path}:${propQualifiedName}`);
|
||||
graph.addNode({
|
||||
id: nodeId,
|
||||
label: 'Property',
|
||||
properties: {
|
||||
name: item.propName,
|
||||
filePath: file.path,
|
||||
startLine: item.startLine,
|
||||
endLine: item.endLine,
|
||||
language,
|
||||
isExported: true,
|
||||
description: item.accessorType,
|
||||
...(item.declaredType
|
||||
? { declaredType: item.declaredType }
|
||||
: routedFieldInfo?.type
|
||||
? { declaredType: routedFieldInfo.type }
|
||||
: {}),
|
||||
...(routedFieldInfo?.visibility !== undefined
|
||||
? { visibility: routedFieldInfo.visibility }
|
||||
: {}),
|
||||
...(routedFieldInfo?.isStatic !== undefined
|
||||
? { isStatic: routedFieldInfo.isStatic }
|
||||
: {}),
|
||||
...(routedFieldInfo?.isReadonly !== undefined
|
||||
? { isReadonly: routedFieldInfo.isReadonly }
|
||||
: {}),
|
||||
},
|
||||
});
|
||||
ctx.model.symbols.add(file.path, item.propName, nodeId, 'Property', {
|
||||
...(propEnclosingClassId ? { ownerId: propEnclosingClassId } : {}),
|
||||
...(item.declaredType
|
||||
? { declaredType: item.declaredType }
|
||||
: routedFieldInfo?.type
|
||||
? { declaredType: routedFieldInfo.type }
|
||||
: {}),
|
||||
});
|
||||
const relId = generateId('DEFINES', `${fileId}->${nodeId}`);
|
||||
graph.addRelationship({
|
||||
id: relId,
|
||||
sourceId: fileId,
|
||||
targetId: nodeId,
|
||||
type: 'DEFINES',
|
||||
confidence: 1.0,
|
||||
reason: '',
|
||||
});
|
||||
if (propEnclosingClassId) {
|
||||
graph.addRelationship({
|
||||
id: generateId('HAS_PROPERTY', `${propEnclosingClassId}->${nodeId}`),
|
||||
sourceId: propEnclosingClassId,
|
||||
targetId: nodeId,
|
||||
type: 'HAS_PROPERTY',
|
||||
confidence: 1.0,
|
||||
reason: '',
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ── Resolution loop: verify constructor bindings and resolve calls ──
|
||||
// The accumulator (if present) is now fully populated from the preparation
|
||||
// loop above, so verifyConstructorBindings sees all provider bindings
|
||||
|
|
@ -930,9 +1102,10 @@ export const processCalls = async (
|
|||
provider,
|
||||
);
|
||||
const srcId = enclosing || generateId('File', file.path);
|
||||
// Defer resolution: Ruby attr_accessor properties are registered during
|
||||
// this same loop, so cross-file lookups fail if the declaring file hasn't
|
||||
// been processed yet. Collect now, resolve after all files are done.
|
||||
// Defer resolution so write-access tracking sees the FINAL graph
|
||||
// state — properties from the pre-pass are present, but receiver-type
|
||||
// resolution can still depend on inference that completes during the
|
||||
// main loop. Resolve after all files have been processed.
|
||||
pendingWrites.push({ receiverTypeName, propertyName, filePath: file.path, srcId });
|
||||
}
|
||||
// Assignment-only capture (no @call sibling): skip the rest of this
|
||||
|
|
@ -1053,47 +1226,8 @@ export const processCalls = async (
|
|||
return;
|
||||
|
||||
case 'properties': {
|
||||
const fileId = generateId('File', file.path);
|
||||
const propEnclosingClassId = findEnclosingClassId(captureMap['call'], file.path);
|
||||
for (const item of routed.items) {
|
||||
const nodeId = generateId('Property', `${file.path}:${item.propName}`);
|
||||
graph.addNode({
|
||||
id: nodeId,
|
||||
label: 'Property',
|
||||
properties: {
|
||||
name: item.propName,
|
||||
filePath: file.path,
|
||||
startLine: item.startLine,
|
||||
endLine: item.endLine,
|
||||
language,
|
||||
isExported: true,
|
||||
description: item.accessorType,
|
||||
},
|
||||
});
|
||||
ctx.model.symbols.add(file.path, item.propName, nodeId, 'Property', {
|
||||
...(propEnclosingClassId ? { ownerId: propEnclosingClassId } : {}),
|
||||
...(item.declaredType ? { declaredType: item.declaredType } : {}),
|
||||
});
|
||||
const relId = generateId('DEFINES', `${fileId}->${nodeId}`);
|
||||
graph.addRelationship({
|
||||
id: relId,
|
||||
sourceId: fileId,
|
||||
targetId: nodeId,
|
||||
type: 'DEFINES',
|
||||
confidence: 1.0,
|
||||
reason: '',
|
||||
});
|
||||
if (propEnclosingClassId) {
|
||||
graph.addRelationship({
|
||||
id: generateId('HAS_PROPERTY', `${propEnclosingClassId}->${nodeId}`),
|
||||
sourceId: propEnclosingClassId,
|
||||
targetId: nodeId,
|
||||
type: 'HAS_PROPERTY',
|
||||
confidence: 1.0,
|
||||
reason: '',
|
||||
});
|
||||
}
|
||||
}
|
||||
// Properties already registered in the pre-pass above.
|
||||
// Skip to avoid duplicate nodes/edges.
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -41,6 +41,24 @@ interface LeidenDetailedResult {
|
|||
modularity: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic PRNG (mulberry32) seed for the vendored Leiden algorithm.
|
||||
* Vendored Leiden defaults `rng: Math.random`, which makes community
|
||||
* assignment non-deterministic across runs. Passing a seeded RNG gives us
|
||||
* reproducible community/modularity output, which is required for the
|
||||
* incremental-indexing equivalence test (incremental ≡ full rebuild).
|
||||
*/
|
||||
const LEIDEN_SEED = 0xc0de;
|
||||
function createSeededRng(seed: number): () => number {
|
||||
let s = seed >>> 0;
|
||||
return () => {
|
||||
s = (s + 0x6d2b79f5) >>> 0;
|
||||
let t = Math.imul(s ^ (s >>> 15), 1 | s);
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// TYPES
|
||||
// ============================================================================
|
||||
|
|
@ -150,6 +168,7 @@ export const processCommunities = async (
|
|||
leiden.detailed(graph, {
|
||||
resolution: isLarge ? 2.0 : 1.0,
|
||||
maxIterations: isLarge ? 3 : 0,
|
||||
rng: createSeededRng(LEIDEN_SEED),
|
||||
}),
|
||||
),
|
||||
new Promise<never>((_, reject) =>
|
||||
|
|
|
|||
|
|
@ -82,6 +82,88 @@ export interface WorkerExtractedData {
|
|||
// Worker-based parallel parsing
|
||||
// ============================================================================
|
||||
|
||||
/**
|
||||
* Merge a list of `ParseWorkerResult`s into the running graph + symbol
|
||||
* table state and produce the chunk-aggregated `WorkerExtractedData`.
|
||||
*
|
||||
* Extracted from `processParsingWithWorkers` so the same merge logic can
|
||||
* be applied to both freshly-parsed worker output AND cached worker
|
||||
* output replayed during incremental analyze. Idempotent on the
|
||||
* accumulator fields (push-only); idempotent on graph if the caller
|
||||
* starts from a clean graph (otherwise duplicate `addNode` calls are
|
||||
* silently no-op'd by `KnowledgeGraph`).
|
||||
*/
|
||||
export const mergeChunkResults = (
|
||||
graph: KnowledgeGraph,
|
||||
symbolTable: SymbolTableWriter,
|
||||
chunkResults: readonly ParseWorkerResult[],
|
||||
): WorkerExtractedData => {
|
||||
const allImports: ExtractedImport[] = [];
|
||||
const allCalls: ExtractedCall[] = [];
|
||||
const allAssignments: ExtractedAssignment[] = [];
|
||||
const allHeritage: ExtractedHeritage[] = [];
|
||||
const allRoutes: ExtractedRoute[] = [];
|
||||
const allFetchCalls: ExtractedFetchCall[] = [];
|
||||
const allDecoratorRoutes: ExtractedDecoratorRoute[] = [];
|
||||
const allToolDefs: ExtractedToolDef[] = [];
|
||||
const allORMQueries: ExtractedORMQuery[] = [];
|
||||
const allConstructorBindings: FileConstructorBindings[] = [];
|
||||
const fileScopeBindingsByFile: FileScopeBindings[] = [];
|
||||
const allParsedFiles: ParsedFile[] = [];
|
||||
|
||||
for (const result of chunkResults) {
|
||||
for (const node of result.nodes) {
|
||||
graph.addNode({
|
||||
id: node.id,
|
||||
label: node.label as NodeLabel,
|
||||
properties: node.properties,
|
||||
});
|
||||
}
|
||||
for (const rel of result.relationships) {
|
||||
graph.addRelationship(rel);
|
||||
}
|
||||
for (const sym of result.symbols) {
|
||||
symbolTable.add(sym.filePath, sym.name, sym.nodeId, sym.type, {
|
||||
parameterCount: sym.parameterCount,
|
||||
requiredParameterCount: sym.requiredParameterCount,
|
||||
parameterTypes: sym.parameterTypes,
|
||||
returnType: sym.returnType,
|
||||
declaredType: sym.declaredType,
|
||||
ownerId: sym.ownerId,
|
||||
qualifiedName: sym.qualifiedName,
|
||||
});
|
||||
}
|
||||
for (const item of result.imports) allImports.push(item);
|
||||
for (const item of result.calls) allCalls.push(item);
|
||||
for (const item of result.assignments) allAssignments.push(item);
|
||||
for (const item of result.heritage) allHeritage.push(item);
|
||||
for (const item of result.routes) allRoutes.push(item);
|
||||
for (const item of result.fetchCalls) allFetchCalls.push(item);
|
||||
for (const item of result.decoratorRoutes) allDecoratorRoutes.push(item);
|
||||
for (const item of result.toolDefs) allToolDefs.push(item);
|
||||
if (result.ormQueries) for (const item of result.ormQueries) allORMQueries.push(item);
|
||||
for (const item of result.constructorBindings) allConstructorBindings.push(item);
|
||||
if (result.fileScopeBindings)
|
||||
for (const item of result.fileScopeBindings) fileScopeBindingsByFile.push(item);
|
||||
if (result.parsedFiles) for (const item of result.parsedFiles) allParsedFiles.push(item);
|
||||
}
|
||||
|
||||
return {
|
||||
imports: allImports,
|
||||
calls: allCalls,
|
||||
assignments: allAssignments,
|
||||
heritage: allHeritage,
|
||||
routes: allRoutes,
|
||||
fetchCalls: allFetchCalls,
|
||||
decoratorRoutes: allDecoratorRoutes,
|
||||
toolDefs: allToolDefs,
|
||||
ormQueries: allORMQueries,
|
||||
constructorBindings: allConstructorBindings,
|
||||
fileScopeBindings: fileScopeBindingsByFile,
|
||||
parsedFiles: allParsedFiles,
|
||||
};
|
||||
};
|
||||
|
||||
const processParsingWithWorkers = async (
|
||||
graph: KnowledgeGraph,
|
||||
files: { path: string; content: string }[],
|
||||
|
|
@ -89,6 +171,14 @@ const processParsingWithWorkers = async (
|
|||
astCache: ASTCache,
|
||||
workerPool: WorkerPool,
|
||||
onFileProgress?: FileProgressCallback,
|
||||
/**
|
||||
* When provided, populated with the raw worker results before merging.
|
||||
* Used by the incremental-indexing parse cache to capture the per-chunk
|
||||
* worker output for caching across runs. The mutation happens in-place
|
||||
* so the caller (parse-impl) can keep a reference. See
|
||||
* `gitnexus/src/storage/parse-cache.ts`.
|
||||
*/
|
||||
outRawResults?: ParseWorkerResult[],
|
||||
): Promise<WorkerExtractedData> => {
|
||||
// Filter to parseable files only
|
||||
const parseableFiles: ParseWorkerInput[] = [];
|
||||
|
|
@ -123,63 +213,16 @@ const processParsingWithWorkers = async (
|
|||
},
|
||||
);
|
||||
|
||||
// Merge results from all workers into graph and symbol table
|
||||
const allImports: ExtractedImport[] = [];
|
||||
const allCalls: ExtractedCall[] = [];
|
||||
const allAssignments: ExtractedAssignment[] = [];
|
||||
const allHeritage: ExtractedHeritage[] = [];
|
||||
const allRoutes: ExtractedRoute[] = [];
|
||||
const allFetchCalls: ExtractedFetchCall[] = [];
|
||||
const allDecoratorRoutes: ExtractedDecoratorRoute[] = [];
|
||||
const allToolDefs: ExtractedToolDef[] = [];
|
||||
const allORMQueries: ExtractedORMQuery[] = [];
|
||||
const allConstructorBindings: FileConstructorBindings[] = [];
|
||||
const fileScopeBindingsByFile: FileScopeBindings[] = [];
|
||||
const allParsedFiles: ParsedFile[] = [];
|
||||
for (const result of chunkResults) {
|
||||
for (const node of result.nodes) {
|
||||
graph.addNode({
|
||||
id: node.id,
|
||||
label: node.label as NodeLabel,
|
||||
properties: node.properties,
|
||||
});
|
||||
}
|
||||
|
||||
for (const rel of result.relationships) {
|
||||
graph.addRelationship(rel);
|
||||
}
|
||||
|
||||
for (const sym of result.symbols) {
|
||||
symbolTable.add(sym.filePath, sym.name, sym.nodeId, sym.type, {
|
||||
parameterCount: sym.parameterCount,
|
||||
requiredParameterCount: sym.requiredParameterCount,
|
||||
parameterTypes: sym.parameterTypes,
|
||||
returnType: sym.returnType,
|
||||
declaredType: sym.declaredType,
|
||||
ownerId: sym.ownerId,
|
||||
qualifiedName: sym.qualifiedName,
|
||||
});
|
||||
}
|
||||
|
||||
for (const item of result.imports) allImports.push(item);
|
||||
for (const item of result.calls) allCalls.push(item);
|
||||
for (const item of result.assignments) allAssignments.push(item);
|
||||
for (const item of result.heritage) allHeritage.push(item);
|
||||
for (const item of result.routes) allRoutes.push(item);
|
||||
for (const item of result.fetchCalls) allFetchCalls.push(item);
|
||||
for (const item of result.decoratorRoutes) allDecoratorRoutes.push(item);
|
||||
for (const item of result.toolDefs) allToolDefs.push(item);
|
||||
if (result.ormQueries) for (const item of result.ormQueries) allORMQueries.push(item);
|
||||
for (const item of result.constructorBindings) allConstructorBindings.push(item);
|
||||
if (result.fileScopeBindings)
|
||||
for (const item of result.fileScopeBindings) fileScopeBindingsByFile.push(item);
|
||||
// RFC #909 Ring 2: aggregate per-file scope artifacts. Tolerant of
|
||||
// workers that don't emit the field yet (older worker builds or
|
||||
// partial rollouts), since the additive contract means undefined =
|
||||
// "this worker produced no ParsedFiles for this chunk".
|
||||
if (result.parsedFiles) for (const item of result.parsedFiles) allParsedFiles.push(item);
|
||||
// Capture the raw chunk results for the incremental parse cache before
|
||||
// merging — the cache stores the unmerged worker output so a future run
|
||||
// can re-merge them into a fresh graph state.
|
||||
if (outRawResults) {
|
||||
for (const r of chunkResults) outRawResults.push(r);
|
||||
}
|
||||
|
||||
// Merge results from all workers into graph and symbol table.
|
||||
const merged = mergeChunkResults(graph, symbolTable, chunkResults);
|
||||
|
||||
// Merge and log skipped languages from workers
|
||||
const skippedLanguages = new Map<string, number>();
|
||||
for (const result of chunkResults) {
|
||||
|
|
@ -196,20 +239,7 @@ const processParsingWithWorkers = async (
|
|||
|
||||
// Final progress
|
||||
onFileProgress?.(total, total, 'done');
|
||||
return {
|
||||
imports: allImports,
|
||||
calls: allCalls,
|
||||
assignments: allAssignments,
|
||||
heritage: allHeritage,
|
||||
routes: allRoutes,
|
||||
fetchCalls: allFetchCalls,
|
||||
decoratorRoutes: allDecoratorRoutes,
|
||||
toolDefs: allToolDefs,
|
||||
ormQueries: allORMQueries,
|
||||
constructorBindings: allConstructorBindings,
|
||||
fileScopeBindings: fileScopeBindingsByFile,
|
||||
parsedFiles: allParsedFiles,
|
||||
};
|
||||
return merged;
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
|
|
@ -732,6 +762,14 @@ export const processParsing = async (
|
|||
scopeTreeCache: ASTCache | undefined,
|
||||
onFileProgress?: FileProgressCallback,
|
||||
workerPool?: WorkerPool,
|
||||
/**
|
||||
* Optional out-parameter for the incremental parse cache. When
|
||||
* provided AND the worker-pool path runs successfully, populated
|
||||
* with the raw `ParseWorkerResult[]` from the workers (pre-merge).
|
||||
* Stays empty for the sequential fallback path (no per-chunk
|
||||
* artifact to cache there). See `gitnexus/src/storage/parse-cache.ts`.
|
||||
*/
|
||||
outRawResults?: ParseWorkerResult[],
|
||||
): Promise<WorkerExtractedData | null> => {
|
||||
let lastProgress = 0;
|
||||
const reportProgress: FileProgressCallback | undefined = onFileProgress
|
||||
|
|
@ -759,6 +797,7 @@ export const processParsing = async (
|
|||
astCache,
|
||||
workerPool,
|
||||
reportProgress,
|
||||
outRawResults,
|
||||
);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
|
|
|
|||
|
|
@ -17,7 +17,10 @@ import {
|
|||
enrichExportedTypeMap,
|
||||
type BindingEntry,
|
||||
} from '../binding-accumulator.js';
|
||||
import { processParsing } from '../parsing-processor.js';
|
||||
import { processParsing, mergeChunkResults } from '../parsing-processor.js';
|
||||
import { fileContentHash, computeChunkHash } from '../../../storage/parse-cache.js';
|
||||
import type { ParseWorkerResult } from '../workers/parse-worker.js';
|
||||
import type { WorkerExtractedData } from '../parsing-processor.js';
|
||||
import {
|
||||
processImports,
|
||||
processImportsFromExtracted,
|
||||
|
|
@ -72,8 +75,21 @@ import { extractORMQueriesInline } from './orm-extraction.js';
|
|||
import { logger } from '../../logger.js';
|
||||
// ── Constants ──────────────────────────────────────────────────────────────
|
||||
|
||||
/** Max bytes of source content to load per parse chunk. */
|
||||
const CHUNK_BYTE_BUDGET = 20 * 1024 * 1024; // 20MB
|
||||
/** Max bytes of source content to load per parse chunk.
|
||||
*
|
||||
* Memory bound for the worker pool dispatch + a granularity knob for
|
||||
* the parse cache. A single file change invalidates only its enclosing
|
||||
* chunk, so smaller budgets → finer-grained invalidation.
|
||||
*
|
||||
* Override via GITNEXUS_CHUNK_BYTE_BUDGET (bytes) — the default of 2MB
|
||||
* gives a useful invalidation floor (~1/N chunks on a multi-MB repo)
|
||||
* while keeping worker dispatch overhead under 5% on cold runs.
|
||||
*/
|
||||
const CHUNK_BYTE_BUDGET = (() => {
|
||||
const env = Number(process.env.GITNEXUS_CHUNK_BYTE_BUDGET);
|
||||
if (Number.isFinite(env) && env > 0) return env;
|
||||
return 2 * 1024 * 1024;
|
||||
})();
|
||||
|
||||
// ── Main parse + resolve function ──────────────────────────────────────────
|
||||
|
||||
|
|
@ -119,6 +135,11 @@ export async function runChunkedParseAndResolve(
|
|||
* source. See plan
|
||||
* docs/plans/2026-04-20-002-perf-parse-heritage-mro-plan.md (Unit 4). */
|
||||
scopeTreeCache: ASTCache;
|
||||
/** Worker-produced ParsedFile artifacts aggregated across chunks.
|
||||
* Threaded into scope-resolution as a re-extract cache so the warm-
|
||||
* cache analyze run can skip the dominant `extractParsedFile` cost
|
||||
* (otherwise ~58s on a 1000-file repo). */
|
||||
parsedFiles: import('gitnexus-shared').ParsedFile[];
|
||||
}> {
|
||||
const ctx = createResolutionContext();
|
||||
const symbolTable = ctx.model.symbols;
|
||||
|
|
@ -142,6 +163,15 @@ export async function runChunkedParseAndResolve(
|
|||
);
|
||||
}
|
||||
|
||||
// Sort parseableScanned alphabetically for stable chunk membership
|
||||
// across runs (Finding 4). Without this, filesystem-scan order can
|
||||
// shift between runs (notably on macOS APFS where directory entry
|
||||
// order can change after modifications) — different files in the
|
||||
// same chunk → different chunk hash → cache miss even when no file
|
||||
// content changed. The cache also becomes platform-specific: a
|
||||
// Linux-built cache misses on macOS for the same repo.
|
||||
parseableScanned.sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0));
|
||||
|
||||
const totalParseable = parseableScanned.length;
|
||||
|
||||
if (totalParseable === 0) {
|
||||
|
|
@ -271,6 +301,20 @@ export async function runChunkedParseAndResolve(
|
|||
const deferredWorkerHeritage: ExtractedHeritage[] = [];
|
||||
const deferredConstructorBindings: FileConstructorBindings[] = [];
|
||||
const deferredAssignments: ExtractedAssignment[] = [];
|
||||
// Aggregated per-file ParsedFile artifacts produced by workers' calls
|
||||
// to `extractParsedFile`. Threaded through to the scope-resolution
|
||||
// phase so it can SKIP its own re-extraction on cache hits — this is
|
||||
// the second-half of the parse-cache speedup since scope-resolution's
|
||||
// re-parse otherwise dominates the warm-cache wall-clock time.
|
||||
const allParsedFiles: import('gitnexus-shared').ParsedFile[] = [];
|
||||
|
||||
// Incremental parse cache (Option B): chunk-level content-addressed.
|
||||
// When the chunk's (filePath, content-hash) signature matches a prior
|
||||
// run's, replay the cached ParseWorkerResult[] instead of dispatching
|
||||
// to workers. See gitnexus/src/storage/parse-cache.ts.
|
||||
const parseCache = options?.parseCache;
|
||||
let chunkCacheHits = 0;
|
||||
let chunkCacheMisses = 0;
|
||||
|
||||
try {
|
||||
for (let chunkIdx = 0; chunkIdx < numChunks; chunkIdx++) {
|
||||
|
|
@ -281,29 +325,89 @@ export async function runChunkedParseAndResolve(
|
|||
.filter((p) => chunkContents.has(p))
|
||||
.map((p) => ({ path: p, content: chunkContents.get(p)! }));
|
||||
|
||||
const chunkWorkerData = await processParsing(
|
||||
graph,
|
||||
chunkFiles,
|
||||
symbolTable,
|
||||
astCache,
|
||||
scopeTreeCache,
|
||||
(current, _total, filePath) => {
|
||||
const globalCurrent = filesParsedSoFar + current;
|
||||
const parsingProgress = 20 + (globalCurrent / totalParseable) * 62;
|
||||
onProgress({
|
||||
phase: 'parsing',
|
||||
percent: Math.round(parsingProgress),
|
||||
message: `Parsing chunk ${chunkIdx + 1}/${numChunks}...`,
|
||||
detail: filePath,
|
||||
stats: {
|
||||
filesProcessed: globalCurrent,
|
||||
totalFiles: totalParseable,
|
||||
nodesCreated: graph.nodeCount,
|
||||
},
|
||||
});
|
||||
},
|
||||
workerPool,
|
||||
);
|
||||
// Compute the chunk's content-hash signature (if cache available).
|
||||
let chunkHash: string | null = null;
|
||||
if (parseCache) {
|
||||
const entries = chunkFiles.map((f) => ({
|
||||
filePath: f.path,
|
||||
contentHash: fileContentHash(f.content),
|
||||
}));
|
||||
chunkHash = computeChunkHash(entries);
|
||||
}
|
||||
|
||||
let chunkWorkerData: WorkerExtractedData | null;
|
||||
const cachedRaw = chunkHash ? parseCache!.entries.get(chunkHash) : undefined;
|
||||
|
||||
// Track every chunk hash we touched so the orchestrator can
|
||||
// prune stale entries (chunks whose composition no longer
|
||||
// corresponds to a live chunk in the current scan) before saving.
|
||||
if (parseCache && chunkHash) parseCache.usedKeys.add(chunkHash);
|
||||
|
||||
if (cachedRaw && cachedRaw.length > 0) {
|
||||
// Cache hit: replay the cached worker output through the same
|
||||
// merge logic the live worker path uses.
|
||||
chunkCacheHits++;
|
||||
chunkWorkerData = mergeChunkResults(graph, symbolTable, cachedRaw);
|
||||
if (isDev) {
|
||||
logger.info(
|
||||
`📦 parse-cache HIT: chunk ${chunkIdx + 1}/${numChunks} (${chunkFiles.length} files, ${chunkHash!.slice(0, 8)})`,
|
||||
);
|
||||
}
|
||||
// Progress update so UI advances even on a cache hit.
|
||||
const cachedFiles = chunkFiles.length;
|
||||
onProgress({
|
||||
phase: 'parsing',
|
||||
percent: Math.round(20 + ((filesParsedSoFar + cachedFiles) / totalParseable) * 62),
|
||||
message: `Parsing chunk ${chunkIdx + 1}/${numChunks} (cache)...`,
|
||||
stats: {
|
||||
filesProcessed: filesParsedSoFar + cachedFiles,
|
||||
totalFiles: totalParseable,
|
||||
nodesCreated: graph.nodeCount,
|
||||
},
|
||||
});
|
||||
} else {
|
||||
// Cache miss: dispatch to workers, capture the raw results, store
|
||||
// them under the chunk hash for the next run.
|
||||
chunkCacheMisses++;
|
||||
const rawResults: ParseWorkerResult[] = [];
|
||||
chunkWorkerData = await processParsing(
|
||||
graph,
|
||||
chunkFiles,
|
||||
symbolTable,
|
||||
astCache,
|
||||
scopeTreeCache,
|
||||
(current, _total, filePath) => {
|
||||
const globalCurrent = filesParsedSoFar + current;
|
||||
const parsingProgress = 20 + (globalCurrent / totalParseable) * 62;
|
||||
onProgress({
|
||||
phase: 'parsing',
|
||||
percent: Math.round(parsingProgress),
|
||||
message: `Parsing chunk ${chunkIdx + 1}/${numChunks}...`,
|
||||
detail: filePath,
|
||||
stats: {
|
||||
filesProcessed: globalCurrent,
|
||||
totalFiles: totalParseable,
|
||||
nodesCreated: graph.nodeCount,
|
||||
},
|
||||
});
|
||||
},
|
||||
workerPool,
|
||||
// Capture raw results only when we have a cache to write to —
|
||||
// otherwise we'd retain extra arrays for nothing.
|
||||
parseCache && chunkHash ? rawResults : undefined,
|
||||
);
|
||||
// Persist the raw results for this chunk hash. Sequential path
|
||||
// doesn't populate rawResults (it writes directly to graph), so
|
||||
// small repos without worker pool simply don't cache. That's fine.
|
||||
if (parseCache && chunkHash && rawResults.length > 0) {
|
||||
parseCache.entries.set(chunkHash, rawResults);
|
||||
if (isDev) {
|
||||
logger.info(
|
||||
`📦 parse-cache MISS+store: chunk ${chunkIdx + 1}/${numChunks} (${chunkFiles.length} files, ${chunkHash.slice(0, 8)})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const chunkBasePercent = 20 + (filesParsedSoFar / totalParseable) * 62;
|
||||
|
||||
|
|
@ -349,6 +453,12 @@ export async function runChunkedParseAndResolve(
|
|||
for (const item of chunkWorkerData.heritage) deferredWorkerHeritage.push(item);
|
||||
for (const item of chunkWorkerData.constructorBindings)
|
||||
deferredConstructorBindings.push(item);
|
||||
// Aggregate worker-produced ParsedFile artifacts so scope-
|
||||
// resolution can use them as a re-extraction cache (skips its
|
||||
// own tree-sitter re-parse on warm runs).
|
||||
if (chunkWorkerData.parsedFiles?.length) {
|
||||
for (const item of chunkWorkerData.parsedFiles) allParsedFiles.push(item);
|
||||
}
|
||||
if (chunkWorkerData.assignments?.length) {
|
||||
for (const item of chunkWorkerData.assignments) deferredAssignments.push(item);
|
||||
}
|
||||
|
|
@ -422,6 +532,12 @@ export async function runChunkedParseAndResolve(
|
|||
astCache.clear();
|
||||
}
|
||||
|
||||
if (isDev && parseCache && (chunkCacheHits > 0 || chunkCacheMisses > 0)) {
|
||||
logger.info(
|
||||
`📦 parse-cache summary: ${chunkCacheHits} chunk hit(s), ${chunkCacheMisses} miss(es) across ${numChunks} chunk(s)`,
|
||||
);
|
||||
}
|
||||
|
||||
const fullWorkerHeritageMap =
|
||||
deferredWorkerHeritage.length > 0
|
||||
? buildHeritageMap(deferredWorkerHeritage, ctx, getHeritageStrategyForLanguage)
|
||||
|
|
@ -621,5 +737,12 @@ export async function runChunkedParseAndResolve(
|
|||
// chunk-local `astCache` above is intentionally NOT exposed
|
||||
// because parse-impl clears it between chunks.
|
||||
scopeTreeCache,
|
||||
// Per-file ParsedFile artifacts produced by workers' calls to
|
||||
// `extractParsedFile`. Empty when only the sequential path ran
|
||||
// (sequential doesn't go through the worker, and extracts ParsedFile
|
||||
// inline rather than emitting it). Consumed by scope-resolution as
|
||||
// a re-extraction cache: when the file's ParsedFile is here,
|
||||
// scope-resolution skips its own `extractParsedFile` call.
|
||||
parsedFiles: allParsedFiles,
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import type { PipelinePhase, PipelineContext, PhaseResult } from './types.js';
|
|||
import { getPhaseOutput } from './types.js';
|
||||
import type { StructureOutput } from './structure.js';
|
||||
import type { BindingAccumulator } from '../binding-accumulator.js';
|
||||
import type { ParsedFile } from 'gitnexus-shared';
|
||||
import type {
|
||||
ExtractedFetchCall,
|
||||
ExtractedRoute,
|
||||
|
|
@ -81,6 +82,19 @@ export interface ParseOutput {
|
|||
* `scopeTreeCache.clear()` after its extract loop finishes.
|
||||
*/
|
||||
readonly scopeTreeCache: ASTCache;
|
||||
/**
|
||||
* Per-file `ParsedFile` artifacts produced by workers' calls to
|
||||
* `extractParsedFile`. Threaded through to `scopeResolutionPhase`
|
||||
* as a re-extraction cache: when a file's ParsedFile is present here,
|
||||
* scope-resolution can skip its own `extractParsedFile` (which would
|
||||
* otherwise re-parse the file with tree-sitter on the main thread,
|
||||
* costing ~58s on a 1000-file repo).
|
||||
*
|
||||
* Empty for files that went through the sequential parse fallback —
|
||||
* sequential doesn't emit ParsedFile artifacts; scope-resolution
|
||||
* falls back to a fresh extract for those.
|
||||
*/
|
||||
readonly parsedFiles: readonly ParsedFile[];
|
||||
}
|
||||
|
||||
export const parsePhase: PipelinePhase<ParseOutput> = {
|
||||
|
|
|
|||
|
|
@ -55,6 +55,19 @@ export interface PipelineOptions {
|
|||
minFiles?: number;
|
||||
minBytes?: number;
|
||||
};
|
||||
/**
|
||||
* Incremental-indexing parse cache. When provided:
|
||||
* - The parse phase looks up each chunk's content hash in
|
||||
* `parseCache.entries`. On hit, it replays the cached
|
||||
* `ParseWorkerResult[]` instead of dispatching to workers.
|
||||
* - On miss, it runs the workers as today and stores the new
|
||||
* results in `parseCache.entries` keyed by chunk hash.
|
||||
* The caller (`run-analyze.ts`) is responsible for loading the cache
|
||||
* before the pipeline runs and persisting it after. Cache survives
|
||||
* `--force` because keys are content-addressed.
|
||||
* See `gitnexus/src/storage/parse-cache.ts`.
|
||||
*/
|
||||
parseCache?: import('../../storage/parse-cache.js').ParseCache;
|
||||
}
|
||||
|
||||
// ── Phase registry ─────────────────────────────────────────────────────────
|
||||
|
|
|
|||
|
|
@ -93,13 +93,25 @@ export const scopeResolutionPhase: PipelinePhase<ScopeResolutionOutput> = {
|
|||
// Worker-mode parses leave the cache empty for those files; they
|
||||
// also fall back to a fresh parse — no correctness impact.
|
||||
const parseOutput = getPhaseOutput<ParseOutput>(deps, 'parse');
|
||||
const { scopeTreeCache, resolutionContext } = parseOutput;
|
||||
const { scopeTreeCache, resolutionContext, parsedFiles: workerParsedFiles } = parseOutput;
|
||||
// SemanticModel populated during `parse`: scope-resolution consumes
|
||||
// TypeRegistry / MethodRegistry / SymbolTable lookups instead of
|
||||
// rebuilding parallel indexes. See ARCHITECTURE.md § "Semantic-model
|
||||
// source of truth".
|
||||
const model = resolutionContext.model;
|
||||
|
||||
// Build a per-file lookup of ParsedFile artifacts the workers (or
|
||||
// sequential extracts) already produced. Threading this into
|
||||
// `runScopeResolution` lets the per-language extract loop short-
|
||||
// circuit `extractParsedFile` — the dominant cost on the warm-cache
|
||||
// path, since workers can't return tree-sitter Trees across the
|
||||
// MessageChannel and scope-resolution would otherwise re-parse
|
||||
// every file from scratch on the main thread.
|
||||
const preExtractedByPath = new Map<string, import('gitnexus-shared').ParsedFile>();
|
||||
for (const pf of workerParsedFiles) {
|
||||
preExtractedByPath.set(pf.filePath, pf);
|
||||
}
|
||||
|
||||
let totalFiles = 0;
|
||||
let totalImports = 0;
|
||||
let totalRefs = 0;
|
||||
|
|
@ -143,6 +155,7 @@ export const scopeResolutionPhase: PipelinePhase<ScopeResolutionOutput> = {
|
|||
files,
|
||||
treeCache: scopeTreeCache,
|
||||
resolutionConfig,
|
||||
preExtractedParsedFiles: preExtractedByPath,
|
||||
onWarn: (msg) => {
|
||||
if (isSemanticModelValidatorEnabled()) {
|
||||
logger.warn(`[scope-resolution:${lang}] ${msg}`);
|
||||
|
|
|
|||
|
|
@ -72,6 +72,22 @@ interface RunScopeResolutionInput {
|
|||
* provider doesn't supply a config loader.
|
||||
*/
|
||||
readonly resolutionConfig?: unknown;
|
||||
/**
|
||||
* Pre-extracted ParsedFile artifacts keyed by file path. When a
|
||||
* file is present here, the extract loop reuses it directly and
|
||||
* skips `extractParsedFile` (which would re-parse the file with
|
||||
* tree-sitter on the main thread). Only files matching the
|
||||
* provider's language are honored — the loop verifies this
|
||||
* implicitly by language filter at the call-site (scopeResolution
|
||||
* phase).
|
||||
*
|
||||
* Worker-mode parses produce these ParsedFile artifacts as a side
|
||||
* effect of `extractParsedFile` running inside the worker; threading
|
||||
* them here is what lets the warm-cache analyze run skip the ~58s
|
||||
* scope-resolution re-parse loop on a multi-thousand-file repo.
|
||||
* Cache miss is safe — falls back to fresh extract.
|
||||
*/
|
||||
readonly preExtractedParsedFiles?: ReadonlyMap<string, ParsedFile>;
|
||||
}
|
||||
|
||||
interface RunScopeResolutionStats {
|
||||
|
|
@ -104,22 +120,37 @@ export function runScopeResolution(
|
|||
const parsedFiles: ParsedFile[] = [];
|
||||
let filesSkipped = 0;
|
||||
const treeCache = input.treeCache;
|
||||
const preExtracted = input.preExtractedParsedFiles;
|
||||
let preExtractedHits = 0;
|
||||
for (const file of files) {
|
||||
const cachedTree = treeCache?.get(file.path);
|
||||
const parsed = extractParsedFile(
|
||||
provider.languageProvider,
|
||||
file.content,
|
||||
file.path,
|
||||
onWarn,
|
||||
cachedTree,
|
||||
);
|
||||
let parsed: ParsedFile | undefined;
|
||||
// Fast path: a worker (during the parse phase) already produced a
|
||||
// ParsedFile for this file via `extractParsedFile`. Reuse it
|
||||
// directly — skips a tree-sitter re-parse on the main thread.
|
||||
if (preExtracted !== undefined) {
|
||||
parsed = preExtracted.get(file.path);
|
||||
if (parsed !== undefined) preExtractedHits++;
|
||||
}
|
||||
if (parsed === undefined) {
|
||||
filesSkipped++;
|
||||
continue;
|
||||
const cachedTree = treeCache?.get(file.path);
|
||||
parsed = extractParsedFile(
|
||||
provider.languageProvider,
|
||||
file.content,
|
||||
file.path,
|
||||
onWarn,
|
||||
cachedTree,
|
||||
);
|
||||
if (parsed === undefined) {
|
||||
filesSkipped++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
provider.populateOwners(parsed);
|
||||
parsedFiles.push(parsed);
|
||||
}
|
||||
if (PROF && preExtracted !== undefined) {
|
||||
logger.warn(`[scope-resolution prof] pre-extracted hits: ${preExtractedHits}/${files.length}`);
|
||||
}
|
||||
provider.populateWorkspaceOwners?.(parsedFiles, { fileContents: getFileContents() });
|
||||
|
||||
// Reconcile scope-resolution's ownership view into the SemanticModel.
|
||||
|
|
|
|||
|
|
@ -1204,6 +1204,77 @@ export const deleteNodesForFile = async (
|
|||
|
||||
export const getEmbeddingTableName = (): string => EMBEDDING_TABLE_NAME;
|
||||
|
||||
/**
|
||||
* Return the distinct repo-relative paths of files that import
|
||||
* `targetFilePath` according to the IMPORTS edges currently in the
|
||||
* DB. Used by the incremental writeback path to expand the
|
||||
* "files-to-rewrite" set so that files importing a changed file get
|
||||
* their edges (which may have been refined by cross-file resolution)
|
||||
* re-emitted, rather than left stale in the DB.
|
||||
*
|
||||
* The DB query reads the *previous* run's state — pre-pipeline, before
|
||||
* any nodes are deleted — so the returned importers are "files that
|
||||
* USED TO import the target". That's the right set to invalidate:
|
||||
* those are the files whose edges in the DB might no longer match
|
||||
* what cross-file resolution produces given the changed file's new
|
||||
* exports.
|
||||
*/
|
||||
export const queryImporters = async (targetFilePath: string): Promise<string[]> => {
|
||||
if (!conn) {
|
||||
throw new Error('LadybugDB not initialized. Call initLbug first.');
|
||||
}
|
||||
const escaped = targetFilePath.replace(/'/g, "''");
|
||||
const cypher = `
|
||||
MATCH (a)-[r:${REL_TABLE_NAME}]->(b)
|
||||
WHERE r.type = 'IMPORTS' AND b.filePath = '${escaped}'
|
||||
RETURN DISTINCT a.filePath AS importer
|
||||
`;
|
||||
try {
|
||||
const queryResult = await conn.query(cypher);
|
||||
const result = Array.isArray(queryResult) ? queryResult[0] : queryResult;
|
||||
const rows = await result.getAll();
|
||||
const out: string[] = [];
|
||||
for (const row of rows) {
|
||||
const v = (row as { importer?: unknown }).importer;
|
||||
if (typeof v === 'string' && v.length > 0) out.push(v);
|
||||
}
|
||||
return out;
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Drop every Community and Process node (and their MEMBER_OF /
|
||||
* STEP_IN_PROCESS edges via DETACH DELETE). Used at the start of an
|
||||
* incremental run so the communities and processes phases regenerate
|
||||
* them from scratch on the merged graph — required for the
|
||||
* "Leiden runs on the FULL graph" correctness invariant.
|
||||
*/
|
||||
export const deleteAllCommunitiesAndProcesses = async (): Promise<{
|
||||
nodesDeleted: number;
|
||||
}> => {
|
||||
if (!conn) {
|
||||
throw new Error('LadybugDB not initialized. Call initLbug first.');
|
||||
}
|
||||
let nodesDeleted = 0;
|
||||
for (const label of ['Community', 'Process']) {
|
||||
try {
|
||||
const countResult = await conn.query(`MATCH (n:${label}) RETURN count(n) AS cnt`);
|
||||
const result = Array.isArray(countResult) ? countResult[0] : countResult;
|
||||
const rows = await result.getAll();
|
||||
const count = Number(rows[0]?.cnt ?? rows[0]?.[0] ?? 0);
|
||||
if (count > 0) {
|
||||
await conn.query(`MATCH (n:${label}) DETACH DELETE n`);
|
||||
nodesDeleted += count;
|
||||
}
|
||||
} catch {
|
||||
// Table may not exist yet on a freshly-initialized DB — fine.
|
||||
}
|
||||
}
|
||||
return { nodesDeleted };
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
// Full-Text Search (FTS) Functions
|
||||
// ============================================================================
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@
|
|||
|
||||
import path from 'path';
|
||||
import fs from 'fs/promises';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { runPipelineFromRepo } from './ingestion/pipeline.js';
|
||||
import {
|
||||
initLbug,
|
||||
|
|
@ -20,6 +21,9 @@ import {
|
|||
executeWithReusedStatement,
|
||||
closeLbug,
|
||||
loadCachedEmbeddings,
|
||||
deleteNodesForFile,
|
||||
deleteAllCommunitiesAndProcesses,
|
||||
queryImporters,
|
||||
} from './lbug/lbug-adapter.js';
|
||||
import { createSearchFTSIndexes } from './search/fts-indexes.js';
|
||||
import {
|
||||
|
|
@ -29,7 +33,15 @@ import {
|
|||
ensureGitNexusIgnored,
|
||||
registerRepo,
|
||||
cleanupOldKuzuFiles,
|
||||
INCREMENTAL_SCHEMA_VERSION,
|
||||
} from '../storage/repo-manager.js';
|
||||
import { computeFileHashes, diffFileHashes } from '../storage/file-hash.js';
|
||||
import {
|
||||
extractChangedSubgraph,
|
||||
computeEffectiveWriteSet,
|
||||
} from './incremental/subgraph-extract.js';
|
||||
import { shadowCandidatesFor } from './incremental/shadow-candidates.js';
|
||||
import { loadParseCache, saveParseCache, pruneCache } from '../storage/parse-cache.js';
|
||||
import {
|
||||
getCurrentCommit,
|
||||
getRemoteUrl,
|
||||
|
|
@ -178,23 +190,81 @@ export async function runFullAnalysis(
|
|||
const currentCommit = repoHasGit ? getCurrentCommit(repoPath) : '';
|
||||
const existingMeta = await loadMeta(storagePath);
|
||||
|
||||
// ── Crash recovery: dirty flag forces full rebuild ────────────────
|
||||
// If the previous incremental run set incrementalInProgress and didn't
|
||||
// clear it, the on-disk index may be in a half-state. Cheapest path
|
||||
// back to a known-good index is to wipe + rebuild from scratch.
|
||||
if (existingMeta?.incrementalInProgress) {
|
||||
log(
|
||||
'Previous incremental run did not complete cleanly (incrementalInProgress flag set); ' +
|
||||
'forcing full rebuild to restore a known-good index.',
|
||||
);
|
||||
options = { ...options, force: true };
|
||||
// Reload meta after clearing the flag in-memory; we still want fileHashes
|
||||
// for the post-rebuild meta carry-over, but force=true ensures the
|
||||
// rebuild path executes.
|
||||
}
|
||||
|
||||
// ── Early-return: already up to date ──────────────────────────────
|
||||
if (existingMeta && !options.force && existingMeta.lastCommit === currentCommit) {
|
||||
// Non-git folders have currentCommit = '' — always rebuild since we can't detect changes
|
||||
if (currentCommit !== '') {
|
||||
await ensureGitNexusIgnored(repoPath);
|
||||
return {
|
||||
// `resolveRepoIdentityRoot` collapses worktree roots to the
|
||||
// canonical repo basename (#1259) but leaves arbitrary subdirs
|
||||
// and `--skip-git` paths unchanged (#1232/#1233 intent preserved).
|
||||
repoName:
|
||||
options.registryName ??
|
||||
getInferredRepoName(repoPath) ??
|
||||
path.basename(resolveRepoIdentityRoot(repoPath)),
|
||||
repoPath,
|
||||
stats: existingMeta.stats ?? {},
|
||||
alreadyUpToDate: true,
|
||||
};
|
||||
// For git repos, even if HEAD matches lastCommit, the working tree
|
||||
// may have uncommitted changes. Only short-circuit when the working
|
||||
// tree is also clean — otherwise fall through to the incremental
|
||||
// path which will hash-diff and update only changed files.
|
||||
//
|
||||
// We exclude paths that GitNexus itself writes during analyze:
|
||||
// .gitnexus/ — db / parse cache / meta.json
|
||||
// .claude/, .cursor/ — auto-generated agent skill files
|
||||
// AGENTS.md, CLAUDE.md — auto-updated stats blocks
|
||||
// Counting them as dirty would perpetually defeat the up-to-date
|
||||
// fast path because the previous analyze just wrote them
|
||||
// (regression vs PR #1233 behavior).
|
||||
const dirty = (() => {
|
||||
try {
|
||||
const out = execFileSync(
|
||||
'git',
|
||||
[
|
||||
'status',
|
||||
'--porcelain',
|
||||
'--',
|
||||
'.',
|
||||
':(exclude).gitnexus',
|
||||
':(exclude).gitnexus/**',
|
||||
':(exclude).claude',
|
||||
':(exclude).claude/**',
|
||||
':(exclude).cursor',
|
||||
':(exclude).cursor/**',
|
||||
':(exclude)AGENTS.md',
|
||||
':(exclude)CLAUDE.md',
|
||||
],
|
||||
{
|
||||
cwd: repoPath,
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
encoding: 'utf8',
|
||||
},
|
||||
);
|
||||
return out.trim().length > 0;
|
||||
} catch {
|
||||
return true; // conservative on git failure
|
||||
}
|
||||
})();
|
||||
if (!dirty) {
|
||||
await ensureGitNexusIgnored(repoPath);
|
||||
return {
|
||||
// `resolveRepoIdentityRoot` collapses worktree roots to the
|
||||
// canonical repo basename (#1259) but leaves arbitrary subdirs
|
||||
// and `--skip-git` paths unchanged (#1232/#1233 intent preserved).
|
||||
repoName:
|
||||
options.registryName ??
|
||||
getInferredRepoName(repoPath) ??
|
||||
path.basename(resolveRepoIdentityRoot(repoPath)),
|
||||
repoPath,
|
||||
stats: existingMeta.stats ?? {},
|
||||
alreadyUpToDate: true,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -243,6 +313,14 @@ export async function runFullAnalysis(
|
|||
);
|
||||
}
|
||||
|
||||
// We *always* load the embedding cache when one is requested (regardless
|
||||
// of the predicted `willTryIncremental`). The post-pipeline branch may
|
||||
// disagree with the prediction (e.g. when the pipeline produces zero
|
||||
// File nodes, `isIncremental` flips false and the full-rebuild path
|
||||
// wipes the DB) — loading unconditionally is cheap insurance against
|
||||
// silently dropping embeddings on a mispredicted run. The re-insert
|
||||
// step gates itself on the actual `isIncremental` value to avoid
|
||||
// PK-conflicts when the incremental writeback path keeps the rows.
|
||||
if (shouldLoadCache && existingMeta) {
|
||||
try {
|
||||
progress('embeddings', 0, 'Caching embeddings...');
|
||||
|
|
@ -270,24 +348,89 @@ export async function runFullAnalysis(
|
|||
}
|
||||
}
|
||||
|
||||
// ── Load incremental parse cache ──────────────────────────────────
|
||||
// Content-addressed: safe to reuse across `--force` runs (chunks whose
|
||||
// file contents haven't changed produce identical worker output).
|
||||
// Loaded into a single ParseCache object that the pipeline mutates
|
||||
// in-place (cache hits leave entries unchanged; misses add new ones).
|
||||
const parseCache = await loadParseCache(storagePath);
|
||||
|
||||
// ── Phase 1: Full Pipeline (0–60%) ────────────────────────────────
|
||||
const pipelineResult = await runPipelineFromRepo(repoPath, (p) => {
|
||||
const phaseLabel = PHASE_LABELS[p.phase] || p.phase;
|
||||
const scaled = Math.round(p.percent * 0.6);
|
||||
const message = p.detail ? `${p.message || phaseLabel} (${p.detail})` : p.message || phaseLabel;
|
||||
progress(p.phase, scaled, message);
|
||||
});
|
||||
const pipelineResult = await runPipelineFromRepo(
|
||||
repoPath,
|
||||
(p) => {
|
||||
const phaseLabel = PHASE_LABELS[p.phase] || p.phase;
|
||||
const scaled = Math.round(p.percent * 0.6);
|
||||
const message = p.detail
|
||||
? `${p.message || phaseLabel} (${p.detail})`
|
||||
: p.message || phaseLabel;
|
||||
progress(p.phase, scaled, message);
|
||||
},
|
||||
{ parseCache },
|
||||
);
|
||||
|
||||
// ── Phase 2: LadybugDB (60–85%) ──────────────────────────────────
|
||||
progress('lbug', 60, 'Loading into LadybugDB...');
|
||||
|
||||
await closeLbug();
|
||||
const lbugFiles = [lbugPath, `${lbugPath}.wal`, `${lbugPath}.lock`];
|
||||
for (const f of lbugFiles) {
|
||||
try {
|
||||
await fs.rm(f, { recursive: true, force: true });
|
||||
} catch {
|
||||
/* swallow */
|
||||
// Compute current per-file content hashes from the pipeline's File nodes.
|
||||
// Used both to drive the incremental DB writeback (when eligible) and to
|
||||
// populate meta.json.fileHashes for the next run.
|
||||
const allFilePaths: string[] = [];
|
||||
pipelineResult.graph.forEachNode((n) => {
|
||||
if (n.label === 'File') {
|
||||
const fp = n.properties?.filePath as string | undefined;
|
||||
if (fp) allFilePaths.push(fp);
|
||||
}
|
||||
});
|
||||
const newFileHashes = await computeFileHashes(repoPath, allFilePaths);
|
||||
|
||||
// Decide incremental vs full at THIS point (post-pipeline, pre-DB).
|
||||
// All eligibility conditions are checked here against the actual
|
||||
// pipeline output — no separate pre-pipeline prediction to desync from
|
||||
// (Bugbot review on PR #1479: a prediction that flipped post-pipeline
|
||||
// could skip the embedding cache load and then take the full-rebuild
|
||||
// path, silently losing embeddings).
|
||||
const isIncremental =
|
||||
!options.force &&
|
||||
!!existingMeta &&
|
||||
existingMeta.schemaVersion === INCREMENTAL_SCHEMA_VERSION &&
|
||||
!!existingMeta.fileHashes &&
|
||||
Object.keys(existingMeta.fileHashes).length > 0 &&
|
||||
repoHasGit &&
|
||||
allFilePaths.length > 0;
|
||||
|
||||
const hashDiff = isIncremental
|
||||
? diffFileHashes(newFileHashes, existingMeta!.fileHashes)
|
||||
: undefined;
|
||||
|
||||
if (isIncremental && hashDiff) {
|
||||
log(
|
||||
`Incremental: changed=${hashDiff.changed.length}, ` +
|
||||
`added=${hashDiff.added.length}, ` +
|
||||
`deleted=${hashDiff.deleted.length} ` +
|
||||
`(skipping wipe + ${
|
||||
allFilePaths.length - hashDiff.toWrite.length
|
||||
} unchanged file rows preserved)`,
|
||||
);
|
||||
// Set the dirty flag BEFORE any destructive DB mutation. Cleared on
|
||||
// success at the meta-save step.
|
||||
await saveMeta(storagePath, {
|
||||
...existingMeta!,
|
||||
incrementalInProgress: {
|
||||
startedAt: Date.now(),
|
||||
toWriteCount: hashDiff.toWrite.length,
|
||||
},
|
||||
});
|
||||
} else {
|
||||
// Full rebuild path: wipe DB files first.
|
||||
await closeLbug();
|
||||
const lbugFiles = [lbugPath, `${lbugPath}.wal`, `${lbugPath}.lock`];
|
||||
for (const f of lbugFiles) {
|
||||
try {
|
||||
await fs.rm(f, { recursive: true, force: true });
|
||||
} catch {
|
||||
/* swallow */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -298,11 +441,145 @@ export async function runFullAnalysis(
|
|||
// must be released to avoid blocking subsequent invocations.
|
||||
|
||||
let lbugMsgCount = 0;
|
||||
await loadGraphToLbug(pipelineResult.graph, pipelineResult.repoPath, storagePath, (msg) => {
|
||||
lbugMsgCount++;
|
||||
const pct = Math.min(84, 60 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 24));
|
||||
progress('lbug', pct, msg);
|
||||
});
|
||||
if (isIncremental && hashDiff) {
|
||||
// ── Incremental DB writeback ───────────────────────────────────
|
||||
// 0. Expand the writable set with transitive importers of
|
||||
// changed/deleted files (bounded BFS).
|
||||
//
|
||||
// Reason (Bugbot/Claude review on PR #1479): when a barrel /
|
||||
// re-export file C changes, cross-file resolution may update
|
||||
// CALLS edges between two unchanged files A and B (A imports
|
||||
// from C, C re-exports something from B). Those refined edges
|
||||
// live in `ctx.graph` but would be excluded from the subgraph
|
||||
// if neither endpoint is in the changed set. To catch this,
|
||||
// files that imported (directly OR transitively, through
|
||||
// other unchanged intermediaries) any changed file get pulled
|
||||
// into the writable set so their rows are deleted + rewritten
|
||||
// against the refined edges.
|
||||
//
|
||||
// BFS bound: MAX_IMPORTER_BFS_DEPTH. Practically sized to
|
||||
// catch nested barrel chains (e.g. `index.ts → submodule/index.ts
|
||||
// → submodule/impl.ts`) without ballooning into a near-full-
|
||||
// rebuild on monorepos with deep re-export pyramids. Beyond
|
||||
// this depth, the "incremental ≡ full-rebuild" invariant is
|
||||
// self-acknowledged as best-effort; `--force` remains the
|
||||
// escape hatch documented in GUARDRAILS.md.
|
||||
//
|
||||
// `queryImporters` reads `IMPORTS` from the pre-pipeline DB
|
||||
// state, so the result is "files that USED TO import the
|
||||
// target" — exactly the set whose previously-stored edges may
|
||||
// no longer match what cross-file resolution produces this run.
|
||||
const MAX_IMPORTER_BFS_DEPTH = 4;
|
||||
const writableFiles = new Set<string>(hashDiff.toWrite);
|
||||
const directlyChangedCount = writableFiles.size;
|
||||
|
||||
// Shadow-seed: for ADDED files, queryImporters returns 0 (the new
|
||||
// file has no IMPORTS rows in the pre-pipeline DB yet). But pre-
|
||||
// existing unchanged files may have IMPORTS edges whose module-
|
||||
// resolution claim the newcomer can steal under standard JS/TS
|
||||
// resolution (Bugbot review on PR #1479). For each added file we
|
||||
// derive the shadow candidates and, if the candidate was a known
|
||||
// file in the prior meta, seed it into the BFS frontier so its
|
||||
// importers — surfaced via queryImporters — get their CALLS edges
|
||||
// re-resolved against the new file. See shadow-candidates.ts for
|
||||
// the full pattern catalogue.
|
||||
const priorFileSet = new Set<string>(
|
||||
existingMeta?.fileHashes ? Object.keys(existingMeta.fileHashes) : [],
|
||||
);
|
||||
const shadowSeed: string[] = [];
|
||||
for (const added of hashDiff.added) {
|
||||
for (const cand of shadowCandidatesFor(added)) {
|
||||
if (priorFileSet.has(cand) && !writableFiles.has(cand)) {
|
||||
shadowSeed.push(cand);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let frontier: string[] = [...hashDiff.toWrite, ...hashDiff.deleted, ...shadowSeed];
|
||||
for (let depth = 0; depth < MAX_IMPORTER_BFS_DEPTH && frontier.length > 0; depth++) {
|
||||
const nextFrontier: string[] = [];
|
||||
for (const f of frontier) {
|
||||
try {
|
||||
const importers = await queryImporters(f);
|
||||
for (const i of importers) {
|
||||
if (!writableFiles.has(i)) {
|
||||
writableFiles.add(i);
|
||||
nextFrontier.push(i);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* per-file importer query failure → skip; correctness degrades on
|
||||
that branch, but DB stays writable. */
|
||||
}
|
||||
}
|
||||
frontier = nextFrontier;
|
||||
}
|
||||
}
|
||||
const importerExpansion = writableFiles.size - directlyChangedCount;
|
||||
if (importerExpansion > 0) {
|
||||
log(
|
||||
`Incremental: +${importerExpansion} importer(s) added to writable set ` +
|
||||
`(BFS depth ≤ ${MAX_IMPORTER_BFS_DEPTH}` +
|
||||
(shadowSeed.length > 0 ? `, ${shadowSeed.length} shadow-seed(s)` : '') +
|
||||
`)`,
|
||||
);
|
||||
}
|
||||
|
||||
// 1. Compute the EFFECTIVE write-set (Finding 1). Two layers,
|
||||
// composed:
|
||||
// (a) `writableFiles` — toWrite ∪ transitive importers of
|
||||
// changed/deleted files (the bounded BFS above, reading
|
||||
// IMPORTS from the pre-pipeline DB).
|
||||
// (b) `computeEffectiveWriteSet` — walks the NEW graph's
|
||||
// edges and pulls in any unchanged-side file that sits
|
||||
// on a writable-boundary-crossing edge (catches refined
|
||||
// cross-file CALLS edges that the pre-run DB couldn't
|
||||
// predict, e.g. a barrel re-export shifting `foo` from
|
||||
// B to D).
|
||||
// The composed set is the input to BOTH deleteNodesForFile
|
||||
// and extractChangedSubgraph — asymmetry between the two would
|
||||
// leave stale rows or PK-conflict at COPY time.
|
||||
const effectiveWriteSet = computeEffectiveWriteSet(pipelineResult.graph, writableFiles);
|
||||
// Deduped: deleted entries may already appear via importer-BFS
|
||||
// expansion (queryImporters can return a now-deleted path), which
|
||||
// would otherwise call deleteNodesForFile twice for the same file
|
||||
// (Bugbot LOW finding on PR #1479).
|
||||
const filesToDelete = [...new Set([...effectiveWriteSet, ...hashDiff.deleted])];
|
||||
for (let i = 0; i < filesToDelete.length; i++) {
|
||||
const f = filesToDelete[i];
|
||||
try {
|
||||
await deleteNodesForFile(f);
|
||||
} catch {
|
||||
/* file may not have rows (e.g. an unparseable file) — fine */
|
||||
}
|
||||
if (i % 20 === 0) {
|
||||
progress('lbug', 62, `Removing rows for changed files (${i}/${filesToDelete.length})...`);
|
||||
}
|
||||
}
|
||||
// 2. Drop graph-wide nodes (Community, Process). They'll be re-inserted
|
||||
// from the fresh pipeline output below. Required for the
|
||||
// "Leiden runs on the FULL graph" correctness invariant.
|
||||
await deleteAllCommunitiesAndProcesses();
|
||||
|
||||
// 3. Extract the changed subgraph from the FULL ctx.graph and write
|
||||
// only that. Unchanged-file rows in the DB stay untouched. Pass
|
||||
// the SAME effectiveWriteSet so the subgraph and the deletes
|
||||
// cover identical files (asymmetry would silently corrupt).
|
||||
const subgraph = extractChangedSubgraph(pipelineResult.graph, effectiveWriteSet);
|
||||
await loadGraphToLbug(subgraph, pipelineResult.repoPath, storagePath, (msg) => {
|
||||
lbugMsgCount++;
|
||||
const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19));
|
||||
progress('lbug', pct, msg);
|
||||
});
|
||||
} else {
|
||||
// ── Full rebuild ───────────────────────────────────────────────
|
||||
await loadGraphToLbug(pipelineResult.graph, pipelineResult.repoPath, storagePath, (msg) => {
|
||||
lbugMsgCount++;
|
||||
const pct = Math.min(84, 60 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 24));
|
||||
progress('lbug', pct, msg);
|
||||
});
|
||||
}
|
||||
|
||||
// ── Phase 3: FTS (85–90%) ─────────────────────────────────────────
|
||||
progress('fts', 85, 'Creating search indexes...');
|
||||
|
|
@ -310,6 +587,19 @@ export async function runFullAnalysis(
|
|||
progress('fts', 90, 'Search indexes ready');
|
||||
|
||||
// ── Phase 3.5: Re-insert cached embeddings ────────────────────────
|
||||
// Runs on BOTH the full-rebuild path and the incremental path:
|
||||
// - Full rebuild: DB was wiped, every cached row needs to come back.
|
||||
// - Incremental: changed-file rows were just deleted by
|
||||
// deleteNodesForFile (which cascades to their
|
||||
// embedding rows) — so their cached vectors need
|
||||
// to come back too. Unchanged-file rows still
|
||||
// exist; re-inserting their cached vectors would
|
||||
// PK-conflict, but the per-batch try/catch below
|
||||
// silently ignores those (matches the existing
|
||||
// "some may fail if node was removed, that's
|
||||
// fine" semantics). Bugbot review on PR #1479
|
||||
// flagged that gating this on `!isIncremental`
|
||||
// silently lost changed-file embeddings.
|
||||
if (cachedEmbeddings.length > 0) {
|
||||
const cachedDims = cachedEmbeddings[0].embedding.length;
|
||||
const { EMBEDDING_DIMS } = await import('./lbug/schema.js');
|
||||
|
|
@ -456,6 +746,12 @@ export async function runFullAnalysis(
|
|||
const effectiveSemanticMode =
|
||||
semanticMode ??
|
||||
(runtimeCapabilities.semanticMode === 'vector-index' ? 'vector-index' : 'exact-scan');
|
||||
|
||||
// Convert the post-run file-hash map to the on-disk Record<string,string>
|
||||
// shape consumed by RepoMeta.fileHashes.
|
||||
const newFileHashesRecord: Record<string, string> = {};
|
||||
for (const [k, v] of newFileHashes) newFileHashesRecord[k] = v;
|
||||
|
||||
const meta = {
|
||||
repoPath,
|
||||
lastCommit: currentCommit,
|
||||
|
|
@ -485,8 +781,33 @@ export async function runFullAnalysis(
|
|||
reason: runtimeCapabilities.reason,
|
||||
},
|
||||
},
|
||||
// Incremental-indexing fields. Populated for git repos so the next
|
||||
// analyze run can take the incremental DB-writeback path. Setting
|
||||
// incrementalInProgress to undefined explicitly clears any prior
|
||||
// dirty flag (full and incremental success paths converge here).
|
||||
schemaVersion: hasGitDir(repoPath) ? INCREMENTAL_SCHEMA_VERSION : undefined,
|
||||
fileHashes: hasGitDir(repoPath) ? newFileHashesRecord : undefined,
|
||||
incrementalInProgress: undefined as { startedAt: number; toWriteCount: number } | undefined,
|
||||
};
|
||||
await saveMeta(storagePath, meta);
|
||||
|
||||
// Persist the incremental parse cache for the next run. Wraps in
|
||||
// try/catch so a cache-write failure never breaks an otherwise
|
||||
// successful indexing run. Prune stale chunk-hash entries first so
|
||||
// the cache file size stays bounded across runs (chunks whose
|
||||
// composition no longer matches anything in the current scan are
|
||||
// dead weight; the parse phase populates `usedKeys` as it processes
|
||||
// chunks).
|
||||
try {
|
||||
const pruned = pruneCache(parseCache, parseCache.usedKeys);
|
||||
if (pruned > 0) {
|
||||
log(`Parse cache: pruned ${pruned} stale chunk entries`);
|
||||
}
|
||||
await saveParseCache(storagePath, parseCache);
|
||||
} catch (e) {
|
||||
log(`Warning: could not save parse cache (${(e as Error).message}); continuing.`);
|
||||
}
|
||||
|
||||
// Forward the --name alias and the registry-collision bypass bit.
|
||||
// `allowDuplicateName` is its own concern — independent from the
|
||||
// pipeline `force` above. The CLI maps it from
|
||||
|
|
|
|||
104
gitnexus/src/storage/file-hash.ts
Normal file
104
gitnexus/src/storage/file-hash.ts
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
/**
|
||||
* Per-file content hashing for incremental DB writeback.
|
||||
*
|
||||
* On every analyze run we compute SHA-256 of every file's content and
|
||||
* store the map in meta.json. The next run compares disk against the
|
||||
* stored map and produces:
|
||||
* - `changed` — content differs (re-emit DB rows for this file)
|
||||
* - `added` — file is new on disk (insert DB rows)
|
||||
* - `deleted` — file was in last meta but no longer on disk (drop rows)
|
||||
*
|
||||
* The pipeline still parses every file (correctness invariant: cross-file
|
||||
* resolution needs full data). What this enables is a SELECTIVE DB
|
||||
* writeback: instead of wipe-and-reload of the whole graph (~50s of CSV
|
||||
* COPY on a 25K-node repo), we only delete-and-rewrite rows for the
|
||||
* changed/added/deleted set.
|
||||
*
|
||||
* See docs/superpowers/specs/2026-05-10-incremental-indexing-design.md
|
||||
* (Option B revision).
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
|
||||
/**
|
||||
* Compute SHA-256 of a single file. Returns null when the file can't be
|
||||
* read — caller treats that as "no signature, assume changed".
|
||||
*/
|
||||
export const computeFileHash = async (absPath: string): Promise<string | null> => {
|
||||
try {
|
||||
const buf = await fs.readFile(absPath);
|
||||
return createHash('sha256').update(buf).digest('hex');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Compute SHA-256 hashes for many files in parallel batches. Files that
|
||||
* fail to read are omitted from the result map.
|
||||
*/
|
||||
export const computeFileHashes = async (
|
||||
repoPath: string,
|
||||
relPaths: readonly string[],
|
||||
): Promise<Map<string, string>> => {
|
||||
const out = new Map<string, string>();
|
||||
const BATCH = 100;
|
||||
for (let i = 0; i < relPaths.length; i += BATCH) {
|
||||
const batch = relPaths.slice(i, i + BATCH);
|
||||
const results = await Promise.all(
|
||||
batch.map(async (rel) => {
|
||||
const h = await computeFileHash(path.join(repoPath, rel));
|
||||
return h ? ([rel, h] as const) : null;
|
||||
}),
|
||||
);
|
||||
for (const r of results) if (r) out.set(r[0], r[1]);
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
/** Result of comparing the current on-disk hashes against stored ones. */
|
||||
export interface FileHashDiff {
|
||||
/** Files whose content hash differs from stored. */
|
||||
changed: string[];
|
||||
/** Files in the current scan that weren't in the stored map. */
|
||||
added: string[];
|
||||
/** Files in the stored map that aren't in the current scan. */
|
||||
deleted: string[];
|
||||
/** All files whose DB rows must be replaced (changed ∪ added). */
|
||||
toWrite: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Diff a current hash map against a previously stored one.
|
||||
*
|
||||
* Sorted output so two runs produce identical diff arrays for the same
|
||||
* changes — useful for stable logging / equivalence checks.
|
||||
*/
|
||||
export const diffFileHashes = (
|
||||
current: ReadonlyMap<string, string>,
|
||||
stored: Readonly<Record<string, string>> | undefined,
|
||||
): FileHashDiff => {
|
||||
const storedMap = new Map<string, string>(stored ? Object.entries(stored) : []);
|
||||
const changed: string[] = [];
|
||||
const added: string[] = [];
|
||||
for (const [p, h] of current) {
|
||||
const prev = storedMap.get(p);
|
||||
if (prev === undefined) added.push(p);
|
||||
else if (prev !== h) changed.push(p);
|
||||
}
|
||||
const deleted: string[] = [];
|
||||
for (const p of storedMap.keys()) {
|
||||
if (!current.has(p)) deleted.push(p);
|
||||
}
|
||||
changed.sort();
|
||||
added.sort();
|
||||
deleted.sort();
|
||||
return {
|
||||
changed,
|
||||
added,
|
||||
deleted,
|
||||
toWrite: [...changed, ...added].sort(),
|
||||
};
|
||||
};
|
||||
213
gitnexus/src/storage/parse-cache.ts
Normal file
213
gitnexus/src/storage/parse-cache.ts
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
/**
|
||||
* Chunk-level content-addressed parse cache.
|
||||
*
|
||||
* The pipeline always parses every file (correctness invariant: cross-file
|
||||
* resolution and downstream phases need full graph data). What this cache
|
||||
* does is skip the tree-sitter worker dispatch when a chunk's contents
|
||||
* haven't changed since the last run.
|
||||
*
|
||||
* Granularity: chunk-level. The parse phase chunks files into ~20MB byte
|
||||
* budgets. The cache key is `sha256(joined(filePath:contentHash for each
|
||||
* file in the chunk, sorted))`. A change to a single file invalidates only
|
||||
* that file's chunk — typically 1 of ~50 chunks on a 1000-file repo.
|
||||
*
|
||||
* Why not per-file:
|
||||
* - Workers process sub-batches and emit aggregated `ParseWorkerResult`s.
|
||||
* Splitting back to per-file would require reworking the worker contract.
|
||||
* - Chunk-level invalidation gives a useful speedup floor (98% on a single
|
||||
* 1-of-50 invalidated chunk) without touching the worker.
|
||||
*
|
||||
* Survives `--force` because it's content-addressed: the same bytes always
|
||||
* produce the same key. `--force` only matters for the LadybugDB writeback;
|
||||
* the cache itself is always safe to reuse.
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
import { createRequire } from 'module';
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.js';
|
||||
|
||||
/**
|
||||
* Cache version composed of:
|
||||
* - A schema bump knob (`SCHEMA_BUMP`) for hand-controlled invalidation
|
||||
* when ParseWorkerResult shape or upstream parse semantics change.
|
||||
* - The current `gitnexus` npm package version, read at module load.
|
||||
* Any release that ships an updated tree-sitter grammar or revised
|
||||
* extractor logic implies a version bump in package.json, which
|
||||
* automatically invalidates the on-disk cache. Without this, a user
|
||||
* running `npm i -g gitnexus@latest` after a parser-affecting
|
||||
* release would silently replay pre-upgrade ParseWorkerResults
|
||||
* against the new graph schema (Bugbot/Claude review on #1479).
|
||||
*
|
||||
* On version mismatch, `loadParseCache` returns an empty cache and the
|
||||
* next save overwrites the on-disk file with the new version baked in.
|
||||
*/
|
||||
const SCHEMA_BUMP = 1;
|
||||
const GITNEXUS_PKG_VERSION = (() => {
|
||||
try {
|
||||
// package.json sits at gitnexus/package.json — two levels up from
|
||||
// gitnexus/src/storage/parse-cache.ts (or its dist/ equivalent).
|
||||
const here = path.dirname(fileURLToPath(import.meta.url));
|
||||
const candidates = [
|
||||
path.join(here, '..', '..', 'package.json'), // src/storage → gitnexus/
|
||||
path.join(here, '..', '..', '..', 'package.json'), // dist/storage → gitnexus/
|
||||
];
|
||||
const requireCJS = createRequire(import.meta.url);
|
||||
for (const c of candidates) {
|
||||
try {
|
||||
const pkg = requireCJS(c);
|
||||
if (typeof pkg?.version === 'string') return pkg.version;
|
||||
} catch {
|
||||
/* try next candidate */
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* fall through to fallback */
|
||||
}
|
||||
return '0.0.0-unknown';
|
||||
})();
|
||||
export const PARSE_CACHE_VERSION = `${SCHEMA_BUMP}+${GITNEXUS_PKG_VERSION}`;
|
||||
|
||||
const CACHE_FILENAME = 'parse-cache.json';
|
||||
|
||||
/** On-disk shape. */
|
||||
interface ParseCacheFile {
|
||||
version: string;
|
||||
/** key = chunk hash (hex) → cached chunk result list. */
|
||||
entries: Record<string, ParseWorkerResult[]>;
|
||||
}
|
||||
|
||||
/** Runtime view: keyed Map for fast lookup; mutated in place during a run. */
|
||||
export interface ParseCache {
|
||||
version: string;
|
||||
entries: Map<string, ParseWorkerResult[]>;
|
||||
/**
|
||||
* Hashes referenced (hit OR miss-and-stored) by the current run.
|
||||
* The parse phase populates this as it processes chunks; the orchestrator
|
||||
* uses it as input to `pruneCache` before saving so entries that no
|
||||
* longer correspond to any chunk in the current scan are discarded.
|
||||
* Transient — never serialized to disk.
|
||||
*/
|
||||
usedKeys: Set<string>;
|
||||
}
|
||||
|
||||
/** SHA-256 hex of a single string or buffer. */
|
||||
const sha256Hex = (input: Buffer | string): string =>
|
||||
createHash('sha256')
|
||||
.update(typeof input === 'string' ? Buffer.from(input) : input)
|
||||
.digest('hex');
|
||||
|
||||
/** Stable hash of a single file's contents — used by callers to compose a chunk hash. */
|
||||
export const fileContentHash = (content: Buffer | string): string => sha256Hex(content);
|
||||
|
||||
/**
|
||||
* Compute the canonical cache key for a chunk's contents.
|
||||
*
|
||||
* `entries` is the list of (filePath, file content hash) for every file
|
||||
* in the chunk. We sort by filePath before hashing so chunks composed of
|
||||
* the same files in different order produce the same key.
|
||||
*/
|
||||
export const computeChunkHash = (
|
||||
entries: Array<{ filePath: string; contentHash: string }>,
|
||||
): string => {
|
||||
const sorted = [...entries].sort((a, b) => (a.filePath < b.filePath ? -1 : 1));
|
||||
const joined = sorted.map((e) => `${e.filePath}:${e.contentHash}`).join('\n');
|
||||
return sha256Hex(joined);
|
||||
};
|
||||
|
||||
/**
|
||||
* JSON replacer that round-trips Map/Set instances through plain JSON.
|
||||
*
|
||||
* `ParseWorkerResult.parsedFiles[*].scopes[*].typeBindings` is a
|
||||
* `ReadonlyMap<string, TypeRef>`; without this transform it serializes
|
||||
* to `{}` and downstream code that iterates / `.get()`s on it crashes
|
||||
* with "is not iterable". Applied symmetrically by `mapReviver` on
|
||||
* load so the in-memory shape stays Map-typed.
|
||||
*/
|
||||
const MAP_TAG = '__$mapEntries$__';
|
||||
const SET_TAG = '__$setValues$__';
|
||||
|
||||
const mapReplacer = (_key: string, value: unknown): unknown => {
|
||||
if (value instanceof Map) return { [MAP_TAG]: Array.from(value.entries()) };
|
||||
if (value instanceof Set) return { [SET_TAG]: Array.from(value.values()) };
|
||||
return value;
|
||||
};
|
||||
|
||||
const mapReviver = (_key: string, value: unknown): unknown => {
|
||||
if (value && typeof value === 'object') {
|
||||
const v = value as Record<string, unknown>;
|
||||
if (Array.isArray(v[MAP_TAG])) return new Map(v[MAP_TAG] as [unknown, unknown][]);
|
||||
if (Array.isArray(v[SET_TAG])) return new Set(v[SET_TAG] as unknown[]);
|
||||
}
|
||||
return value;
|
||||
};
|
||||
|
||||
/**
|
||||
* Load the parse cache. Returns an empty cache on any failure (missing
|
||||
* file, corrupt JSON, version mismatch). Never throws on a normal load.
|
||||
*/
|
||||
export const loadParseCache = async (storagePath: string): Promise<ParseCache> => {
|
||||
const cachePath = path.join(storagePath, CACHE_FILENAME);
|
||||
try {
|
||||
const raw = await fs.readFile(cachePath, 'utf-8');
|
||||
const data = JSON.parse(raw, mapReviver) as ParseCacheFile;
|
||||
if (
|
||||
typeof data !== 'object' ||
|
||||
data === null ||
|
||||
data.version !== PARSE_CACHE_VERSION ||
|
||||
typeof data.entries !== 'object' ||
|
||||
data.entries === null
|
||||
) {
|
||||
return emptyCache();
|
||||
}
|
||||
const entries = new Map<string, ParseWorkerResult[]>();
|
||||
for (const [k, v] of Object.entries(data.entries)) {
|
||||
if (Array.isArray(v)) entries.set(k, v as ParseWorkerResult[]);
|
||||
}
|
||||
return { version: PARSE_CACHE_VERSION, entries, usedKeys: new Set<string>() };
|
||||
} catch {
|
||||
return emptyCache();
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Persist the cache to disk atomically (write-and-rename) so a crash
|
||||
* mid-write doesn't leave a corrupt file.
|
||||
*/
|
||||
export const saveParseCache = async (storagePath: string, cache: ParseCache): Promise<void> => {
|
||||
await fs.mkdir(storagePath, { recursive: true });
|
||||
const cachePath = path.join(storagePath, CACHE_FILENAME);
|
||||
const tmpPath = `${cachePath}.tmp`;
|
||||
const out: ParseCacheFile = {
|
||||
version: cache.version,
|
||||
entries: Object.fromEntries(cache.entries),
|
||||
};
|
||||
// Compact JSON; this file can be tens of MB on a large repo and pretty-
|
||||
// printing roughly doubles size for no value.
|
||||
await fs.writeFile(tmpPath, JSON.stringify(out, mapReplacer), 'utf-8');
|
||||
await fs.rename(tmpPath, cachePath);
|
||||
};
|
||||
|
||||
/**
|
||||
* Drop entries whose hashes are not in `usedHashes`. Called at the end
|
||||
* of a run so chunks that no longer correspond to any current chunk
|
||||
* don't keep their stale entries forever.
|
||||
*/
|
||||
export const pruneCache = (cache: ParseCache, usedHashes: ReadonlySet<string>): number => {
|
||||
let removed = 0;
|
||||
for (const k of cache.entries.keys()) {
|
||||
if (!usedHashes.has(k)) {
|
||||
cache.entries.delete(k);
|
||||
removed++;
|
||||
}
|
||||
}
|
||||
return removed;
|
||||
};
|
||||
|
||||
const emptyCache = (): ParseCache => ({
|
||||
version: PARSE_CACHE_VERSION,
|
||||
entries: new Map<string, ParseWorkerResult[]>(),
|
||||
usedKeys: new Set<string>(),
|
||||
});
|
||||
|
|
@ -71,8 +71,40 @@ export interface RepoMeta {
|
|||
processes?: number;
|
||||
embeddings?: number;
|
||||
};
|
||||
/**
|
||||
* Bumped whenever incremental-indexing invariants change in an
|
||||
* incompatible way (delete-and-rewrite logic, subgraph extraction,
|
||||
* graph-wide node handling). On mismatch, runFullAnalysis forces a
|
||||
* full rebuild rather than risk an inconsistent incremental update.
|
||||
*/
|
||||
schemaVersion?: number;
|
||||
/**
|
||||
* SHA-256 of every file's content at the time of the last successful
|
||||
* indexing run. The next run computes current hashes and diffs against
|
||||
* this map to determine which files' DB rows must be replaced.
|
||||
* Map keys are repo-relative paths.
|
||||
*/
|
||||
fileHashes?: Record<string, string>;
|
||||
/**
|
||||
* Crash-recovery dirty flag. Written to meta.json BEFORE any
|
||||
* destructive DB mutation in an incremental run; cleared on success
|
||||
* by overwriting meta.json. If a run crashes between, the next run
|
||||
* sees the flag and forces a full rebuild — the cheapest path back
|
||||
* to a known-good index.
|
||||
*/
|
||||
incrementalInProgress?: {
|
||||
/** When the incremental run started (epoch ms). */
|
||||
startedAt: number;
|
||||
/** Number of files in the writable set, for diagnostic logs. */
|
||||
toWriteCount: number;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Bumped whenever incremental-indexing invariants change incompatibly.
|
||||
*/
|
||||
export const INCREMENTAL_SCHEMA_VERSION = 1;
|
||||
|
||||
export interface IndexedRepo {
|
||||
repoPath: string;
|
||||
storagePath: string;
|
||||
|
|
@ -186,12 +218,23 @@ export const loadMeta = async (storagePath: string): Promise<RepoMeta | null> =>
|
|||
};
|
||||
|
||||
/**
|
||||
* Save metadata to storage
|
||||
* Save metadata to storage.
|
||||
*
|
||||
* Atomic via tmp-file + rename (matches `saveParseCache`'s pattern). The
|
||||
* `incrementalInProgress` dirty flag travels through this file — a crash
|
||||
* mid-write would leave a corrupt `meta.json` that the next run's
|
||||
* `loadMeta` would silently treat as "no prior index", losing the dirty
|
||||
* flag and skipping the recovery full-rebuild. Write-and-rename rules
|
||||
* that out: the rename is atomic on POSIX and on Windows (`fs.rename`
|
||||
* on `node:fs/promises` uses `MoveFileEx(REPLACE_EXISTING)`), so either
|
||||
* the old or the new file is observed at every moment.
|
||||
*/
|
||||
export const saveMeta = async (storagePath: string, meta: RepoMeta): Promise<void> => {
|
||||
await fs.mkdir(storagePath, { recursive: true });
|
||||
const metaPath = path.join(storagePath, 'meta.json');
|
||||
await fs.writeFile(metaPath, JSON.stringify(meta, null, 2), 'utf-8');
|
||||
const tmpPath = `${metaPath}.tmp`;
|
||||
await fs.writeFile(tmpPath, JSON.stringify(meta, null, 2), 'utf-8');
|
||||
await fs.rename(tmpPath, metaPath);
|
||||
};
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -25,5 +25,5 @@
|
|||
"MEMBER_OF": 12,
|
||||
"STEP_IN_PROCESS": 12
|
||||
},
|
||||
"edgeDigest": "a418debec537cf959fe56fd1fbbbfb59a640398cdb3c61ce0bcb8056c1f45110"
|
||||
"edgeDigest": "6f414427a20c037df3e336f055c83f987e7d381c9bfa73b4d2be690cb8103302"
|
||||
}
|
||||
|
|
|
|||
124
gitnexus/test/unit/incremental-file-hash.test.ts
Normal file
124
gitnexus/test/unit/incremental-file-hash.test.ts
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { mkdtemp, writeFile, rm } from 'fs/promises';
|
||||
import { tmpdir } from 'os';
|
||||
import path from 'path';
|
||||
import { computeFileHash, computeFileHashes, diffFileHashes } from '../../src/storage/file-hash.js';
|
||||
|
||||
describe('diffFileHashes', () => {
|
||||
it('classifies files into changed / added / deleted / toWrite', () => {
|
||||
const stored = { a: 'h-a', b: 'h-b', c: 'h-c' };
|
||||
const current = new Map<string, string>([
|
||||
['a', 'h-a'], // unchanged
|
||||
['b', 'h-b-NEW'], // changed
|
||||
['d', 'h-d'], // added
|
||||
// 'c' is gone → deleted
|
||||
]);
|
||||
const diff = diffFileHashes(current, stored);
|
||||
expect(diff.changed).toEqual(['b']);
|
||||
expect(diff.added).toEqual(['d']);
|
||||
expect(diff.deleted).toEqual(['c']);
|
||||
// toWrite is the union of changed ∪ added (rows to be (re)written)
|
||||
expect(diff.toWrite.sort()).toEqual(['b', 'd']);
|
||||
});
|
||||
|
||||
it('treats no stored map as "everything is added"', () => {
|
||||
const current = new Map<string, string>([
|
||||
['x', 'h1'],
|
||||
['y', 'h2'],
|
||||
]);
|
||||
const diff = diffFileHashes(current, undefined);
|
||||
expect(diff.added.sort()).toEqual(['x', 'y']);
|
||||
expect(diff.changed).toEqual([]);
|
||||
expect(diff.deleted).toEqual([]);
|
||||
expect(diff.toWrite.sort()).toEqual(['x', 'y']);
|
||||
});
|
||||
|
||||
it('returns sorted arrays for stable cross-platform comparison', () => {
|
||||
const stored = { z: 'h', a: 'h', m: 'h' };
|
||||
const current = new Map<string, string>([
|
||||
['z', 'h2'],
|
||||
['a', 'h2'],
|
||||
['m', 'h2'],
|
||||
]);
|
||||
const diff = diffFileHashes(current, stored);
|
||||
expect(diff.changed).toEqual(['a', 'm', 'z']);
|
||||
expect(diff.toWrite).toEqual(['a', 'm', 'z']);
|
||||
});
|
||||
|
||||
it('handles empty current map (all stored files become deleted)', () => {
|
||||
const stored = { a: 'h1', b: 'h2' };
|
||||
const diff = diffFileHashes(new Map(), stored);
|
||||
expect(diff.deleted).toEqual(['a', 'b']);
|
||||
expect(diff.changed).toEqual([]);
|
||||
expect(diff.added).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeFileHash', () => {
|
||||
it('produces a stable SHA-256 hex digest', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
|
||||
try {
|
||||
const f = path.join(dir, 'a.txt');
|
||||
await writeFile(f, 'hello world\n', 'utf-8');
|
||||
const h1 = await computeFileHash(f);
|
||||
const h2 = await computeFileHash(f);
|
||||
expect(h1).toBe(h2);
|
||||
expect(h1).toMatch(/^[a-f0-9]{64}$/);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('returns null on missing file (caller treats as "no signature")', async () => {
|
||||
const h = await computeFileHash('/definitely/does/not/exist/here.xyz');
|
||||
expect(h).toBeNull();
|
||||
});
|
||||
|
||||
it('different content → different hash', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
|
||||
try {
|
||||
const a = path.join(dir, 'a.txt');
|
||||
const b = path.join(dir, 'b.txt');
|
||||
await writeFile(a, 'hello', 'utf-8');
|
||||
await writeFile(b, 'goodbye', 'utf-8');
|
||||
const ha = await computeFileHash(a);
|
||||
const hb = await computeFileHash(b);
|
||||
expect(ha).not.toBeNull();
|
||||
expect(hb).not.toBeNull();
|
||||
expect(ha).not.toBe(hb);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeFileHashes', () => {
|
||||
it('hashes a small batch of files in parallel', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
|
||||
try {
|
||||
await writeFile(path.join(dir, 'one.txt'), 'A', 'utf-8');
|
||||
await writeFile(path.join(dir, 'two.txt'), 'B', 'utf-8');
|
||||
await writeFile(path.join(dir, 'three.txt'), 'C', 'utf-8');
|
||||
const map = await computeFileHashes(dir, ['one.txt', 'two.txt', 'three.txt']);
|
||||
expect(map.size).toBe(3);
|
||||
expect(map.get('one.txt')).toMatch(/^[a-f0-9]{64}$/);
|
||||
// All distinct since contents differ
|
||||
const hashes = [...map.values()];
|
||||
expect(new Set(hashes).size).toBe(3);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('omits files that fail to read (no entry in result)', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-fh-'));
|
||||
try {
|
||||
await writeFile(path.join(dir, 'real.txt'), 'X', 'utf-8');
|
||||
const map = await computeFileHashes(dir, ['real.txt', 'phantom.txt']);
|
||||
expect(map.has('real.txt')).toBe(true);
|
||||
expect(map.has('phantom.txt')).toBe(false);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
263
gitnexus/test/unit/incremental-orchestration.test.ts
Normal file
263
gitnexus/test/unit/incremental-orchestration.test.ts
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
/**
|
||||
* Integration coverage for the `runFullAnalysis` incremental-orchestration
|
||||
* wiring (Claude PR-review Finding 2).
|
||||
*
|
||||
* These tests exercise the *real runtime path* — they call
|
||||
* `runFullAnalysis` against a real on-disk git repo backed by a real
|
||||
* LadybugDB at `<repo>/.gitnexus/`, and assert behaviours that pure
|
||||
* unit tests on `diffFileHashes` / `extractChangedSubgraph` cannot
|
||||
* catch:
|
||||
*
|
||||
* - the `isIncremental` decision (post-pipeline eligibility check)
|
||||
* - `incrementalInProgress` dirty-flag set-before-mutation and
|
||||
* clear-on-success
|
||||
* - the importer-closure expansion (1-hop reached via the writable
|
||||
* set, transitive reachable via bounded BFS)
|
||||
* - the "forced full rebuild on dirty-flag-from-prior-crash" path
|
||||
*
|
||||
* Each test creates a temporary git repo, runs the analyzer, and asserts
|
||||
* on the resulting `meta.json` and graph state. Cleanup is best-effort
|
||||
* (Windows LadybugDB handle release can lag; `cleanupTempDir` retries).
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import { writeFile, readFile, copyFile, mkdir } from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
getStoragePaths,
|
||||
saveMeta,
|
||||
loadMeta,
|
||||
INCREMENTAL_SCHEMA_VERSION,
|
||||
type RepoMeta,
|
||||
} from '../../src/storage/repo-manager.js';
|
||||
import { createTempDir } from '../helpers/test-db.js';
|
||||
|
||||
const HERE = path.dirname(fileURLToPath(import.meta.url));
|
||||
const FIXTURE_SRC = path.resolve(HERE, '..', 'fixtures', 'mini-repo', 'src');
|
||||
|
||||
/**
|
||||
* Copy the mini-repo fixture into a fresh git-initialized temp directory.
|
||||
* Returns the temp handle so the caller owns cleanup.
|
||||
*/
|
||||
async function setupMiniRepo(): Promise<{ dbPath: string; cleanup: () => Promise<void> }> {
|
||||
const tmp = await createTempDir('gitnexus-incr-orch-');
|
||||
const dest = path.join(tmp.dbPath, 'src');
|
||||
await mkdir(dest, { recursive: true });
|
||||
// Copy mini-repo fixture files
|
||||
const names = [
|
||||
'index.ts',
|
||||
'handler.ts',
|
||||
'validator.ts',
|
||||
'formatter.ts',
|
||||
'middleware.ts',
|
||||
'logger.ts',
|
||||
'db.ts',
|
||||
];
|
||||
for (const n of names) {
|
||||
await copyFile(path.join(FIXTURE_SRC, n), path.join(dest, n));
|
||||
}
|
||||
execSync('git init', { cwd: tmp.dbPath, stdio: 'pipe' });
|
||||
execSync('git -c user.name=test -c user.email=t@t -c commit.gpgsign=false add -A', {
|
||||
cwd: tmp.dbPath,
|
||||
stdio: 'pipe',
|
||||
});
|
||||
execSync('git -c user.name=test -c user.email=t@t -c commit.gpgsign=false commit -q -m initial', {
|
||||
cwd: tmp.dbPath,
|
||||
stdio: 'pipe',
|
||||
});
|
||||
return tmp;
|
||||
}
|
||||
|
||||
describe('runFullAnalysis — incremental orchestration', () => {
|
||||
it('first run populates fileHashes + schemaVersion and clears incrementalInProgress on success', async () => {
|
||||
const repo = await setupMiniRepo();
|
||||
try {
|
||||
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
|
||||
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
|
||||
|
||||
const { storagePath } = getStoragePaths(repo.dbPath);
|
||||
const meta = await loadMeta(storagePath);
|
||||
expect(meta).not.toBeNull();
|
||||
expect(meta!.schemaVersion).toBe(INCREMENTAL_SCHEMA_VERSION);
|
||||
expect(meta!.fileHashes).toBeDefined();
|
||||
expect(Object.keys(meta!.fileHashes ?? {}).length).toBeGreaterThan(0);
|
||||
// Dirty flag MUST be cleared after a successful run.
|
||||
expect(meta!.incrementalInProgress).toBeUndefined();
|
||||
} finally {
|
||||
await repo.cleanup();
|
||||
}
|
||||
}, 180_000);
|
||||
|
||||
it('second run on unchanged state takes the alreadyUpToDate fast path', async () => {
|
||||
const repo = await setupMiniRepo();
|
||||
try {
|
||||
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
|
||||
const first = await runFullAnalysis(
|
||||
repo.dbPath,
|
||||
{ skipAgentsMd: true },
|
||||
{ onProgress: () => {} },
|
||||
);
|
||||
expect(first.alreadyUpToDate).toBeUndefined();
|
||||
|
||||
const second = await runFullAnalysis(
|
||||
repo.dbPath,
|
||||
{ skipAgentsMd: true },
|
||||
{ onProgress: () => {} },
|
||||
);
|
||||
// lastCommit==HEAD && working tree clean (mod GitNexus output) →
|
||||
// early-return fast path.
|
||||
expect(second.alreadyUpToDate).toBe(true);
|
||||
} finally {
|
||||
await repo.cleanup();
|
||||
}
|
||||
}, 300_000);
|
||||
|
||||
it('second run after a comment-only edit takes the incremental path, clears the dirty flag, and preserves graph stats exactly', async () => {
|
||||
const repo = await setupMiniRepo();
|
||||
try {
|
||||
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
|
||||
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
|
||||
const { storagePath } = getStoragePaths(repo.dbPath);
|
||||
const firstMeta = await loadMeta(storagePath);
|
||||
|
||||
// Modify a source file with a COMMENT-ONLY edit — by construction
|
||||
// this changes the content hash (driving the incremental code path)
|
||||
// without changing any symbol, scope binding, call edge, import,
|
||||
// or community membership. Therefore every graph-stat invariant
|
||||
// (files / nodes / edges / communities / processes) MUST be
|
||||
// bit-identical to the first run. Anything else is a regression.
|
||||
const target = path.join(repo.dbPath, 'src', 'logger.ts');
|
||||
const before = await readFile(target, 'utf-8');
|
||||
await writeFile(target, before + '\n// touched by test\n', 'utf-8');
|
||||
|
||||
const second = await runFullAnalysis(
|
||||
repo.dbPath,
|
||||
{ skipAgentsMd: true },
|
||||
{ onProgress: () => {} },
|
||||
);
|
||||
// The early-return alreadyUpToDate path must NOT fire (the dirty
|
||||
// tree should kick the run through to incremental writeback).
|
||||
expect(second.alreadyUpToDate).toBeUndefined();
|
||||
|
||||
const secondMeta = await loadMeta(storagePath);
|
||||
expect(secondMeta).not.toBeNull();
|
||||
// Dirty flag must be cleared on success.
|
||||
expect(secondMeta!.incrementalInProgress).toBeUndefined();
|
||||
// fileHashes[logger.ts] must have rotated to the new content.
|
||||
expect(secondMeta!.fileHashes?.['src/logger.ts']).toBeDefined();
|
||||
expect(secondMeta!.fileHashes?.['src/logger.ts']).not.toBe(
|
||||
firstMeta!.fileHashes?.['src/logger.ts'],
|
||||
);
|
||||
// Exact-equality stats invariant. DoD §2.7: avoid bounds-only
|
||||
// assertions that would mask a regression dropping half the graph.
|
||||
expect(secondMeta!.stats?.files).toBe(firstMeta!.stats?.files);
|
||||
expect(secondMeta!.stats?.nodes).toBe(firstMeta!.stats?.nodes);
|
||||
expect(secondMeta!.stats?.edges).toBe(firstMeta!.stats?.edges);
|
||||
expect(secondMeta!.stats?.communities).toBe(firstMeta!.stats?.communities);
|
||||
expect(secondMeta!.stats?.processes).toBe(firstMeta!.stats?.processes);
|
||||
} finally {
|
||||
await repo.cleanup();
|
||||
}
|
||||
}, 300_000);
|
||||
|
||||
it('incremental output is byte-equivalent to a full rebuild (incremental ≡ --force on the same repo state)', async () => {
|
||||
// The central correctness contract of this PR: an incremental run
|
||||
// and a full rebuild from the same repo state must produce identical
|
||||
// graph stats. We exercise it end-to-end:
|
||||
//
|
||||
// 1. setup mini-repo + run analyze (populates the index)
|
||||
// 2. edit one source file (comment-only — same graph)
|
||||
// 3. run incremental analyze → record secondMeta
|
||||
// 4. run analyze --force from the same state → record forceMeta
|
||||
// 5. assert every stats invariant is exactly equal.
|
||||
//
|
||||
// Steps 3 and 4 share the same on-disk file contents, so any
|
||||
// divergence is purely an artifact of the writeback strategy. If
|
||||
// any invariant differs, the PR's load-bearing claim is violated.
|
||||
const repo = await setupMiniRepo();
|
||||
try {
|
||||
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
|
||||
|
||||
// Step 1: initial index.
|
||||
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
|
||||
|
||||
// Step 2: comment-only edit, same as the test above.
|
||||
const target = path.join(repo.dbPath, 'src', 'logger.ts');
|
||||
const original = await readFile(target, 'utf-8');
|
||||
await writeFile(target, original + '\n// equivalence test touch\n', 'utf-8');
|
||||
|
||||
// Step 3: incremental writeback for the edited file.
|
||||
const incremental = await runFullAnalysis(
|
||||
repo.dbPath,
|
||||
{ skipAgentsMd: true },
|
||||
{ onProgress: () => {} },
|
||||
);
|
||||
expect(incremental.alreadyUpToDate).toBeUndefined();
|
||||
const { storagePath } = getStoragePaths(repo.dbPath);
|
||||
const secondMeta = await loadMeta(storagePath);
|
||||
expect(secondMeta).not.toBeNull();
|
||||
|
||||
// Step 4: force a full rebuild from the SAME on-disk file state.
|
||||
const forced = await runFullAnalysis(
|
||||
repo.dbPath,
|
||||
{ skipAgentsMd: true, force: true },
|
||||
{ onProgress: () => {} },
|
||||
);
|
||||
expect(forced.alreadyUpToDate).toBeUndefined();
|
||||
const forceMeta = await loadMeta(storagePath);
|
||||
expect(forceMeta).not.toBeNull();
|
||||
|
||||
// Step 5: exact-equality across every stat. `toEqual` would also
|
||||
// work but `toBe` per-field makes a failure pinpoint the field.
|
||||
expect(secondMeta!.stats?.files).toBe(forceMeta!.stats?.files);
|
||||
expect(secondMeta!.stats?.nodes).toBe(forceMeta!.stats?.nodes);
|
||||
expect(secondMeta!.stats?.edges).toBe(forceMeta!.stats?.edges);
|
||||
expect(secondMeta!.stats?.communities).toBe(forceMeta!.stats?.communities);
|
||||
expect(secondMeta!.stats?.processes).toBe(forceMeta!.stats?.processes);
|
||||
} finally {
|
||||
await repo.cleanup();
|
||||
}
|
||||
}, 600_000);
|
||||
|
||||
it('a stale incrementalInProgress flag at startup forces a full rebuild that clears it', async () => {
|
||||
const repo = await setupMiniRepo();
|
||||
try {
|
||||
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
|
||||
// First run lays down a normal index.
|
||||
await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} });
|
||||
|
||||
// Manually corrupt meta.json with a stale dirty flag — simulates
|
||||
// a crashed previous incremental run.
|
||||
const { storagePath } = getStoragePaths(repo.dbPath);
|
||||
const meta = await loadMeta(storagePath);
|
||||
expect(meta).not.toBeNull();
|
||||
const tampered: RepoMeta = {
|
||||
...meta!,
|
||||
incrementalInProgress: {
|
||||
startedAt: Date.now() - 60_000,
|
||||
toWriteCount: 3,
|
||||
},
|
||||
};
|
||||
await saveMeta(storagePath, tampered);
|
||||
|
||||
// Next run must detect the flag, force a full rebuild (which
|
||||
// overwrites meta), and clear the flag.
|
||||
const recovered = await runFullAnalysis(
|
||||
repo.dbPath,
|
||||
{ skipAgentsMd: true },
|
||||
{ onProgress: () => {} },
|
||||
);
|
||||
// A full rebuild was taken — the alreadyUpToDate fast path
|
||||
// explicitly cannot fire because the dirty-flag check rewrote
|
||||
// `options.force` to true.
|
||||
expect(recovered.alreadyUpToDate).toBeUndefined();
|
||||
|
||||
const after = await loadMeta(storagePath);
|
||||
expect(after!.incrementalInProgress).toBeUndefined();
|
||||
} finally {
|
||||
await repo.cleanup();
|
||||
}
|
||||
}, 300_000);
|
||||
});
|
||||
243
gitnexus/test/unit/incremental-parse-cache.test.ts
Normal file
243
gitnexus/test/unit/incremental-parse-cache.test.ts
Normal file
|
|
@ -0,0 +1,243 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { mkdtemp, rm } from 'fs/promises';
|
||||
import { tmpdir } from 'os';
|
||||
import path from 'path';
|
||||
import {
|
||||
PARSE_CACHE_VERSION,
|
||||
computeChunkHash,
|
||||
fileContentHash,
|
||||
loadParseCache,
|
||||
saveParseCache,
|
||||
pruneCache,
|
||||
type ParseCache,
|
||||
} from '../../src/storage/parse-cache.js';
|
||||
import type { ParseWorkerResult } from '../../src/core/ingestion/workers/parse-worker.js';
|
||||
|
||||
const minimalResult = (overrides: Partial<ParseWorkerResult> = {}): ParseWorkerResult => ({
|
||||
nodes: [],
|
||||
relationships: [],
|
||||
symbols: [],
|
||||
imports: [],
|
||||
calls: [],
|
||||
assignments: [],
|
||||
heritage: [],
|
||||
routes: [],
|
||||
fetchCalls: [],
|
||||
decoratorRoutes: [],
|
||||
toolDefs: [],
|
||||
ormQueries: [],
|
||||
constructorBindings: [],
|
||||
fileScopeBindings: [],
|
||||
parsedFiles: [],
|
||||
skippedLanguages: {},
|
||||
fileCount: 0,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('computeChunkHash', () => {
|
||||
it('produces a stable hex hash for a fixed set of (filePath, contentHash) entries', () => {
|
||||
const entries = [
|
||||
{ filePath: 'a.ts', contentHash: 'h-a' },
|
||||
{ filePath: 'b.ts', contentHash: 'h-b' },
|
||||
{ filePath: 'c.ts', contentHash: 'h-c' },
|
||||
];
|
||||
const h1 = computeChunkHash(entries);
|
||||
const h2 = computeChunkHash(entries);
|
||||
expect(h1).toBe(h2);
|
||||
expect(h1).toMatch(/^[a-f0-9]{64}$/);
|
||||
});
|
||||
|
||||
it('is order-independent (same files in different order → same hash)', () => {
|
||||
const order1 = [
|
||||
{ filePath: 'a.ts', contentHash: 'h-a' },
|
||||
{ filePath: 'b.ts', contentHash: 'h-b' },
|
||||
];
|
||||
const order2 = [
|
||||
{ filePath: 'b.ts', contentHash: 'h-b' },
|
||||
{ filePath: 'a.ts', contentHash: 'h-a' },
|
||||
];
|
||||
expect(computeChunkHash(order1)).toBe(computeChunkHash(order2));
|
||||
});
|
||||
|
||||
it('changes when any file content changes', () => {
|
||||
const before = [
|
||||
{ filePath: 'a.ts', contentHash: 'h-a' },
|
||||
{ filePath: 'b.ts', contentHash: 'h-b' },
|
||||
];
|
||||
const after = [
|
||||
{ filePath: 'a.ts', contentHash: 'h-a' },
|
||||
{ filePath: 'b.ts', contentHash: 'h-b-NEW' }, // b.ts content changed
|
||||
];
|
||||
expect(computeChunkHash(before)).not.toBe(computeChunkHash(after));
|
||||
});
|
||||
|
||||
it('changes when chunk membership changes (file added or removed)', () => {
|
||||
const small = [
|
||||
{ filePath: 'a.ts', contentHash: 'h-a' },
|
||||
{ filePath: 'b.ts', contentHash: 'h-b' },
|
||||
];
|
||||
const bigger = [...small, { filePath: 'c.ts', contentHash: 'h-c' }];
|
||||
expect(computeChunkHash(small)).not.toBe(computeChunkHash(bigger));
|
||||
});
|
||||
});
|
||||
|
||||
describe('fileContentHash', () => {
|
||||
it('hashes a string deterministically', () => {
|
||||
expect(fileContentHash('hello')).toBe(fileContentHash('hello'));
|
||||
expect(fileContentHash('hello')).not.toBe(fileContentHash('hello!'));
|
||||
expect(fileContentHash('hello')).toMatch(/^[a-f0-9]{64}$/);
|
||||
});
|
||||
|
||||
it('handles Buffer input identical to its string form', () => {
|
||||
const s = 'sentinel';
|
||||
expect(fileContentHash(Buffer.from(s))).toBe(fileContentHash(s));
|
||||
});
|
||||
});
|
||||
|
||||
describe('PARSE_CACHE_VERSION', () => {
|
||||
it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => {
|
||||
// Looks like "1+1.6.4" — schema bump prefix + actual gitnexus version
|
||||
expect(PARSE_CACHE_VERSION).toMatch(/^\d+\+\d+\.\d+\.\d+/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('pruneCache', () => {
|
||||
it('drops entries whose hashes are not in the used-set', () => {
|
||||
const cache: ParseCache = {
|
||||
version: PARSE_CACHE_VERSION,
|
||||
entries: new Map<string, ParseWorkerResult[]>([
|
||||
['hash-A', [minimalResult()]],
|
||||
['hash-B', [minimalResult()]],
|
||||
['hash-C', [minimalResult()]],
|
||||
]),
|
||||
usedKeys: new Set<string>(['hash-A']),
|
||||
};
|
||||
const removed = pruneCache(cache, cache.usedKeys);
|
||||
expect(removed).toBe(2);
|
||||
expect([...cache.entries.keys()].sort()).toEqual(['hash-A']);
|
||||
});
|
||||
|
||||
it('returns 0 when every entry is in use', () => {
|
||||
const cache: ParseCache = {
|
||||
version: PARSE_CACHE_VERSION,
|
||||
entries: new Map<string, ParseWorkerResult[]>([
|
||||
['hash-A', [minimalResult()]],
|
||||
['hash-B', [minimalResult()]],
|
||||
]),
|
||||
usedKeys: new Set<string>(['hash-A', 'hash-B']),
|
||||
};
|
||||
expect(pruneCache(cache, cache.usedKeys)).toBe(0);
|
||||
expect(cache.entries.size).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadParseCache / saveParseCache (round-trip)', () => {
|
||||
it('round-trips an empty cache', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
||||
try {
|
||||
const cache: ParseCache = {
|
||||
version: PARSE_CACHE_VERSION,
|
||||
entries: new Map(),
|
||||
usedKeys: new Set(),
|
||||
};
|
||||
await saveParseCache(dir, cache);
|
||||
const loaded = await loadParseCache(dir);
|
||||
expect(loaded.version).toBe(PARSE_CACHE_VERSION);
|
||||
expect(loaded.entries.size).toBe(0);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('returns an empty cache when the file is missing', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
||||
try {
|
||||
const loaded = await loadParseCache(dir);
|
||||
expect(loaded.entries.size).toBe(0);
|
||||
expect(loaded.usedKeys.size).toBe(0);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('returns an empty cache on version mismatch (next-run regen)', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
||||
try {
|
||||
// Write a cache file with a different version directly
|
||||
const fs = await import('fs/promises');
|
||||
await fs.writeFile(
|
||||
path.join(dir, 'parse-cache.json'),
|
||||
JSON.stringify({ version: 'foreign-99', entries: { h: [] } }),
|
||||
'utf-8',
|
||||
);
|
||||
const loaded = await loadParseCache(dir);
|
||||
expect(loaded.entries.size).toBe(0); // mismatch → empty
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('returns an empty cache on corrupt JSON', async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
||||
try {
|
||||
const fs = await import('fs/promises');
|
||||
await fs.writeFile(path.join(dir, 'parse-cache.json'), '{not-json', 'utf-8');
|
||||
const loaded = await loadParseCache(dir);
|
||||
expect(loaded.entries.size).toBe(0);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('round-trips Map and Set values through the JSON replacer/reviver', async () => {
|
||||
// ParsedFile.scopes[*].typeBindings is a ReadonlyMap<string, TypeRef>.
|
||||
// Without the replacer/reviver pair, JSON.stringify collapses Maps to
|
||||
// {} and downstream code that does .get() / iterates entries crashes
|
||||
// with "is not iterable". This test pins the round-trip behaviour.
|
||||
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
||||
try {
|
||||
const innerMap = new Map<string, string>([
|
||||
['k1', 'v1'],
|
||||
['k2', 'v2'],
|
||||
]);
|
||||
const innerSet = new Set<string>(['s1', 's2']);
|
||||
// Stash the live Map/Set inside a synthetic ParseWorkerResult — we
|
||||
// only need the serializer to traverse them. Casting to bypass the
|
||||
// strict shape isn't a problem here: this test is about JSON
|
||||
// round-tripping of arbitrary nested Map/Set values, not full
|
||||
// ParseWorkerResult contents.
|
||||
const fake = minimalResult({
|
||||
parsedFiles: [
|
||||
{
|
||||
filePath: 't.ts',
|
||||
// Cast through unknown to satisfy the readonly Scope shape
|
||||
// while still smuggling a live Map into the serializer's
|
||||
// traversal path — see comment block above.
|
||||
scopes: [{ id: 's1', typeBindings: innerMap, extras: innerSet }],
|
||||
} as unknown as ParseWorkerResult['parsedFiles'][number],
|
||||
],
|
||||
});
|
||||
|
||||
const cache: ParseCache = {
|
||||
version: PARSE_CACHE_VERSION,
|
||||
entries: new Map<string, ParseWorkerResult[]>([['chunk-h', [fake]]]),
|
||||
usedKeys: new Set(['chunk-h']),
|
||||
};
|
||||
await saveParseCache(dir, cache);
|
||||
const loaded = await loadParseCache(dir);
|
||||
const reloaded = loaded.entries.get('chunk-h')?.[0];
|
||||
expect(reloaded).toBeDefined();
|
||||
const scope = (reloaded as ParseWorkerResult).parsedFiles[0]?.scopes[0] as unknown as {
|
||||
typeBindings?: unknown;
|
||||
extras?: unknown;
|
||||
};
|
||||
expect(scope.typeBindings).toBeInstanceOf(Map);
|
||||
expect((scope.typeBindings as Map<string, string>).get('k1')).toBe('v1');
|
||||
expect((scope.typeBindings as Map<string, string>).size).toBe(2);
|
||||
expect(scope.extras).toBeInstanceOf(Set);
|
||||
expect((scope.extras as Set<string>).has('s2')).toBe(true);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
75
gitnexus/test/unit/incremental-shadow-candidates.test.ts
Normal file
75
gitnexus/test/unit/incremental-shadow-candidates.test.ts
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { shadowCandidatesFor } from '../../src/core/incremental/shadow-candidates.js';
|
||||
|
||||
describe('shadowCandidatesFor', () => {
|
||||
it('returns an empty list when the input has no recognised module extension', () => {
|
||||
expect(shadowCandidatesFor('README.md')).toEqual([]);
|
||||
expect(shadowCandidatesFor('src/foo')).toEqual([]);
|
||||
expect(shadowCandidatesFor('binary.so')).toEqual([]);
|
||||
});
|
||||
|
||||
it('enumerates same-basename / different-extension candidates (pattern a)', () => {
|
||||
const out = shadowCandidatesFor('src/foo/bar.ts');
|
||||
// All non-.ts module extensions on the same path should appear.
|
||||
expect(out).toContain('src/foo/bar.tsx');
|
||||
expect(out).toContain('src/foo/bar.js');
|
||||
expect(out).toContain('src/foo/bar.jsx');
|
||||
expect(out).toContain('src/foo/bar.mjs');
|
||||
expect(out).toContain('src/foo/bar.cjs');
|
||||
expect(out).toContain('src/foo/bar.d.ts');
|
||||
// ...but NOT the same .ts (you can't shadow yourself).
|
||||
expect(out).not.toContain('src/foo/bar.ts');
|
||||
});
|
||||
|
||||
it('enumerates directory-style index candidates (pattern b) for both path separators', () => {
|
||||
const out = shadowCandidatesFor('src/foo/bar.ts');
|
||||
// POSIX form
|
||||
expect(out).toContain('src/foo/bar/index.ts');
|
||||
expect(out).toContain('src/foo/bar/index.tsx');
|
||||
expect(out).toContain('src/foo/bar/index.js');
|
||||
// Windows form
|
||||
expect(out).toContain('src/foo/bar\\index.ts');
|
||||
expect(out).toContain('src/foo/bar\\index.js');
|
||||
});
|
||||
|
||||
it('enumerates bare-file shadows when the added file is a directory index (pattern c)', () => {
|
||||
const out = shadowCandidatesFor('src/foo/index.ts');
|
||||
// Adding foo/index.ts can shadow foo.{ext} (rare but real — converting
|
||||
// a single-file module into a directory module).
|
||||
expect(out).toContain('src/foo.ts');
|
||||
expect(out).toContain('src/foo.tsx');
|
||||
expect(out).toContain('src/foo.js');
|
||||
expect(out).toContain('src/foo.jsx');
|
||||
expect(out).toContain('src/foo.mjs');
|
||||
expect(out).toContain('src/foo.cjs');
|
||||
});
|
||||
|
||||
it('also handles the Windows-separator form of `foo\\index.ts`', () => {
|
||||
const out = shadowCandidatesFor('src\\foo\\index.ts');
|
||||
expect(out).toContain('src\\foo.ts');
|
||||
expect(out).toContain('src\\foo.tsx');
|
||||
expect(out).toContain('src\\foo.js');
|
||||
});
|
||||
|
||||
it('handles `.d.ts` as a single extension token (not `.ts`)', () => {
|
||||
// The longest-match scan in shadowCandidatesFor puts `.d.ts` first.
|
||||
// For `foo.d.ts`, the noExt portion is "foo" (not "foo.d"), so the
|
||||
// pattern (a) candidates should be the non-.d.ts module variants.
|
||||
const out = shadowCandidatesFor('types/foo.d.ts');
|
||||
expect(out).toContain('types/foo.ts');
|
||||
expect(out).toContain('types/foo.tsx');
|
||||
expect(out).toContain('types/foo.js');
|
||||
// Not the .d.ts itself.
|
||||
expect(out).not.toContain('types/foo.d.ts');
|
||||
});
|
||||
|
||||
it('deduplicates output (no candidate appears twice)', () => {
|
||||
const out = shadowCandidatesFor('src/foo/bar.ts');
|
||||
expect(out.length).toBe(new Set(out).size);
|
||||
});
|
||||
|
||||
it('never includes the input path itself', () => {
|
||||
const input = 'src/foo/bar.ts';
|
||||
expect(shadowCandidatesFor(input)).not.toContain(input);
|
||||
});
|
||||
});
|
||||
169
gitnexus/test/unit/incremental-subgraph-extract.test.ts
Normal file
169
gitnexus/test/unit/incremental-subgraph-extract.test.ts
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
/**
|
||||
* Tests for incremental DB writeback subgraph extraction.
|
||||
*
|
||||
* Locks the Finding 1 fix (PR #1479 review): cross-file edges between
|
||||
* two unchanged files MUST land in the writeback subgraph when a third
|
||||
* (changed) file alters their cross-file resolution. The pre-fix
|
||||
* behaviour silently dropped those edges, leaving stale rows in the DB.
|
||||
*
|
||||
* These tests use synthetic graphs constructed via createKnowledgeGraph
|
||||
* directly — they don't run the parser, so they're cheap and stable.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import type { GraphNode, GraphRelationship } from 'gitnexus-shared';
|
||||
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
|
||||
import {
|
||||
extractChangedSubgraph,
|
||||
computeEffectiveWriteSet,
|
||||
} from '../../src/core/incremental/subgraph-extract.js';
|
||||
|
||||
const makeFileNode = (id: string, filePath: string, label = 'Function'): GraphNode =>
|
||||
({
|
||||
id,
|
||||
label,
|
||||
properties: { filePath, name: id },
|
||||
}) as unknown as GraphNode;
|
||||
|
||||
const makeWideNode = (id: string, label: 'Community' | 'Process'): GraphNode =>
|
||||
({
|
||||
id,
|
||||
label,
|
||||
properties: {},
|
||||
}) as unknown as GraphNode;
|
||||
|
||||
const makeRel = (
|
||||
id: string,
|
||||
sourceId: string,
|
||||
targetId: string,
|
||||
type = 'CALLS',
|
||||
): GraphRelationship =>
|
||||
({
|
||||
id,
|
||||
sourceId,
|
||||
targetId,
|
||||
type,
|
||||
properties: {},
|
||||
}) as unknown as GraphRelationship;
|
||||
|
||||
describe('extractChangedSubgraph', () => {
|
||||
it('includes nodes whose filePath is in the explicit toWriteSet', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('a', '/repo/a.ts'));
|
||||
g.addNode(makeFileNode('c', '/repo/c.ts'));
|
||||
|
||||
const sub = extractChangedSubgraph(g, new Set(['/repo/c.ts']));
|
||||
|
||||
expect(sub.nodes.map((n) => n.id).sort()).toEqual(['c']);
|
||||
});
|
||||
|
||||
it('always includes graph-wide nodes (Community, Process)', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('a', '/repo/a.ts'));
|
||||
g.addNode(makeWideNode('comm-1', 'Community'));
|
||||
g.addNode(makeWideNode('proc-1', 'Process'));
|
||||
|
||||
const sub = extractChangedSubgraph(g, new Set([])); // no files changed
|
||||
|
||||
expect(sub.nodes.map((n) => n.id).sort()).toEqual(['comm-1', 'proc-1']);
|
||||
});
|
||||
|
||||
it('includes a relationship when at least one endpoint is writable', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('a:fn', '/repo/a.ts'));
|
||||
g.addNode(makeFileNode('c:fn', '/repo/c.ts'));
|
||||
g.addRelationship(makeRel('e1', 'a:fn', 'c:fn', 'CALLS'));
|
||||
|
||||
// toWriteSet already includes A (the orchestrator expanded it via
|
||||
// computeEffectiveWriteSet) — both endpoints writable, edge fires.
|
||||
const sub = extractChangedSubgraph(g, new Set(['/repo/a.ts', '/repo/c.ts']));
|
||||
|
||||
expect(sub.nodes.map((n) => n.id).sort()).toEqual(['a:fn', 'c:fn']);
|
||||
expect(sub.relationships.map((r) => r.id)).toEqual(['e1']);
|
||||
});
|
||||
|
||||
it('skips a relationship entirely between unchanged files', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('x:fn', '/repo/x.ts'));
|
||||
g.addNode(makeFileNode('y:fn', '/repo/y.ts'));
|
||||
g.addRelationship(makeRel('e1', 'x:fn', 'y:fn', 'CALLS'));
|
||||
|
||||
const sub = extractChangedSubgraph(g, new Set(['/repo/c.ts']));
|
||||
|
||||
expect(sub.nodes).toEqual([]);
|
||||
expect(sub.relationships).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeEffectiveWriteSet (Finding 1)', () => {
|
||||
it('barrel re-export — expands the writable set to the consumer file', () => {
|
||||
// Scenario: file C (a barrel) used to re-export from B; now re-exports
|
||||
// from D. File A is unchanged byte-wise but its CALLS to foo() now
|
||||
// resolve to D instead of B. Both A and D are unchanged at the file
|
||||
// level — but A's edges have shifted.
|
||||
//
|
||||
// Pre-fix: toWriteSet={C} → A's nodes not deleted, A→D edge not
|
||||
// inserted (neither endpoint writable). DB ends up with
|
||||
// stale A→B and missing A→D.
|
||||
// Post-fix: the new graph has A→C (A still imports the barrel), so
|
||||
// A crosses the writable boundary and joins the effective
|
||||
// write set. deleteNodesForFile(A) then clears the stale
|
||||
// rows and the subgraph carries the new A→D edge.
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('a:fn', '/repo/a.ts'));
|
||||
g.addNode(makeFileNode('b:fn', '/repo/b.ts'));
|
||||
g.addNode(makeFileNode('c:re-export', '/repo/c.ts'));
|
||||
g.addNode(makeFileNode('d:fn', '/repo/d.ts'));
|
||||
g.addRelationship(makeRel('e1', 'a:fn', 'c:re-export', 'IMPORTS'));
|
||||
g.addRelationship(makeRel('e2', 'a:fn', 'd:fn', 'CALLS'));
|
||||
|
||||
const effective = computeEffectiveWriteSet(g, new Set(['/repo/c.ts']));
|
||||
|
||||
expect([...effective].sort()).toEqual(['/repo/a.ts', '/repo/c.ts']);
|
||||
});
|
||||
|
||||
it('picks up edges pointing INTO the changed file (symmetric case)', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('b:fn', '/repo/b.ts'));
|
||||
g.addNode(makeFileNode('c:fn', '/repo/c.ts'));
|
||||
g.addRelationship(makeRel('e1', 'b:fn', 'c:fn', 'CALLS'));
|
||||
|
||||
const effective = computeEffectiveWriteSet(g, new Set(['/repo/c.ts']));
|
||||
|
||||
expect([...effective].sort()).toEqual(['/repo/b.ts', '/repo/c.ts']);
|
||||
});
|
||||
|
||||
it('does not expand when no edge crosses the writable boundary', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('x:fn', '/repo/x.ts'));
|
||||
g.addNode(makeFileNode('y:fn', '/repo/y.ts'));
|
||||
g.addRelationship(makeRel('e1', 'x:fn', 'y:fn', 'CALLS'));
|
||||
|
||||
const effective = computeEffectiveWriteSet(g, new Set(['/repo/c.ts']));
|
||||
|
||||
expect([...effective].sort()).toEqual(['/repo/c.ts']);
|
||||
});
|
||||
|
||||
it('ignores edges to graph-wide nodes (no filePath)', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('a:fn', '/repo/a.ts'));
|
||||
g.addNode(makeWideNode('comm-1', 'Community'));
|
||||
g.addRelationship(makeRel('e1', 'a:fn', 'comm-1', 'BELONGS_TO'));
|
||||
|
||||
const effective = computeEffectiveWriteSet(g, new Set(['/repo/a.ts']));
|
||||
|
||||
expect([...effective].sort()).toEqual(['/repo/a.ts']);
|
||||
});
|
||||
|
||||
it('does not mutate the input set', () => {
|
||||
const g = createKnowledgeGraph();
|
||||
g.addNode(makeFileNode('a:fn', '/repo/a.ts'));
|
||||
g.addNode(makeFileNode('c:fn', '/repo/c.ts'));
|
||||
g.addRelationship(makeRel('e1', 'a:fn', 'c:fn', 'CALLS'));
|
||||
|
||||
const input = new Set(['/repo/c.ts']);
|
||||
computeEffectiveWriteSet(g, input);
|
||||
|
||||
expect([...input]).toEqual(['/repo/c.ts']);
|
||||
});
|
||||
});
|
||||
Loading…
Add table
Reference in a new issue