* fix(ingestion): reduce parse-phase memory for huge repos (#1983)
Stop retaining full parse-cache chunks in RAM alongside the merged graph,
slim on-disk shards, defer worker ParsedFile emission for scope-resolver
languages, and add GITNEXUS_DEBUG_HEAP probes for OOM diagnosis.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ingestion): address #2038 tri-review findings (parse-phase memory)
Resolves the confirmed review findings on PR #2038:
- P1: thread exportedTypeMap through the sequential parse path
(processParsingSequential) so a no-worker run over a partially-warm
cache no longer silently drops the sequential-miss files' exported
types. Cache hits made exportedTypeMap.size > 0, suppressing the
end-of-loop buildExportedTypeMapFromGraph rebuild, but the sequential
path never populated the map. Regression test added (fails on the
pre-fix tree, passes after) plus a fully-sequential differential oracle.
- P2: saveParseCache builds its on-disk index from hashes actually
written/copied (writtenKeys), never a usedKeys hash whose shard write
or copy was skipped — no more phantom index entries.
- P2: add a unit test asserting SCOPE_RESOLUTION_LANGUAGES stays in sync
with SCOPE_RESOLVERS (asymmetric drift would lose a language's ParsedFile).
- Backfill cache coverage: loadParseCacheChunk missing/corrupt -> undefined,
pruneCache onDiskKeys branch, slim preserves nodes, saveParseCache
copy-evicted-shard round-trip.
- Cleanups: single-source heap-probe gating via isDebugHeapEnabled();
hoist the per-chunk mkdir in persistParseCacheChunk behind a
process-scoped Set; gate COBOL's unused worker-side ParsedFile
extraction (graph nodes still come from cobolPhase) while keeping
fileCount/progress unconditional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): remove dead worker-side ParsedFile extraction
After #2038 gated worker `ParsedFile` emission behind `!isScopeResolutionLanguage(language)`, and with all 16 SupportedLanguages registered in SCOPE_RESOLVERS, that gate was structurally always true — the worker already produced no ParsedFiles and scope-resolution re-extracts each file from source on the main thread (run.ts). Remove the now-dead machinery:
- Drop both worker `extractParsedFile` call-sites (tree-sitter processFileGroup + the standalone-provider branch) and the `result.parsedFiles.push`. The standalone branch keeps fileCount/onFileProcessed per file. `result.parsedFiles` stays declared but empty (field removal deferred).
- Remove the now-orphaned `scopeSourceKind` var + `ScopeCaptureSourceKind`/`extractParsedFile`/`isScopeResolutionLanguage` imports.
- Delete the consumerless `migrated-languages.ts` (isScopeResolutionLanguage + SCOPE_RESOLUTION_LANGUAGES) and its drift-guard test — parse-worker was their only importer. Also improves AGENTS.md "shared ingestion code must not name languages" compliance.
`extractParsedFile` and the scope-extractor-bridge stay (scope-resolution/run.ts + Vue resolver use them). Behavior-preserving: worker-sequential-parity passes before and after; tsc/eslint clean; no baseline/golden drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): worker-pool-only parsing; remove sequential parser (#1983)
Completes the #1983 huge-repo parse-OOM effort by making the worker pool
GitNexus's sole parse path.
Parallel serialization (the perf core): workers serialize their ParsedFiles to
a disk store in parallel and stream them back to scope-resolution, so the main
thread no longer re-parses every file (the tree-sitter native-memory leak that
caused the OOM). Adds chunk merge-pipelining + work-proportional chunk sizing so
the pool stays saturated.
Remove the sequential parser: `--workers 0`, `GITNEXUS_WORKER_POOL_SIZE=0`, and
`skipWorkers` now hard-error (no silent degrade — #1741); the small-repo
threshold no longer selects an in-process path; pool creation stays lazy /
cache-miss-gated so warm all-hit runs never spawn workers.
Worker-path parity fixes — removing sequential surfaced two pre-existing gaps
that tiny-fixture tests had masked by running below the worker threshold, both
fixed by carrying per-file metadata as DATA across the worker boundary (never
re-parsing on the main thread, preserving the OOM fix):
- C++: templateConstraints wired into worker node identity (SFINAE overload
disambiguation) + ADL / inline-namespace capture side-channel serialized
onto the ParsedFile.
- Kotlin: companion-scope side-channel serialized the same way (companion /
static dispatch).
Validation: tsc + build clean; full suite green (10,190 pass — the only
deterministic failures were the now-fixed C++/Kotlin worker-path gaps; the 2
remaining full-run failures are pre-existing load flakiness, green in
isolation); cpp-pipeline benchmark stays linear on a 1-worker pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ingestion): wire C static-linkage side-channel + ADL O(1) collect + tri-review cleanups (#1983)
Follow-up to the worker-pool-only refactor, from a tri-review of the parse path.
- C static-linkage side-channel (P1): cProvider had no collect/applyCaptureSideChannel,
so on the now-sole worker path C `static` file-local marks were lost across the worker
boundary -> false cross-file CALLS edges + over-broad #include wildcard visibility on
every C analysis (the Linux kernel is C). Mirror the C++/Kotlin wiring: serialize
`staticNames` per file onto ParsedFile.captureSideChannel and restore it on the main
thread (no re-parse). + a worker-path regression test (the existing c-static-isolation
fixture passed vacuously — its collision resolves via #include before the global
free-call fallback ever consults static-linkage).
- captureSideChannel `kind` discriminant: add `kind:'cpp'`/`kind:'c'` tags + guards
(Kotlin already had one) now that C/C++/Kotlin share the single generic field.
- Perf: collectCppAdlSideChannel scanned the whole argInfoBySite/noAdlSites maps per file
(O(F^2) per sub-batch, ~100M parseSiteKey calls at kernel scale). Add per-filePath
lockstep indexes -> O(1) collect; serialized snapshot byte-identical.
- Cleanups: inline the one-line processParsingWithWorkers wrapper into processParsing;
drop the always-empty WorkerExtractedData.calls/assignments/constructorBindings fields;
remove the voided astCache param from processParsing; refresh stale "sequential
fallback" JSDoc.
Validation: tsc + build clean; cpp 297/297, c 8/8 (incl. the new worker-path
static-linkage guard), typescript + parsedfile-store green; cpp ADL benchmark stays linear.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): index C/C++ #include resolution in finalize (O(n²)→O(n))
Kernel-scale C/C++ analysis ground in finalizeScopeModel because three
per-#include operations each did a full O(F) scan with no index — the
finalize O(n²) that surfaced once the #1983 parse-phase OOM was fixed:
- expand{C,Cpp}WildcardNames: parsedFiles.find() per wildcard edge → O(R·F)
- resolveImportTarget: new Set(allFilePaths) rebuilt per #include
- resolveCImportTarget: suffix-match scanned all workspace paths
Each is replaced with a WeakMap-per-pass index keyed on the stable
parsedFiles/allFilePaths references that scope-resolution run.ts passes
once per pass:
- Map<ScopeId,ParsedFile> for wildcard expansion (c/static-linkage.ts +
cpp/file-local-linkage.ts)
- memoized augmented header set (c/scope-resolver.ts + cpp/scope-resolver.ts)
- basename-bucketed suffix index in resolveCImportTarget (c/import-target.ts),
shared by C and C++ since resolveCppImportTarget delegates to it
Collapses the C/C++ finalize from O(R·F) to O(R+F). Pure-perf, byte-identical
edge output: 962 targeted tests green (490 C + 472 C/C++ scope-resolution);
the basename index preserves the exact endsWith('/'+target) match and the
fewest-path-components-then-lexicographic tie-break.
The kernel's ~25-30k .h headers are classified C++, so both providers must
be fixed. Proven on the Linux kernel: the C finalize completed
(sr-post-finalize lang=c → sr-end lang=c), which the pre-fix run never
reached in 16+ min of grinding.
Build-independent follow-ups (separate from this finalize fix), documented
for later: emitFreeCallFallback same-name buckets (emit phase),
buildGraphNodeLookup + precount global setup, the ParsedFile store-load,
the dart/go/ruby expand-wildcards .find siblings, and the ~26GB
scope-resolution memory floor (full kernel completion needs >~40GB RAM).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(bench): regenerate C scope-capture baseline for the #1983 c-static-linkage-worker fixture
bench/scope-capture/measure.mjs fingerprints emitCScopeCaptures over the
lang-resolution/c-* fixture corpus. The #1983 PR added the
c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c — the
worker-path static-linkage side-channel test) but did not regenerate the C
baseline, so `--check` has been red on this branch (main, lacking the
fixture, still matches 0de009b).
Pure fixture-corpus drift — no c/captures.ts or query change branch-vs-main,
existing fixtures' captures byte-identical (c-captures.test.ts 45/45),
scaling stays linear (~0.97). Regenerated: 0de009b -> 39f3a83. Bench now
PASS (14 languages). Unrelated to the finalize O(n²) fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): lower kernel-scale resident memory floor + setup cost
Reduce the scope-resolution resident-memory floor and setup throughput on
huge repos (Linux kernel), the wall that remains after #1983 (parse OOM) and
the finalize O(n^2) fix (b71c77b8). Five units; all preserve byte-identical
edge output (C fixture 177n/255e + c/cpp/cross-file/php/static-linkage suites
green, 619 tests).
U1 (src/cli/analyze.ts): RAM-aware auto heap-cap. Replace the hardcoded
16384MB cap with computeHeapCapMb = max(16384, floor(0.75*effectiveRAM)),
where effectiveRAM = min(os.totalmem(), process.constrainedMemory()) with the
unconstrained-sentinel guard. Add --max-semi-space-size=128 on the respawn.
A user-supplied NODE_OPTIONS heap still wins (no re-exec). Verified: 23973MB
on a 31964MB box, 16384 floor on small machines, cgroup-aware, sentinel safe.
U2 (src/storage/parsedfile-store.ts, .../pipeline/phase.ts): export forceGc()
and call it at the per-language eviction boundary, so a finished language's
ParsedFiles are reclaimed before the next language's store-load instead of
collected lazily under the next pass's allocation pressure (which at cap>=RAM
degrades into swap-thrash). Measured on a real drivers/net/ethernet run:
C 2113->894MB and C++ 1754->1057MB reclaimed at the boundary (no fragmentation
defeat). Answers the plan's Open Question 1.
U3 (src/storage/parsedfile-store.ts): intern def objects by nodeId in the load
reviver so a SymbolDefinition's three serialized copies (localDefs /
scope.ownedDefs / scope.bindings[].def) collapse to one shared object on load.
Per-shard def pool (a def's copies are shard-local). Measured ~42% off the
def-object retained heap (3->1; 1.8M->600k distinct objects on 600k defs).
U4 (.../passes/free-call-fallback.ts): memoize pickUniqueGlobalCallable's
post-filter candidate list per (name, callerFilePath), only when no per-caller
visibility filter applies (the list is then a pure function of name+file), so
repeated free calls of one name from a file reuse the same-name-bucket scan
instead of re-walking a potentially huge bucket per site. The cached array is
read-only-consumed by the .filter()-based arity/overload narrowers. Exported
pickUniqueGlobalCallable + buildGlobalCallableIndex and added an equivalence
test (memoized == un-memoized reference for every (name, file, arity),
including warm-cache repeats and cross-file file-local exclusion).
U5 (.../pipeline/phase.ts): replace the O(L*F) per-language precount + repeated
scannedFiles.filter() with a single O(F) partition-by-language pass; bracket
buildGraphNodeLookup with scope-setup-nodeLookup heap probes so the long setup
is no longer silent.
Plan: docs/plans/2026-06-06-001-perf-kernel-scope-resolution-memory-plan.md
(U6 out-of-core global index deferred). Note: the kernel's full C++ pass floor
(~20k headers + the 8.8GB graph) likely still exceeds 24GB by itself, which is
why U6 remains the only unit that clears the wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(test): match OOM-guidance e2e assertions to the U1 reworded hint
The analyze-heap-oom-e2e real-child-OOM test still asserted the pre-U1
wording ('...out of memory.' + a hardcoded 24576 cap). U1 reworded the hint
to mention the auto heap-cap and use a <MB> placeholder, so the three
toContain substrings no longer matched (the assertion at line 62 failed on
all platforms). Update them to the current message. The unit twin
(analyze-heap-respawn) was already updated in 85bfc216; this integration
test was missed by the targeted local run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(lbug): U6a — deterministic id-sorted graph output behind GITNEXUS_SORT_GRAPH_OUTPUT
First increment of U6 (out-of-core scope-resolution). Adds an optional
deterministic ordering of node + relationship CSV rows by their unique graph
id, behind GITNEXUS_SORT_GRAPH_OUTPUT (default OFF = today's graph-insertion
order, byte-identical — the iterator is returned untouched). With the flag ON
the CSV becomes a pure function of the node/edge SET rather than of emit order.
This is the structural enabler for the windowed/out-of-core resolve (U6b-U6d):
csv-generator.ts:518 currently iterates graph.iterRelationships() in insertion
order with NO terminal sort, so any deviation from parsedFiles-order emit would
change bytes. With U6a on, a windowed emit need only reproduce the same edge
SET, not the global insertion order — removing the single largest byte-identical
hazard from every later windowing step.
Verified: default off keeps the existing csv-pipeline suite byte-identical; on,
node rows are id-sorted and output is independent of graph insertion order
(set-build) with the same node/edge set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(storage): U6d foundation — disk-backed scope store + lazy ScopeTree
Adds scope-index-store.ts: persistScopeShards (per-file scope shards via the
proven mapReplacer + def-interning reviver) + DiskBackedScopeTree, a lazy
ScopeTree that serves getScope from a bounded LRU of decoded shards plus a small
resident skeleton (scopeId -> {shard, childIds, parent}). Exports
makeInterningReviver from parsedfile-store for reuse.
This is the contained, highest-risk mechanism of U6d (out-of-core scope
resolution): the emit passes reach the heavy per-Scope binding payload
(~17-20GB on the kernel) ONLY through scopeTree.getScope (a point lookup) and
getChildren — they never read parsed.scopes directly — so moving that payload to
disk behind getScope is transparent. Every consumer reads a Scope BY VALUE, so a
value-faithful disk round-trip is byte-identical to resolution.
Proven in isolation: DiskBackedScopeTree is value-identical to buildScopeTree
for getScope/getChildren/getParent/getAncestors/has/size across multiple files
and after LRU eviction, and preserves the def-identity collapse (ownedDefs[i]
=== binding.def). Nothing wires it yet (the resolution-pipeline integration is
the next increment) — zero production impact; default off.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): U6d integration — seal scopeTree to disk before emit (GITNEXUS_DISK_SCOPE_INDEX)
Wires the U6d out-of-core scope index into the live pipeline behind
GITNEXUS_DISK_SCOPE_INDEX (default OFF = byte-identical). When on:
- finalize-orchestrator builds a TransitionalScopeTree (validated, fully
resident) instead of buildScopeTree, so finalize/propagate/resolve are
unchanged.
- After resolve, before emit, run.ts seals it: persists the scopes to a
file-sharded scope-index-store, swaps the model's scopeTree to disk-backed
serving from the inside (the frozen bundle can't be reassigned, but the
wrapper nulls its own resident backing), and drops the heavy Scope.bindings
payload from all THREE holders — the model's tree (seal), the caller's
preExtractedParsedFiles, and run.ts's own parsedFiles (scope-stripped copies
for emit). Emit reads scopes only via scopeTree.getScope (a point lookup,
now disk-backed + LRU) — verified it never reads parsed.scopes.
Purpose: lower the per-language resident PEAK (kernel C pass ~20→~12 GB by
moving the ~8-9 GB scope payload to disk) so the analysis fits on smaller-RAM
machines. At >=24 GB the full kernel already fits with U1-U5 (U2's 8.7 GB
inter-language forceGc reclaim keeps each pass under cap) — empirically
confirmed — so this is the sub-24 GB lever, not needed at 24 GB.
Byte-identical evidence: DiskBackedScopeTree/TransitionalScopeTree return
value-identical scopes vs buildScopeTree (getScope/getChildren/getParent/
getAncestors, across files + after LRU eviction + post-seal); emit reads only
getScope + referenceSites; flag-off (394 tests) and flag-on-resident (91 tests)
resolver suites stay green; an end-to-end A/B on a 212-file C+cpp+rust subset
produced identical 17,444 nodes / 31,343 edges with the seal firing per language
(c: 410→141 MB reclaimed). Kernel-scale peak-drop measurement pending the
in-flight verdict run freeing memory.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): U6d — id-back workspaceIndex so the disk seal can reclaim scopes
The kernel run revealed the contained scopeTree seal didn't lower the heap:
WorkspaceResolutionIndex held Scope OBJECTS (classScopeByDefId / moduleScopeByFile),
built from every ParsedFile and live through emit, so the ~28k module + class
scopes stayed pinned past the seal (sr-seal-pre 17,583 -> sr-seal-post 17,771 MB,
no drop). It was the sole residual Scope-object holder (SemanticModel holds none).
Fix: classScopeByDefId / moduleScopeByFile become id-backed ScopeByKeyView
instances — a ReadonlyMap<K, Scope> facade over a K->ScopeId map + the scopeTree,
whose .get fetches via scopeTree.getScope(id). The index now pins only ids, so
once the tree seals to disk the scopes become collectible. Byte-identical: the
view returns the same Scope the resident tree holds (or a value-identical revived
one in disk mode), and iteration keeps the old insertion order. buildWorkspace
ResolutionIndex takes an optional scopeTree (live pipeline passes it); without it
(unit tests) the legacy direct Scope-object maps are returned unchanged.
Verified byte-identical: 733 tests across workspace-index / imported-return-types
/ c / cpp / cross-file / go / java. Kernel peak-drop re-measurement to follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): U6d — precompute exportedCallableByName (fix disk-getScope thrash)
The workspaceIndex id-backing freed the kernel scopes but exposed a throughput
collapse: findExportedDefByName's workspace fallback (walkers.ts:1019) scanned
EVERY module scope's bindings per unresolved free call, and under the U6d
disk-backed scopeTree each module-scope access faulted a shard in from disk —
lib ON went ~1min -> ~7.5min.
Fix: precompute the fallback result once into
WorkspaceResolutionIndex.exportedCallableByName (simpleName -> first module-local
callable def, first-file-wins — the exact semantics the scan returned), built
from the resident module-scope bindings at index-build time. findExportedDefByName
now does an O(1) lookup with zero disk reads.
Result: lib ON ~7.5min -> 21s (cache-warm), byte-identical 17,444/31,343; 758
tests green across workspace-index + c/cpp/cross-file/go/python.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: rename cryptic U-unit codes to descriptive names in comments
The plan-unit shorthand (U3/U4/U6a/U6d/...) was meaningless in the code.
Renamed in comments + test descriptions (no behavior change, byte-identical):
out-of-core scope index (was U6)
deterministic output (was U6a)
disk-backed scope seal (was U6d)
def-object interning (was U3)
free-call candidate cache (was U4)
Also renamed throughout the PR title/summary. Pushed commit messages keep
their original U-codes as historical record.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ingestion): durable ParsedFile shards for warm-cache coverage (#2038)
On a warm re-analyze where every chunk is a parse-cache HIT, no parse worker
runs, the run-scoped ParsedFile store is cleared at parse start, and the cached
ParseWorkerResult carries no ParsedFiles (the worker writes them to the store
and empties them from the message). Scope-resolution then found an empty store
and fell back to main-thread extractParsedFile — re-opening the #1983
tree-sitter native-leak OOM the disk store closes (abhigyanpatwari review on
parse-cache.ts).
Fix: workers ALSO write their ParsedFiles to a durable, content-addressed store
(parsedfile-cache/) keyed by chunk hash, mirroring the parse cache's lifecycle
(version-gated by PARSE_CACHE_VERSION, pruned in lockstep to the surviving
keys). On a warm hit the chunk's durable shards are byte-COPIED into the
run-scoped store (no re-parse, no re-serialize -> byte-identical), so
scope-resolution streams them exactly as on a cold run. A coherence gate
re-dispatches the worker whenever a cached chunk's durable shards are missing
(migration / pruned / version-stale) -- never the main-thread extract.
- worker-pool/parse-worker: thread chunkHash through dispatch->job->flush
(incl. split/requeue) so the worker tags its durable shard by content
- parsedfile-store: durable persist / restore / index / prune API (sibling
dir, never cleared per run); content-addressing makes stale reuse impossible
- parse-impl: load durable index, gate the cache hit on durable coverage,
restore on hit, dispatch chunkHash on miss
- run-analyze: prune+save the durable store to the parse cache's surviving keys
- saveParseCache returns its written keys (the durable keepKeys)
Verified on linux/lib: warm preExtractedHits = full coverage (520/207/1, zero
main-thread re-parse), byte-identical cold==warm (17,456n/31,353e), warm 8.5x
faster. New two-run + mixed-mode + coherence-gate regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ingestion): clear stale scope-index-store shards on each seal (#2038)
The disk-backed scope index writes sequential s<n>.json shards into a shared
<storagePath>/scope-index-store/ dir, with the index resetting per
persistScopeShards call. A seal that writes fewer shards than a previous one
(a later language with fewer files, or a re-run of a shrunken repo) left stale
tail shards on disk indefinitely -- never read by the disk-backed tree, but
multi-GB on kernel-scale repos.
Add clearScopeIndexStore() and clear at the start of persistScopeShards: the
previously sealed language has finished emit and been released before the next
seal runs, so its DiskBackedScopeTree never reads those shards again. Unit
tests: a stale prior-run shard is removed, a fewer-files re-seal leaves no tail
shards, and the helper is idempotent.
Addresses abhigyanpatwari review on run.ts (disk hygiene for the
GITNEXUS_DISK_SCOPE_INDEX path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ingestion): resolve FastAPI include_router(prefix=...) cross-file routes
FastAPI sub-route files declare paths via @router.<verb> while the entry
file mounts the router with app.include_router(<router>, prefix='/x').
Previously both the ingestion-layer Route graph nodes and the group-layer
ExtractedContract URLs lost the cross-file prefix, breaking provider <->
consumer matching.
Ingestion layer:
- parse-worker emits routerIncludes / routerImports + decoratorReceiver
- parsing-processor / parse-impl thread the new fields and aggregate
prefixesByModule across chunks; decorator routes whose receiver is
'router' are duplicated once per matching prefix
- routes.ts joins prefix via normalizeExtractedRoutePath
Group layer:
- HttpLanguagePlugin gains an optional prepareRepo() pre-pass and a
repoContext arg to scan(); python.ts builds prefixesByModule and
falls back to the bare path when no entry matches
- http-route-extractor caches one repoContext per plugin
Tests:
- 3 new http-route-extractor cases (attr / named-import / no-prefix)
- ParseWorkerResult literals in 3 test files updated to the new shape
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(ingestion,group): address PR #1877 review — relative imports, cross-package collisions, host names, ingestion tests
Follow-ups to the FastAPI `include_router(prefix=...)` cross-file fix
based on PR #1877's automated production-readiness review. Three
correctness gaps and one test coverage gap addressed:
1. Relative-import support in the worker regex (FINDING 2)
`FROM_IMPORT_ROUTER_RE` now accepts module paths starting with a
`.` (e.g. `from .calls import router as calls_router`). The
previous `[A-Za-z_][\w.]*` rejected leading dots and silently
dropped every relative-import Shape-B include — a real pattern
from the PR description's own motivating example. The matching
helpers now strip leading dots before keying so absolute and
relative imports collapse to the same module key.
2. Cross-package same-name module collisions (FINDING 3)
Two-tier module keying replaces the previous basename-only key:
• short key — `users` (file basename without `.py`)
• long key — `api/users` (parent dir + stem)
`prefixesByLongKey` is consulted first and only falls back to
`prefixesByShortKey` when no long-key match is available. Both
the ingestion pipeline (parse-impl.ts) and the group extractor
(http-patterns/python.ts) carry the same scheme so the graph
nodes and HTTP contracts agree on which prefix applies.
New protocol field `ExtractedRouterModuleAlias` (parse-worker →
parsing-processor → parse-impl) lets Shape-A
`<host>.include_router(<mod>.router, prefix='/x')` calls promote
to a long key when the same file imports `<mod>` via
`from <pkg> import <mod>`. Without this, `api/users.py` and
`admin/users.py` collided on the basename `users` and the admin
file's routes inherited the `/users` prefix that was only meant
for `api/users.py`.
3. Non-`app` host variable names (FINDING 4)
The group-layer `INCLUDE_ROUTER_*_PATTERNS` queries pinned the
host identifier to the literal `"app"` and dropped every
`application = FastAPI()` / `api = FastAPI()` pattern — the
constraint was redundant given that the call shape
(`include_router` invoked with a router argument and a
`prefix=` keyword) is already specific enough. The pin is
removed; the ingestion regex was already unrestricted.
4. Ingestion-layer regression tests (FINDING 1)
The previous PR added group-layer tests
(`http-route-extractor.test.ts`) but zero in-tree tests for the
ingestion path. Two new suites pin the
worker → parse-impl → routes flow:
- `test/unit/fastapi-router-bindings.test.ts` (23 cases):
`extractFastAPIRouterBindings()` is split into a stand-alone
module so it can be unit-tested without booting a worker
thread, then pinned for regex shape, two-tier key emission,
relative-import support, and negative cases.
- `test/integration/fastapi-prefix-pipeline.test.ts` (5 cases)
plus `test/fixtures/fastapi-prefix-app/` — runs the full
`runPipelineFromRepo()` against a realistic multi-package
fixture (containing both `api/users.py` and `admin/users.py`)
and inspects the resulting `Route` graph nodes for cross-file
prefix joining and absence of cross-package bleed.
Verification
- `npx tsc --noEmit`: pass
- PR-touched test suites (6 files / 117 cases): all green
- `npx prettier --check`: pass on touched files
- `npx eslint`: 0 errors on touched files
Cache / compatibility
The new `routerModuleAliases?` field on `ParseWorkerResult` and
`routerModuleAliases` on `WorkerExtractedData` are optional /
guarded with `?? []`, so historical parse-cache entries continue
to load without forced re-scan.
Refs PR #1877.
* refactor(ingestion): move fastapi-router-bindings out of workers/ — pure module, not a worker
Addresses @magyargergo's `CHANGES_REQUESTED` review on PR #1877:
> Sorry I just found that we are introducing a new worker in the PR.
`gitnexus/src/core/ingestion/workers/fastapi-router-bindings.ts` was a
**pure-function module** — it never imported `worker_threads` or
`parentPort`, never spawned a worker, and was never registered as a
worker entry. It was placed in `workers/` purely because it was split
out of `workers/parse-worker.ts` to make its functions unit-testable
without booting a worker thread (parse-worker is itself the worker
entry and cannot be loaded from the main thread).
To remove the misleading directory placement:
• The implementation moves to
`gitnexus/src/core/ingestion/route-extractors/fastapi-router-bindings.ts`,
alongside the other framework-specific route extractors (`expo`,
`nextjs`, `php`, `laravel`, `middleware`, `response-shapes`).
• `workers/parse-worker.ts` keeps a thin re-export so the worker
entry can keep using `extractFastAPIRouterBindings` directly. The
re-export now carries an explicit comment stating that the imported
file is **not** a worker and that the `workers/` directory
deliberately hosts only true worker entries (`parse-worker.ts`,
`worker-pool.ts`, `quarantine.ts`).
• The new file's leading docstring opens with "NOT A WORKER" and
explains why it exists where it does.
• The unit test (`test/unit/fastapi-router-bindings.test.ts`) is
updated to import from the new path.
No behaviour change. The function body, signatures, and exported types
are identical.
Verification
• `npx tsc --noEmit`: pass
• `npx tsc` (dist rebuild): pass
• `test/unit/fastapi-router-bindings.test.ts` (23 cases): all green
• `test/integration/fastapi-prefix-pipeline.test.ts` (5 cases): all green
• `test/unit/group/http-route-extractor.test.ts` (63 cases): all green
• `npx prettier --check` on touched files: pass
• `npx eslint` on touched files: 0 errors
Refs PR #1877.
* refactor(ingestion): drop parse-worker re-exports; consumers import router types directly from route-extractors
Addresses @magyargergo's two remaining review comments on PR #1877:
1. **`gitnexus/src/core/ingestion/workers/parse-worker.ts:247`** —
"Can you please remove them and update the call sites?"
The `export type { ExtractedRouterInclude, ExtractedRouterImport,
ExtractedRouterModuleAlias } from '../route-extractors/...'` block
in parse-worker.ts is gone. The remaining `import type {…}` is
purely local — used only to type the corresponding fields on
`ParseWorkerResult` below — and the leading comment now says so
explicitly ("this file does NOT re-export them"). The
`extractFastAPIRouterBindings` symbol is also no longer re-exported
from parse-worker.ts; it's still imported here so the worker entry
can call it per file, but downstream consumers must reach it via
`route-extractors/fastapi-router-bindings` directly.
Call sites updated:
- `gitnexus/src/core/ingestion/parsing-processor.ts`
- `gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts`
Both files now `import type { ExtractedRouterInclude,
ExtractedRouterImport, ExtractedRouterModuleAlias }` directly from
`route-extractors/fastapi-router-bindings.js`. The worker types
they still need (`ParseWorkerResult`, `ExtractedToolDef`, etc.)
keep coming from `workers/parse-worker.js`.
The unit + integration tests already imported from the new path,
so no test changes were required.
2. **`gitnexus/src/core/ingestion/parsing-processor.ts:168`** —
suggested simplification:
for (const item of result.routerIncludes ?? []) allRouterIncludes.push(item);
for (const item of result.routerImports ?? []) allRouterImports.push(item);
for (const item of result.routerModuleAliases ?? []) allRouterModuleAliases.push(item);
Applied verbatim. Replaces the previous `if (result.…) for …`
guards. The cache-compat semantics are unchanged — historical
parse-cache entries that lack these fields still load cleanly,
the new form just spells the fallback inline.
No behavior change, no tests touched, no public API change.
Verification
• `npx tsc --noEmit`: pass
• `npx tsc` (dist rebuild): pass
• PR-touched test suites (6 files / 117 cases): all green
• `npx prettier --check` on touched files: pass
• `npx eslint` on touched files: 0 errors
Refs PR #1877.
* refactor(ingestion): hoist fastapi-router-bindings type imports to top of parse-worker.ts
Move the `import type { ExtractedRouterInclude, ExtractedRouterImport,
ExtractedRouterModuleAlias }` block to the top of the file with the
other type imports, and drop the comment that previously sat next to
ExtractedDecoratorRoute.
---------
Co-authored-by: henry <zhangwei2017@unipus.cn>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
* docs: incremental indexing design spec
Captures the design agreed in brainstorming on 2026-05-10:
- Transitive importer closure with public-surface-change optimization
- Git-only change detection (non-git repos: full rebuild as today)
- New default behavior; --force opts out
- New hydratePhase + loadGraphFromLbug primitive
- Iterative closure expansion with parseCache reuse
- incrementalInProgress dirty flag for crash recovery
Prior art: PR #592 (zenprocess), PR #533 (davidbeesley),
PR #1146 (azeemshaik025) — referenced and credited.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(communities): seed Leiden RNG for deterministic community detection
The vendored Leiden algorithm defaults to Math.random for tie-breaking
and randomized walks, which produces non-deterministic community
assignments and modularity values across runs on the same graph.
Pass a seeded mulberry32 RNG (LEIDEN_SEED=0xC0DE) so:
- The same graph always produces the same partition
- Modularity values are reproducible
- Equivalence tests for incremental indexing can compare community
assignments byte-for-byte
This is foundational for the upcoming incremental-indexing feature
(see docs/superpowers/specs/2026-05-10-incremental-indexing-design.md)
where the correctness contract is incremental output ≡ full rebuild
output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(incremental): change-detection, surface signatures, closure expansion
Three new modules supporting the incremental-indexing pipeline:
* core/incremental/git-diff.ts — getChangedFilesSinceCommit() unions
'git diff lastCommit HEAD' (committed) with 'git status --porcelain'
(dirty tree). Renames flattened to delete(orig) + add(new). Throws
LastCommitMissingError when lastCommit is gone (caller falls back to
full rebuild).
* core/incremental/surface.ts — extractSurfaceSignature() produces a
stable hash of a file's publicly-visible symbols (functions, classes,
methods, interfaces, types, heritage). Body-only edits → same hash.
Signature/heritage changes → different hash. Drives the closure
scoping optimization.
* core/incremental/closure.ts — computeImporterClosure() iterative
fixpoint: parse each closure file, extract surface, query DB
importers, expand. Uses a parseCache so each file is parsed once.
Generic over TParseResult so closure logic is decoupled from the
pipeline's parse representation.
32 unit tests across the three modules. Tests cover edge cases:
clean tree, dirty-only, mixed, renames, deletes, multi-hop cascade,
cycle termination, surface invariance, etc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(lbug): loadGraphFromLbug, queryImporters, deleteAllCommunitiesAndProcesses
Three new primitives in lbug-adapter.ts to support incremental indexing:
* loadGraphFromLbug(graph, unchangedFilePaths) — streams all nodes for
files in the set across every hydratable node table (excludes
Community/Process — graph-wide, regenerated downstream). Then loads
edges where both endpoints belong to loaded nodes, excluding
MEMBER_OF / STEP_IN_PROCESS edges (also graph-wide).
FilePaths chunked at 200 per query to keep statement size bounded
on huge repos. Endpoint-level join filters by source-side filePath
in the query, target-side checked JS-side via the loadedNodeIds set.
* queryImporters(targetFilePath) — returns DISTINCT a.filePath where
a -[IMPORTS]-> b and b.filePath = target. Powers closure expansion:
when a changed file's surface signature changes, all its importers
must be re-parsed.
* deleteAllCommunitiesAndProcesses() — drops Community/Process nodes
(and their edges via DETACH DELETE) at the start of each incremental
run so the communities/processes phases regenerate them from the
fully-merged graph. Required for the 'Leiden runs on full graph'
correctness invariant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(pipeline): hydrate phase + parse-filter for incremental indexing
Wires the incremental-indexing infrastructure into the phase-based
pipeline. Three coordinated changes:
* New hydratePhase (deps: structure) — loads node/edge state for files
OUTSIDE ctx.options.filesToParse from the existing LadybugDB index.
Runs before parse so the parse phase can produce a partial graph
while downstream phases (mro, communities, processes) still see the
full graph. No-op in full-rebuild mode (filesToParse unset).
* PipelineOptions.filesToParse: optional ReadonlySet<string>. When
set, parse phase filters scanned files to this set; hydrate fills
the complement. Set by runFullAnalysis when it detects an eligible
incremental run; never set by callers directly.
* gitnexus-shared PipelinePhase enum: 'hydrate' added so progress
callbacks can report the new phase distinctly from 'structure'.
Phase order: scan → structure → hydrate → markdown,cobol → parse
→ routes,tools,orm → crossFile → scopeResolution → mro → communities
→ processes. Communities (Leiden) still runs on the full graph,
satisfying the correctness invariant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(analyze): incremental orchestrator branch + meta schema
Wires incremental indexing into runFullAnalysis. Highlights:
* RepoMeta schema extended: schemaVersion, surfaceSignatures, and
incrementalInProgress fields. INCREMENTAL_SCHEMA_VERSION = 1.
* core/incremental/file-hash.ts — v1 surface signature: SHA-256 of file
content. v2 will switch to a true surface-only signature (defined in
surface.ts) so body-only edits don't expand the closure. The plumbing
is signature-agnostic so the swap is local.
* core/incremental/orchestrator.ts — eligibility check, closure
computation (uses file-hash as the surface signal), dirty-flag
management, subgraph extraction, signature merge.
* run-analyze.ts adds:
- hasDirtyTree() check on the existing 'lastCommit==HEAD' early-exit
so an uncommitted edit triggers re-index (was a coarse equality
check before).
- incremental branch: try incremental first; fall through to full
rebuild on any setup failure or eligibility miss.
- runIncrementalBranch() — opens existing DB, deletes closure-file
rows + Community/Process, runs pipeline with filesToParse, writes
only the changed-subgraph back, refreshes FTS, updates meta with
new surfaceSignatures and clears the dirty flag.
- Full-rebuild path now populates surfaceSignatures + schemaVersion
in meta.json so the next run is eligible for incremental.
Crash recovery: incrementalInProgress is set BEFORE any DB mutation
and cleared on success by overwriting meta.json. A crash anywhere in
between leaves the flag set, and the next analyze run forces a full
rebuild (cheapest path back to a known-good index).
v1 limitation documented: body-only edits trigger 1-hop closure
expansion (content-hash signal). True surface-only optimization is
deferred to v2 — see design doc for the integration path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(incremental): drop invalid --no-renames=false from git diff
The flag --no-renames=false isn't valid git syntax (it's parsed as a
file path). Git's default rename detection is on; removing the flag
keeps that behavior.
Caught while running an end-to-end smoke test against a small fixture
repo: incremental setup failed with 'Command failed: git diff
--name-status -z --no-renames=false ...'. After the fix, the
incremental path runs cleanly: closure is computed, hydrate phase
loads unchanged-file state from DB, parse phase only re-parses files
in closure, and the writeback updates only changed nodes/edges.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert v1 incremental indexing (5 commits)
Reverts the v1 design that parsed only closure files into a fresh
graph and tried to hydrate the rest from DB. Real-repo equivalence
test failed: cross-file resolution operates on partial parse data
(closure files only), so CALLS edges that resolve through unchanged
files silently fall off. Diff against full rebuild on the same
edited state: -50 nodes, -425 edges, -5 communities, -48 processes.
Architecture pivot: switch to PR #533-style content-addressed parse
cache. Pipeline parses every file (cache-served when possible),
giving cross-file resolution full data, with DB writeback then
restricted to changed-file rows.
Reverts:
d4b9de47 fix(incremental): drop invalid --no-renames=false
f35f7634 feat(analyze): incremental orchestrator branch + meta schema
bc039686 feat(pipeline): hydrate phase + parse-filter
98bb893d feat(lbug): loadGraphFromLbug, queryImporters, ...
aa8d7ae3 feat(incremental): change-detection, surface signatures, closure
Kept:
d9e340b0 feat(communities): seed Leiden RNG (foundational)
8235ca36 docs: incremental indexing design spec (will be revised)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(analyze): incremental DB writeback (Option B)
Equivalence-preserving incremental analyze. The pipeline still parses
every file (correctness invariant: cross-file resolution / scope
resolution / MRO / community detection all need full graph data); the
saving comes from selectively replacing only changed-file rows in
LadybugDB instead of wiping and reloading the whole graph.
How it works:
* On every analyze, we hash all source files (SHA-256 of content) and
store the map in meta.json.fileHashes alongside schemaVersion.
* The next run loads the prior map and diffs:
- changed: content hash differs → file's DB rows replaced.
- added: not in prior map → file's DB rows inserted.
- deleted: in prior map but not on disk → file's DB rows dropped.
* If the diff is non-empty AND no --force / no schema mismatch / no
dirty flag, take the incremental path:
- Set incrementalInProgress dirty flag (BEFORE any DB mutation).
- Open existing DB (no wipe).
- deleteNodesForFile() for each changed/added/deleted file.
- deleteAllCommunitiesAndProcesses() — Leiden regenerates these.
- extractChangedSubgraph() from the in-memory ctx.graph: nodes whose
filePath is in the writable set + Community + Process + edges with
at least one endpoint in the writable set (edges entirely between
hydrated unchanged nodes are skipped — already in DB).
- loadGraphToLbug() on the subgraph. Unchanged-file rows in DB
untouched.
- Recreate FTS indexes.
- Update meta with new fileHashes; clear dirty flag.
* Otherwise full-rebuild path runs as before.
Crash recovery: incrementalInProgress is the dirty flag. Set before
destructive ops; cleared on success. Set on next-run startup → forces
full rebuild (cheapest path back to known-good).
Other changes:
* Dirty-tree gate on the existing 'lastCommit==HEAD' early-return:
uncommitted edits no longer slip through as 'already up to date'.
* deleteAllCommunitiesAndProcesses helper in lbug-adapter.
* Skip the embedding cache+restore cycle when willTryIncremental is
true — embeddings stay in DB; re-inserting them would PK-conflict.
End-to-end equivalence verified on this repo (993 files, 24K nodes):
incremental run produces byte-identical {nodes, edges, clusters,
flows} to a full rebuild from the same edited state.
Speedup is currently modest (~5% on this repo) because the parse
phase still runs in full. Parse-cache integration is a separate
follow-up that composes cleanly on top of this work.
See docs/superpowers/specs/2026-05-10-incremental-indexing-design.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(analyze): chunk-level parse cache for full incremental speedup
Composes with the incremental DB writeback (commit 27f3b49d) to deliver
the major-speedup half of incremental indexing. Previously, the parse
phase ran in full on every analyze; the speedup came purely from
selective DB rewriting. With this commit the parse phase also reuses
prior tree-sitter output for chunks whose contents haven't changed.
How it works:
* Cache layer (gitnexus/src/storage/parse-cache.ts):
- File: <repo>/.gitnexus/parse-cache.json. Versioned, atomic write.
- Key: chunk content hash = sha256(sorted(filePath:fileContentHash
for each file in chunk)).
- Value: ParseWorkerResult[] (raw worker output for the chunk,
pre-merge).
- Granularity: per chunk (~20MB byte-budget). A change to one file
invalidates only its chunk — typically 1 of ~50 on a 1000-file
repo (~98% cache hit ratio on a small edit).
* Worker contract (gitnexus/src/core/ingestion/parsing-processor.ts):
- Extracted the chunk-result merge loop into a public
mergeChunkResults() so the same logic applies to live worker
output AND replayed cache entries.
- processParsingWithWorkers / processParsing accept an optional
outRawResults out-parameter that captures worker output before
merging — used by parse-impl to populate the cache after a miss.
* Parse phase wiring (parse-impl.ts):
- For each chunk, compute its content hash (after reading file
contents). Cache hit → mergeChunkResults() on cached results,
skip the worker dispatch entirely. Cache miss → run workers
normally, capture raw results, store under the chunk hash.
- Cache mutations happen in-place on the ParseCache passed via
PipelineOptions.parseCache.
* Lifecycle (run-analyze.ts):
- loadParseCache() before pipeline runs.
- Cache passed via runPipelineFromRepo's PipelineOptions.
- saveParseCache() after the pipeline + DB writeback succeed.
Equivalence verified on this repo (993 files, 24K nodes):
Cold (no cache, full work): 141.1s
Warm cache + 1-file edit, incremental: 63.6s ← 55% speedup
Warm cache + 1-file edit, --force: 71.6s ← 49% speedup
All three runs produce byte-identical {nodes, edges, clusters,
flows}. The cache survives --force (content-addressed = always
correct), so even forced rebuilds get the parse-skip benefit.
Why chunk-level rather than per-file: workers process sub-batches and
emit aggregated ParseWorkerResults. Per-file granularity would require
restructuring the worker contract; chunk-level captures most of the
practical speedup with no worker-side changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf(parse-impl): smaller default chunk budget (20MB→2MB) for cache granularity
The parse cache is keyed at chunk granularity. With the previous 20MB
budget, a typical mid-size repo (e.g. this worktree at 9MB total
parseable source) fits in a single chunk — meaning ANY file change
invalidates the whole chunk and re-parses every file.
2MB default produces ~5x more chunks on the same input, so a one-file
edit invalidates ~1/N of cached chunks instead of the whole thing.
Cold-run overhead from more chunks is <5% (one extra serialization
pass per chunk).
Override via GITNEXUS_CHUNK_BYTE_BUDGET env var for benchmarking.
Measured on this repo (~9MB / 887 parseable files):
Cold (no cache): 143s
Warm cache, no source changes: 2s (early-return)
Warm cache + 1-file edit: 81s (~43% off cold)
Speedup is bounded by the scopeResolution phase (~58s flat regardless
of parse cache) and by GitNexus's own auto-writes during analyze
(AGENTS.md / .claude/skills/ etc. mutate between runs and invalidate
chunks containing them). Both are addressable in follow-ups.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): reuse worker-produced ParsedFile + stabilize chunk order
Two compounding optimizations that drop warm-cache analyze from
~134s to ~38s on a 1000-file repo (72% faster), and cold rebuild
from ~143s to ~86s (40% faster) by short-circuiting work that was
previously re-done.
1. SCOPE-RESOLUTION: REUSE WORKER PARSEDFILE
Previously, the scope-resolution phase re-parsed every file with
tree-sitter on the main thread (~58s on a 1000-file repo) because
worker-produced tree-sitter Trees can't cross the worker MessageChannel.
But the worker ALSO produces a artifact via
, which structured-clones fine — and it's exactly
what scope-resolution would re-derive. Threading those ParsedFiles
through the parse phase () into
( map) lets scope-
resolution skip its extract loop on a per-file basis.
The fast path is bounded only by per file (cheap
graph mutation). On this repo: scopeResolution went from 58s → 5s.
2. MAP-PRESERVING PARSE-CACHE SERIALIZATION
is a
which JSON.stringify collapses to . The first attempt at threading
parsedFiles through the parse cache crashed at runtime with
"importerModule.typeBindings is not iterable" because cached entries
came back as plain objects.
Added a JSON replacer/reviver pair in parse-cache.ts that round-trips
Map and Set instances through tagged plain objects (). Symmetric: save uses replacer, load uses reviver.
3. STABLE CHUNK ORDERING
The byte-budget chunker walked files in filesystem-scan order, which
on Windows isn't guaranteed to be stable across runs. Even with
identical source content, two scans could place files in different
chunks, shifting chunk hashes and causing 100% parse-cache misses.
Added a deterministic alphabetical sort on before
chunking. Chunk membership is now stable across runs, so a single-file
edit invalidates exactly one chunk, not all of them.
Measured on this repo (993 files, 24K nodes):
Cold rebuild: 86s (was 143s)
Warm cache, no source changes: 3s (early-return)
Warm cache + 1-file edit: 38s (was 134s)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(incremental): update spec + AGENTS.md + GUARDRAILS.md for shipped design
- Rewrite docs/superpowers/specs/2026-05-10-incremental-indexing-design.md
to describe the architecture that actually shipped (parse cache +
incremental DB writeback + scope-resolution short-circuit), with the
v1 hydrate-phase post-mortem preserved as historical context.
- AGENTS.md "Keeping the Index Fresh" section: note that incremental
is the new default and --force is the explicit opt-out; mention
the parse-cache file location and that it's safe to delete.
- GUARDRAILS.md Signs: add an "Index seems corrupt or incremental is
misbehaving" entry pointing users to --force as the manual escape
hatch (the dirty flag handles automatic recovery).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(autofix): apply prettier + eslint fixes via /autofix command
* fix(incremental): bugbot review + CI test failures
Bugbot (PR #1479):
- Medium: pruneCache was exported but never called -> cache grew
unbounded. Wire pruneCache into run-analyze before saveParseCache,
using a transient usedKeys Set on ParseCache that the parse phase
populates as it processes chunks.
- Low: willTryIncremental (pre-pipeline) and isIncremental
(post-pipeline) could desync, silently dropping embeddings on
mispredicted runs. Removed the prediction; the embedding cache
now loads unconditionally when shouldLoadCache is true. The
re-insert step gates on the actual isIncremental value to avoid
PK-conflicts when the incremental-writeback path keeps DB rows.
CI test failures:
- cli-e2e #1169 + run-analyze.test.ts #1233: my dirty-tree gate on
the lastCommit==HEAD early-return saw GitNexus's own auto-generated
outputs (.claude/, .cursor/, AGENTS.md, CLAUDE.md) as dirty,
perpetually defeating the up-to-date fast path. Extended the
pathspec exclusion to cover all auto-gen outputs, not just
.gitnexus/.
- ruby field-type disambig: my chunk-stability sort exposed a
pre-existing order-dependency in Ruby cross-file resolution
(`user.address.save -> Address#save` only resolves correctly when
user.rb parses before address.rb in some configurations). Removed
the sort. Filesystem ordering is stable enough in practice that
the parse cache still hits the common case; the pre-existing
fragility is left for a separate fix.
- pipeline-graph-golden: regenerated. Seeded Leiden RNG produces a
partition different from the previous Math.random snapshot.
- staleness `parallel calls` was a CI timing flake; passes locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(incremental): re-insert cached embeddings on incremental path
Bugbot re-review caught: deleteNodesForFile cascades to the
CodeEmbedding table (DELETE WHERE e.nodeId STARTS WITH ...), so
changed-file embedding rows are wiped along with their nodes. The
previous fix gated re-insert on `!isIncremental`, which silently
dropped those embeddings — a regression versus the full-rebuild path's
"preserve embeddings by default" guarantee.
Remove the `!isIncremental` gate. The per-batch try/catch already
handles the unchanged-file PK-conflict case ("some may fail if node
was removed, that's fine") with the same semantics, so re-inserting
the full cached set on incremental works:
- changed-file rows: deleted, then re-inserted from cache (preserved)
- unchanged-file rows: still in DB, re-insert PK-conflicts and is
silently ignored (existing rows are correct)
Cost: re-inserting ~24K embeddings on incremental when only a few
files changed — most are no-op conflicts. Bounded by batch size of
200; ~3-5s overhead. Worth it for correctness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(incremental): address Claude+Bugbot review findings + remove design doc
Addresses CHANGES_REQUESTED review on PR #1479:
1. Remove docs/superpowers/specs/2026-05-10-incremental-indexing-design.md
per maintainer request.
2. BLOCKER (Claude Finding 1, Bugbot Round 3): Stale cross-file edges
between unchanged files. extractChangedSubgraph excluded edges where
both endpoints were unchanged-file nodes — when a barrel/re-export
file changes, cross-file resolution may update CALLS edges between
two unchanged files that would then be silently lost.
Fix: 1-hop importer-closure expansion of the writable set in
run-analyze.ts. Before deleting/rewriting rows, query DB for
importers of every changed/deleted file and add them to the writable
set. Their nodes get deleted+rewritten too, so cross-file's refined
edges land in the DB. Re-added queryImporters to lbug-adapter.ts.
3. BLOCKER (Claude Finding 3): Parse cache key omitted parser version.
After a GitNexus upgrade, the cache silently replays pre-upgrade
ParseWorkerResults against the new schema → wrong CALLS/IMPORTS/
scope edges with no visible signal.
Fix: PARSE_CACHE_VERSION now embeds the gitnexus npm package
version (read at module load via createRequire on package.json).
Format: `${SCHEMA_BUMP}+${PKG_VERSION}` e.g. "1+1.6.4". Any release
that bumps package.json automatically invalidates the on-disk cache.
Mismatched versions fall through to an empty cache (next save
overwrites with the new version baked in).
4. BLOCKER (Claude Finding 2): No automated tests for incremental
behavior. Added 28 unit tests across 3 files:
- incremental-file-hash.test.ts (10 tests)
diffFileHashes classification, computeFileHash determinism,
computeFileHashes batch / missing-file tolerance, sorted output.
- incremental-parse-cache.test.ts (12 tests)
computeChunkHash stability and order-independence, version
prefix format, pruneCache, load/save round-trip on empty /
missing / corrupt / version-mismatched files, AND a Map/Set
round-trip test that pins the JSON replacer/reviver behaviour
(without it, ParsedFile.scopes[*].typeBindings collapses to
{} and downstream `.get()` / iteration throws).
- incremental-subgraph-extract.test.ts (6 tests)
writable-set node inclusion, Community/Process always kept,
edge inclusion when at least one endpoint is writable, MEMBER_OF
edges via graph-wide endpoints, empty subgraph case.
5. Medium (Claude Finding 6): AGENTS.md "Keeping the Index Fresh"
said "only changed files are re-parsed." Imprecise — the pipeline
parses every file every run; the cache skips tree-sitter for chunks
whose contents haven't changed. Reworded to match the design doc.
Test plan still expects:
[x] Typecheck clean
[x] All 28 new unit tests pass
[x] All previously-failing tests still pass on the rebased branch
[x] Equivalence verified locally (incremental ≡ --force, byte-identical
stats on this repo)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(incremental): round 3 review feedback — bounded BFS, atomic meta, integration test, docs
Addresses remaining findings on PR #1479 from Claude's re-review of
commit ad7bd31 + verifies the outstanding Bugbot HIGH severity.
1. F1 — Transitive importer expansion (Claude, was Medium-but-noted).
Previous 1-hop importer expansion missed barrel re-export chains
(A imports C, C re-exports B; when B changes, only C was pulled in
— A was left with potentially-stale CALLS edges to refined targets).
Replaced the single pass with a bounded BFS over the IMPORTS graph
(depth ≤ 4). Catches nested barrel pyramids without ballooning into
a near-full rebuild on monorepos with deep re-export trees. `--force`
remains the escape hatch documented in GUARDRAILS.md for cases that
exceed the bound.
2. F2 — Integration test for incremental orchestration (Claude, BLOCKER,
DoD §2.7). The unit tests added in ad7bd31 covered `diffFileHashes`,
`extractChangedSubgraph`, `computeChunkHash`, `pruneCache`, and the
Map/Set JSON round-trip — but none of them exercised the real
`runFullAnalysis` orchestration. Added gitnexus/test/unit/
incremental-orchestration.test.ts with four end-to-end tests against
a real git-initialized fixture repo + real LadybugDB:
a. First run populates fileHashes + schemaVersion and clears
incrementalInProgress on success.
b. Second run on unchanged state takes the alreadyUpToDate fast
path (early-return).
c. Second run after a source edit takes the incremental path
(not full rebuild) and rotates fileHashes for the touched file
while keeping the dirty flag cleared.
d. A pre-set incrementalInProgress flag forces a full rebuild
that clears it (crash-recovery wire).
These would catch any regression that wires `isIncremental` from a
pre-pipeline prediction (the Bugbot finding from commit 5eb0597) or
accidentally re-gates the embedding re-insert on `!isIncremental`
(the Bugbot finding from commit 60c10f1).
3. F3 — GUARDRAILS.md docs accuracy (Claude, Low). Line 33 still said
"only changed files are re-parsed" — AGENTS.md was already corrected
in ad7bd31 but GUARDRAILS.md was missed. Reworded to match.
4. F5 — Atomic saveMeta (Claude, Medium; vvladescu-tb fork). The dirty
flag (`incrementalInProgress`) travels through meta.json. A crash
mid-write would leave a corrupt meta.json that `loadMeta` would
silently treat as "no prior index", losing the flag and skipping
recovery. Switched to tmp-file + rename matching saveParseCache.
5. Bugbot's "Subgraph edges reference nodes absent from subgraph"
(HIGH severity). Verified as FALSE POSITIVE: `getNodeLabel` in
lbug-adapter.ts derives labels from the node-ID string (parses
the table prefix), not from the in-memory graph. The CSV
generator writes (src_id, dst_id, type) rows without consulting
node objects; `splitRelCsvByLabelPair` routes by ID-derived label;
`COPY ... (from=X, to=Y)` resolves both endpoints against the live
LadybugDB where unchanged-file nodes still exist. No fix needed.
All 213 tests pass locally (including the 4 new integration tests
and the previously-failing CI tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(incremental): address Bugbot round-4 findings (added-file shadow seed + dedupe)
Bugbot review on commit e23e4400 surfaced two new findings against the
incremental writeback in run-analyze.ts:
HIGH — Incremental BFS misses importers of newly added files.
queryImporters() reads the pre-pipeline DB. For a NEWLY ADDED
file there are no IMPORTS rows pointing to it yet, so unchanged
files whose pre-existing import statements now resolve to the
newcomer keep stale CALLS edges pointing at the OLD resolution
target.
LOW — Deleted files double-counted in filesToDelete.
hashDiff.deleted entries can reappear in writableFiles via the
BFS expansion (queryImporters can return a now-deleted path),
so deleteNodesForFile() ran twice for the same file.
Fixes:
- Add gitnexus/src/core/incremental/shadow-candidates.ts: derive
the pre-existing file paths whose JS/TS module-resolution claim
an added file can steal. Pattern catalogue: same-basename/
different-extension, bare-file-beats-directory-index, and
directory-index-beats-bare-file. Emit both POSIX and Windows
separators because the prior fileHashes map may have been
written from either OS.
- In run-analyze.ts, seed the BFS frontier with shadow candidates
that exist in the prior meta.fileHashes. Their importers — found
via queryImporters — get pulled into the writable set so their
CALLS edges re-resolve against the new file.
- Dedupe filesToDelete via Set to avoid the double-call.
Tests: gitnexus/test/unit/incremental-shadow-candidates.test.ts —
8 cases covering each shadow pattern, separator handling, .d.ts as
a single extension token, deduplication, and the no-self-shadow
invariant. All 40 incremental tests (file-hash, parse-cache,
subgraph-extract, shadow-candidates, orchestration) pass locally.
Note on the third Bugbot finding ("Subgraph edges reference nodes
absent from subgraph"): re-anchored from a prior review pass — the
code at subgraph-extract.ts:48 is unchanged. Already verified as a
false positive: getNodeLabel parses labels from ID strings, CSV
write is by ID, and COPY resolves against the live DB.
* chore(autofix): apply prettier + eslint fixes via /autofix command
* test(incremental): exact-equality stats invariant + analyze ≡ analyze --force
Addresses the only remaining Claude production-readiness review finding
on PR #1479 (Low-Medium, test-quality only — Claude itself said it does
NOT block merge, but the central PR claim "incremental ≡ full rebuild"
deserves explicit CI coverage rather than implicit trust).
Changes to gitnexus/test/unit/incremental-orchestration.test.ts:
1) Tighten the existing "comment-only edit takes incremental path" test.
- Replace toBeGreaterThan(0) bounds assertions on stats.files and
stats.nodes with exact toBe(firstMeta) per-field equality across
files / nodes / edges / communities / processes. DoD §2.7 calls
out bounds-only assertions as masking regressions that drop half
the graph; this swap closes that gap.
- Rationale: a comment-only edit must change the file content hash
(driving the incremental path) without changing any graph data.
Therefore every stat MUST be identical to the first run. Anything
else is a regression.
2) New test: incremental output is byte-equivalent to a full rebuild.
- Run analyze → comment-only edit → analyze (incremental writeback)
→ analyze --force (full rebuild from same on-disk state).
- Assert files / nodes / edges / communities / processes are exactly
equal across the incremental and the --force passes.
- This is the PR's central correctness contract, now proven by a
test that exercises the real runtime path end-to-end against a
real on-disk LadybugDB.
All 5 orchestration tests pass locally (52s), including the new
equivalence test — every stat field matches exactly between incremental
and --force on the mini-repo fixture.
tsc --noEmit clean.
* fix(incremental): F1 cross-file edge consistency + F4 stable chunk sort + unit coverage (#1511)
Patch addressing two of the still-open changes-requested findings on PR
#1479, rebased onto the current feat/incremental-indexing head. F3
(parser fingerprint in the cache key), F5 (atomic saveMeta), and F6
(AGENTS.md phrasing) were already handled on the branch, so the
corresponding parts of the original patch were dropped as redundant.
F1 (Blocker) — Cross-file edges between unchanged files
Adds `computeEffectiveWriteSet(graph, toWriteSet)` to
subgraph-extract.ts: a single pass over the new graph's edges that
pulls the unchanged-side file of every writable-boundary-crossing
edge into the write set. run-analyze composes it ON TOP of the
existing importer-BFS expansion and feeds the combined set to BOTH
`deleteNodesForFile` and `extractChangedSubgraph`, so the delete
cascade and the writeback subgraph cover identical files (asymmetry
would leave stale rows or PK-conflict at COPY time). The BFS reads
IMPORTS from the pre-pipeline DB (catches files that *stopped*
importing a changed file); the edge walk reads the new graph
(catches refined CALLS edges the pre-run DB couldn't predict, e.g.
a barrel re-export shifting a symbol from B to D). `extractChangedSubgraph`
stays a pure filter — all expansion is the orchestrator's job.
F4 (Medium) — Restore alphabetical chunk sort
`parseableScanned` is sorted before chunking. Filesystem-scan order
isn't stable enough across runs/platforms (notably macOS APFS) to
keep chunk hashes consistent, so the parse cache thrashes without
it. The pre-existing Ruby cross-file resolution order-dependency the
old comment cited is independent — the sort surfaces it but doesn't
cause it; tracked separately rather than leaving the cache cold.
Tests — incremental-subgraph-extract.test.ts
Locks the F1 invariants: `extractChangedSubgraph` is a pure filter
(includes only the set it's given, plus graph-wide nodes; edges
fire on one writable endpoint), and `computeEffectiveWriteSet`
covers the barrel-re-export scenario, the symmetric edge-into-
changed-file case, the no-boundary-crossed no-op, graph-wide-node
edges, and input-immutability. Supersedes the prior
extractChangedSubgraph-only test file on the branch.
Co-authored-by: Val Vladescu <vvladescu-tb@users.noreply.github.com>
* chore(autofix): apply prettier + eslint fixes via /autofix command
* fix(call-processor): register properties in pre-pass to fix order-dependent field type disambiguation + regenerate golden snapshot
Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2d66666f-861c-432e-a4b0-11f2aefca98a
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
* fix(call-processor): port worker-path property enrichment into the sequential pre-pass
Copilot's pre-pass in 8184439 fixed the Ruby attr_accessor order-dependence,
but it copied the OLD in-loop registration logic, not the canonical worker
path in parse-worker.ts. That left the sequential and worker paths emitting
non-identical Property nodes/symbols for the same source — silently breaking
the `incremental ≡ --force` invariant the moment a repo crosses the worker
threshold between runs.
Two concrete divergences are closed here:
* Node id: worker keys Property as `${file}:${className}.${propName}`
(qualified). Pre-pass was using `${file}:${propName}` (unqualified).
Same source produced different graph ids depending on which path ran.
* Field metadata: worker enriches each routed property with
`provider.fieldExtractor` + `getFieldInfo`, falling back to
`routedFieldInfo.type` for `declaredType` when the routing payload
lacks one (e.g. types discovered from `@address = Address.new`
ctor assignments rather than YARD `@return [Type]`), and propagates
`visibility` / `isStatic` / `isReadonly`. Pre-pass did none of this,
so on the sequential path `resolveFieldAccessType` failed to walk
chains where the type only came from the FieldExtractor.
The pre-pass now mirrors parse-worker.ts:1803-1898 verbatim, with one
deliberate difference: the FieldInfo cache is scoped to a single
`processCalls` invocation rather than module-level (the worker process
is short-lived; the main thread is not, and a module-level cache would
leak state between analyze runs).
Also drops the now-stale "Defer resolution: Ruby attr_accessor properties
are registered during this same loop" comment on `pendingWrites.push` —
the rationale is no longer accurate after Copilot's pre-pass, but the
deferral is still needed so write-access tracking sees inference that
completes during the main loop. Comment updated to reflect that.
Verification:
* `tsc --noEmit`: 0 errors
* test/unit (call-processor, call-routing, field-extraction, ruby-self-call): 224 passing
* test/integration (ruby, ruby-sequential-mixin, pipeline-graph-golden): 137 passing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(call-processor): key fieldInfoCache by filePath:startIndex, not raw byte offset
Claude's review of 255bdf6 caught a real collision in the FieldInfoCache I
added: keying by `classNode.startIndex` alone is a per-file byte offset, so
two files that both begin with a class at byte 0 — extremely common in Ruby /
Python, where files frequently open with `class Foo`, `module Foo` — collide
on the same cache entry. The second file's `getFieldInfo` then returns the
first file's FieldInfo map, producing wrong `declaredType` / `visibility` /
`isReadonly` on its properties.
Same shape as the bug that already exists in parse-worker.ts:377 (also keyed
by `classNode.startIndex` in a module-level map, persistent across files
processed by the same worker). Fixing the symmetric pre-existing leak in
parse-worker.ts is a separate, scoped follow-up — left out of this commit to
keep the fix minimal and reviewable.
Cache map and key are now both string-typed. Composite key
`${context.filePath}:${classNode.startIndex}` keeps the within-file hit rate
(one FieldExtractor.extract() per class regardless of how many
`attr_accessor` lines it has) while eliminating cross-file aliasing.
Verification on the patched HEAD:
* `tsc --noEmit`: 0 errors
* test/unit (call-processor, call-routing, field-extraction, ruby-self-call): 224 passing
* test/integration (ruby, ruby-sequential-mixin, pipeline-graph-golden): 137 passing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Val Vladescu <val.vladescu@thirdbridge.com>
Co-authored-by: Val Vladescu <vvladescu-tb@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
Gergő Magyar