GitNexus/gitnexus/test/integration/parse-impl-large-fixture.test.ts
Gergő Magyar 95f87fc12a
perf(ingestion): Linux-kernel-scale analysis — worker-pool parse + finalize O(n²) + scope-resolution memory wall (#1983) (#2038)
* fix(ingestion): reduce parse-phase memory for huge repos (#1983)

Stop retaining full parse-cache chunks in RAM alongside the merged graph,
slim on-disk shards, defer worker ParsedFile emission for scope-resolver
languages, and add GITNEXUS_DEBUG_HEAP probes for OOM diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ingestion): address #2038 tri-review findings (parse-phase memory)

Resolves the confirmed review findings on PR #2038:

- P1: thread exportedTypeMap through the sequential parse path
  (processParsingSequential) so a no-worker run over a partially-warm
  cache no longer silently drops the sequential-miss files' exported
  types. Cache hits made exportedTypeMap.size > 0, suppressing the
  end-of-loop buildExportedTypeMapFromGraph rebuild, but the sequential
  path never populated the map. Regression test added (fails on the
  pre-fix tree, passes after) plus a fully-sequential differential oracle.
- P2: saveParseCache builds its on-disk index from hashes actually
  written/copied (writtenKeys), never a usedKeys hash whose shard write
  or copy was skipped — no more phantom index entries.
- P2: add a unit test asserting SCOPE_RESOLUTION_LANGUAGES stays in sync
  with SCOPE_RESOLVERS (asymmetric drift would lose a language's ParsedFile).
- Backfill cache coverage: loadParseCacheChunk missing/corrupt -> undefined,
  pruneCache onDiskKeys branch, slim preserves nodes, saveParseCache
  copy-evicted-shard round-trip.
- Cleanups: single-source heap-probe gating via isDebugHeapEnabled();
  hoist the per-chunk mkdir in persistParseCacheChunk behind a
  process-scoped Set; gate COBOL's unused worker-side ParsedFile
  extraction (graph nodes still come from cobolPhase) while keeping
  fileCount/progress unconditional.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ingestion): remove dead worker-side ParsedFile extraction

After #2038 gated worker `ParsedFile` emission behind `!isScopeResolutionLanguage(language)`, and with all 16 SupportedLanguages registered in SCOPE_RESOLVERS, that gate was structurally always true — the worker already produced no ParsedFiles and scope-resolution re-extracts each file from source on the main thread (run.ts). Remove the now-dead machinery:

- Drop both worker `extractParsedFile` call-sites (tree-sitter processFileGroup + the standalone-provider branch) and the `result.parsedFiles.push`. The standalone branch keeps fileCount/onFileProcessed per file. `result.parsedFiles` stays declared but empty (field removal deferred).
- Remove the now-orphaned `scopeSourceKind` var + `ScopeCaptureSourceKind`/`extractParsedFile`/`isScopeResolutionLanguage` imports.
- Delete the consumerless `migrated-languages.ts` (isScopeResolutionLanguage + SCOPE_RESOLUTION_LANGUAGES) and its drift-guard test — parse-worker was their only importer. Also improves AGENTS.md "shared ingestion code must not name languages" compliance.

`extractParsedFile` and the scope-extractor-bridge stay (scope-resolution/run.ts + Vue resolver use them). Behavior-preserving: worker-sequential-parity passes before and after; tsc/eslint clean; no baseline/golden drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ingestion): worker-pool-only parsing; remove sequential parser (#1983)

Completes the #1983 huge-repo parse-OOM effort by making the worker pool
GitNexus's sole parse path.

Parallel serialization (the perf core): workers serialize their ParsedFiles to
a disk store in parallel and stream them back to scope-resolution, so the main
thread no longer re-parses every file (the tree-sitter native-memory leak that
caused the OOM). Adds chunk merge-pipelining + work-proportional chunk sizing so
the pool stays saturated.

Remove the sequential parser: `--workers 0`, `GITNEXUS_WORKER_POOL_SIZE=0`, and
`skipWorkers` now hard-error (no silent degrade — #1741); the small-repo
threshold no longer selects an in-process path; pool creation stays lazy /
cache-miss-gated so warm all-hit runs never spawn workers.

Worker-path parity fixes — removing sequential surfaced two pre-existing gaps
that tiny-fixture tests had masked by running below the worker threshold, both
fixed by carrying per-file metadata as DATA across the worker boundary (never
re-parsing on the main thread, preserving the OOM fix):
  - C++: templateConstraints wired into worker node identity (SFINAE overload
    disambiguation) + ADL / inline-namespace capture side-channel serialized
    onto the ParsedFile.
  - Kotlin: companion-scope side-channel serialized the same way (companion /
    static dispatch).

Validation: tsc + build clean; full suite green (10,190 pass — the only
deterministic failures were the now-fixed C++/Kotlin worker-path gaps; the 2
remaining full-run failures are pre-existing load flakiness, green in
isolation); cpp-pipeline benchmark stays linear on a 1-worker pool.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): wire C static-linkage side-channel + ADL O(1) collect + tri-review cleanups (#1983)

Follow-up to the worker-pool-only refactor, from a tri-review of the parse path.

- C static-linkage side-channel (P1): cProvider had no collect/applyCaptureSideChannel,
  so on the now-sole worker path C `static` file-local marks were lost across the worker
  boundary -> false cross-file CALLS edges + over-broad #include wildcard visibility on
  every C analysis (the Linux kernel is C). Mirror the C++/Kotlin wiring: serialize
  `staticNames` per file onto ParsedFile.captureSideChannel and restore it on the main
  thread (no re-parse). + a worker-path regression test (the existing c-static-isolation
  fixture passed vacuously — its collision resolves via #include before the global
  free-call fallback ever consults static-linkage).

- captureSideChannel `kind` discriminant: add `kind:'cpp'`/`kind:'c'` tags + guards
  (Kotlin already had one) now that C/C++/Kotlin share the single generic field.

- Perf: collectCppAdlSideChannel scanned the whole argInfoBySite/noAdlSites maps per file
  (O(F^2) per sub-batch, ~100M parseSiteKey calls at kernel scale). Add per-filePath
  lockstep indexes -> O(1) collect; serialized snapshot byte-identical.

- Cleanups: inline the one-line processParsingWithWorkers wrapper into processParsing;
  drop the always-empty WorkerExtractedData.calls/assignments/constructorBindings fields;
  remove the voided astCache param from processParsing; refresh stale "sequential
  fallback" JSDoc.

Validation: tsc + build clean; cpp 297/297, c 8/8 (incl. the new worker-path
static-linkage guard), typescript + parsedfile-store green; cpp ADL benchmark stays linear.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): index C/C++ #include resolution in finalize (O(n²)→O(n))

Kernel-scale C/C++ analysis ground in finalizeScopeModel because three
per-#include operations each did a full O(F) scan with no index — the
finalize O(n²) that surfaced once the #1983 parse-phase OOM was fixed:

- expand{C,Cpp}WildcardNames: parsedFiles.find() per wildcard edge → O(R·F)
- resolveImportTarget: new Set(allFilePaths) rebuilt per #include
- resolveCImportTarget: suffix-match scanned all workspace paths

Each is replaced with a WeakMap-per-pass index keyed on the stable
parsedFiles/allFilePaths references that scope-resolution run.ts passes
once per pass:

- Map<ScopeId,ParsedFile> for wildcard expansion (c/static-linkage.ts +
  cpp/file-local-linkage.ts)
- memoized augmented header set (c/scope-resolver.ts + cpp/scope-resolver.ts)
- basename-bucketed suffix index in resolveCImportTarget (c/import-target.ts),
  shared by C and C++ since resolveCppImportTarget delegates to it

Collapses the C/C++ finalize from O(R·F) to O(R+F). Pure-perf, byte-identical
edge output: 962 targeted tests green (490 C + 472 C/C++ scope-resolution);
the basename index preserves the exact endsWith('/'+target) match and the
fewest-path-components-then-lexicographic tie-break.

The kernel's ~25-30k .h headers are classified C++, so both providers must
be fixed. Proven on the Linux kernel: the C finalize completed
(sr-post-finalize lang=c → sr-end lang=c), which the pre-fix run never
reached in 16+ min of grinding.

Build-independent follow-ups (separate from this finalize fix), documented
for later: emitFreeCallFallback same-name buckets (emit phase),
buildGraphNodeLookup + precount global setup, the ParsedFile store-load,
the dart/go/ruby expand-wildcards .find siblings, and the ~26GB
scope-resolution memory floor (full kernel completion needs >~40GB RAM).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(bench): regenerate C scope-capture baseline for the #1983 c-static-linkage-worker fixture

bench/scope-capture/measure.mjs fingerprints emitCScopeCaptures over the
lang-resolution/c-* fixture corpus. The #1983 PR added the
c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c — the
worker-path static-linkage side-channel test) but did not regenerate the C
baseline, so `--check` has been red on this branch (main, lacking the
fixture, still matches 0de009b).

Pure fixture-corpus drift — no c/captures.ts or query change branch-vs-main,
existing fixtures' captures byte-identical (c-captures.test.ts 45/45),
scaling stays linear (~0.97). Regenerated: 0de009b -> 39f3a83. Bench now
PASS (14 languages). Unrelated to the finalize O(n²) fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): lower kernel-scale resident memory floor + setup cost

Reduce the scope-resolution resident-memory floor and setup throughput on
huge repos (Linux kernel), the wall that remains after #1983 (parse OOM) and
the finalize O(n^2) fix (b71c77b8). Five units; all preserve byte-identical
edge output (C fixture 177n/255e + c/cpp/cross-file/php/static-linkage suites
green, 619 tests).

U1 (src/cli/analyze.ts): RAM-aware auto heap-cap. Replace the hardcoded
16384MB cap with computeHeapCapMb = max(16384, floor(0.75*effectiveRAM)),
where effectiveRAM = min(os.totalmem(), process.constrainedMemory()) with the
unconstrained-sentinel guard. Add --max-semi-space-size=128 on the respawn.
A user-supplied NODE_OPTIONS heap still wins (no re-exec). Verified: 23973MB
on a 31964MB box, 16384 floor on small machines, cgroup-aware, sentinel safe.

U2 (src/storage/parsedfile-store.ts, .../pipeline/phase.ts): export forceGc()
and call it at the per-language eviction boundary, so a finished language's
ParsedFiles are reclaimed before the next language's store-load instead of
collected lazily under the next pass's allocation pressure (which at cap>=RAM
degrades into swap-thrash). Measured on a real drivers/net/ethernet run:
C 2113->894MB and C++ 1754->1057MB reclaimed at the boundary (no fragmentation
defeat). Answers the plan's Open Question 1.

U3 (src/storage/parsedfile-store.ts): intern def objects by nodeId in the load
reviver so a SymbolDefinition's three serialized copies (localDefs /
scope.ownedDefs / scope.bindings[].def) collapse to one shared object on load.
Per-shard def pool (a def's copies are shard-local). Measured ~42% off the
def-object retained heap (3->1; 1.8M->600k distinct objects on 600k defs).

U4 (.../passes/free-call-fallback.ts): memoize pickUniqueGlobalCallable's
post-filter candidate list per (name, callerFilePath), only when no per-caller
visibility filter applies (the list is then a pure function of name+file), so
repeated free calls of one name from a file reuse the same-name-bucket scan
instead of re-walking a potentially huge bucket per site. The cached array is
read-only-consumed by the .filter()-based arity/overload narrowers. Exported
pickUniqueGlobalCallable + buildGlobalCallableIndex and added an equivalence
test (memoized == un-memoized reference for every (name, file, arity),
including warm-cache repeats and cross-file file-local exclusion).

U5 (.../pipeline/phase.ts): replace the O(L*F) per-language precount + repeated
scannedFiles.filter() with a single O(F) partition-by-language pass; bracket
buildGraphNodeLookup with scope-setup-nodeLookup heap probes so the long setup
is no longer silent.

Plan: docs/plans/2026-06-06-001-perf-kernel-scope-resolution-memory-plan.md
(U6 out-of-core global index deferred). Note: the kernel's full C++ pass floor
(~20k headers + the 8.8GB graph) likely still exceeds 24GB by itself, which is
why U6 remains the only unit that clears the wall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(test): match OOM-guidance e2e assertions to the U1 reworded hint

The analyze-heap-oom-e2e real-child-OOM test still asserted the pre-U1
wording ('...out of memory.' + a hardcoded 24576 cap). U1 reworded the hint
to mention the auto heap-cap and use a <MB> placeholder, so the three
toContain substrings no longer matched (the assertion at line 62 failed on
all platforms). Update them to the current message. The unit twin
(analyze-heap-respawn) was already updated in 85bfc216; this integration
test was missed by the targeted local run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(lbug): U6a — deterministic id-sorted graph output behind GITNEXUS_SORT_GRAPH_OUTPUT

First increment of U6 (out-of-core scope-resolution). Adds an optional
deterministic ordering of node + relationship CSV rows by their unique graph
id, behind GITNEXUS_SORT_GRAPH_OUTPUT (default OFF = today's graph-insertion
order, byte-identical — the iterator is returned untouched). With the flag ON
the CSV becomes a pure function of the node/edge SET rather than of emit order.

This is the structural enabler for the windowed/out-of-core resolve (U6b-U6d):
csv-generator.ts:518 currently iterates graph.iterRelationships() in insertion
order with NO terminal sort, so any deviation from parsedFiles-order emit would
change bytes. With U6a on, a windowed emit need only reproduce the same edge
SET, not the global insertion order — removing the single largest byte-identical
hazard from every later windowing step.

Verified: default off keeps the existing csv-pipeline suite byte-identical; on,
node rows are id-sorted and output is independent of graph insertion order
(set-build) with the same node/edge set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(storage): U6d foundation — disk-backed scope store + lazy ScopeTree

Adds scope-index-store.ts: persistScopeShards (per-file scope shards via the
proven mapReplacer + def-interning reviver) + DiskBackedScopeTree, a lazy
ScopeTree that serves getScope from a bounded LRU of decoded shards plus a small
resident skeleton (scopeId -> {shard, childIds, parent}). Exports
makeInterningReviver from parsedfile-store for reuse.

This is the contained, highest-risk mechanism of U6d (out-of-core scope
resolution): the emit passes reach the heavy per-Scope binding payload
(~17-20GB on the kernel) ONLY through scopeTree.getScope (a point lookup) and
getChildren — they never read parsed.scopes directly — so moving that payload to
disk behind getScope is transparent. Every consumer reads a Scope BY VALUE, so a
value-faithful disk round-trip is byte-identical to resolution.

Proven in isolation: DiskBackedScopeTree is value-identical to buildScopeTree
for getScope/getChildren/getParent/getAncestors/has/size across multiple files
and after LRU eviction, and preserves the def-identity collapse (ownedDefs[i]
=== binding.def). Nothing wires it yet (the resolution-pipeline integration is
the next increment) — zero production impact; default off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): U6d integration — seal scopeTree to disk before emit (GITNEXUS_DISK_SCOPE_INDEX)

Wires the U6d out-of-core scope index into the live pipeline behind
GITNEXUS_DISK_SCOPE_INDEX (default OFF = byte-identical). When on:

- finalize-orchestrator builds a TransitionalScopeTree (validated, fully
  resident) instead of buildScopeTree, so finalize/propagate/resolve are
  unchanged.
- After resolve, before emit, run.ts seals it: persists the scopes to a
  file-sharded scope-index-store, swaps the model's scopeTree to disk-backed
  serving from the inside (the frozen bundle can't be reassigned, but the
  wrapper nulls its own resident backing), and drops the heavy Scope.bindings
  payload from all THREE holders — the model's tree (seal), the caller's
  preExtractedParsedFiles, and run.ts's own parsedFiles (scope-stripped copies
  for emit). Emit reads scopes only via scopeTree.getScope (a point lookup,
  now disk-backed + LRU) — verified it never reads parsed.scopes.

Purpose: lower the per-language resident PEAK (kernel C pass ~20→~12 GB by
moving the ~8-9 GB scope payload to disk) so the analysis fits on smaller-RAM
machines. At >=24 GB the full kernel already fits with U1-U5 (U2's 8.7 GB
inter-language forceGc reclaim keeps each pass under cap) — empirically
confirmed — so this is the sub-24 GB lever, not needed at 24 GB.

Byte-identical evidence: DiskBackedScopeTree/TransitionalScopeTree return
value-identical scopes vs buildScopeTree (getScope/getChildren/getParent/
getAncestors, across files + after LRU eviction + post-seal); emit reads only
getScope + referenceSites; flag-off (394 tests) and flag-on-resident (91 tests)
resolver suites stay green; an end-to-end A/B on a 212-file C+cpp+rust subset
produced identical 17,444 nodes / 31,343 edges with the seal firing per language
(c: 410→141 MB reclaimed). Kernel-scale peak-drop measurement pending the
in-flight verdict run freeing memory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): U6d — id-back workspaceIndex so the disk seal can reclaim scopes

The kernel run revealed the contained scopeTree seal didn't lower the heap:
WorkspaceResolutionIndex held Scope OBJECTS (classScopeByDefId / moduleScopeByFile),
built from every ParsedFile and live through emit, so the ~28k module + class
scopes stayed pinned past the seal (sr-seal-pre 17,583 -> sr-seal-post 17,771 MB,
no drop). It was the sole residual Scope-object holder (SemanticModel holds none).

Fix: classScopeByDefId / moduleScopeByFile become id-backed ScopeByKeyView
instances — a ReadonlyMap<K, Scope> facade over a K->ScopeId map + the scopeTree,
whose .get fetches via scopeTree.getScope(id). The index now pins only ids, so
once the tree seals to disk the scopes become collectible. Byte-identical: the
view returns the same Scope the resident tree holds (or a value-identical revived
one in disk mode), and iteration keeps the old insertion order. buildWorkspace
ResolutionIndex takes an optional scopeTree (live pipeline passes it); without it
(unit tests) the legacy direct Scope-object maps are returned unchanged.

Verified byte-identical: 733 tests across workspace-index / imported-return-types
/ c / cpp / cross-file / go / java. Kernel peak-drop re-measurement to follow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): U6d — precompute exportedCallableByName (fix disk-getScope thrash)

The workspaceIndex id-backing freed the kernel scopes but exposed a throughput
collapse: findExportedDefByName's workspace fallback (walkers.ts:1019) scanned
EVERY module scope's bindings per unresolved free call, and under the U6d
disk-backed scopeTree each module-scope access faulted a shard in from disk —
lib ON went ~1min -> ~7.5min.

Fix: precompute the fallback result once into
WorkspaceResolutionIndex.exportedCallableByName (simpleName -> first module-local
callable def, first-file-wins — the exact semantics the scan returned), built
from the resident module-scope bindings at index-build time. findExportedDefByName
now does an O(1) lookup with zero disk reads.

Result: lib ON ~7.5min -> 21s (cache-warm), byte-identical 17,444/31,343; 758
tests green across workspace-index + c/cpp/cross-file/go/python.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: rename cryptic U-unit codes to descriptive names in comments

The plan-unit shorthand (U3/U4/U6a/U6d/...) was meaningless in the code.
Renamed in comments + test descriptions (no behavior change, byte-identical):
  out-of-core scope index   (was U6)
  deterministic output      (was U6a)
  disk-backed scope seal    (was U6d)
  def-object interning      (was U3)
  free-call candidate cache (was U4)
Also renamed throughout the PR title/summary. Pushed commit messages keep
their original U-codes as historical record.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): durable ParsedFile shards for warm-cache coverage (#2038)

On a warm re-analyze where every chunk is a parse-cache HIT, no parse worker
runs, the run-scoped ParsedFile store is cleared at parse start, and the cached
ParseWorkerResult carries no ParsedFiles (the worker writes them to the store
and empties them from the message). Scope-resolution then found an empty store
and fell back to main-thread extractParsedFile — re-opening the #1983
tree-sitter native-leak OOM the disk store closes (abhigyanpatwari review on
parse-cache.ts).

Fix: workers ALSO write their ParsedFiles to a durable, content-addressed store
(parsedfile-cache/) keyed by chunk hash, mirroring the parse cache's lifecycle
(version-gated by PARSE_CACHE_VERSION, pruned in lockstep to the surviving
keys). On a warm hit the chunk's durable shards are byte-COPIED into the
run-scoped store (no re-parse, no re-serialize -> byte-identical), so
scope-resolution streams them exactly as on a cold run. A coherence gate
re-dispatches the worker whenever a cached chunk's durable shards are missing
(migration / pruned / version-stale) -- never the main-thread extract.

- worker-pool/parse-worker: thread chunkHash through dispatch->job->flush
  (incl. split/requeue) so the worker tags its durable shard by content
- parsedfile-store: durable persist / restore / index / prune API (sibling
  dir, never cleared per run); content-addressing makes stale reuse impossible
- parse-impl: load durable index, gate the cache hit on durable coverage,
  restore on hit, dispatch chunkHash on miss
- run-analyze: prune+save the durable store to the parse cache's surviving keys
- saveParseCache returns its written keys (the durable keepKeys)

Verified on linux/lib: warm preExtractedHits = full coverage (520/207/1, zero
main-thread re-parse), byte-identical cold==warm (17,456n/31,353e), warm 8.5x
faster. New two-run + mixed-mode + coherence-gate regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): clear stale scope-index-store shards on each seal (#2038)

The disk-backed scope index writes sequential s<n>.json shards into a shared
<storagePath>/scope-index-store/ dir, with the index resetting per
persistScopeShards call. A seal that writes fewer shards than a previous one
(a later language with fewer files, or a re-run of a shrunken repo) left stale
tail shards on disk indefinitely -- never read by the disk-backed tree, but
multi-GB on kernel-scale repos.

Add clearScopeIndexStore() and clear at the start of persistScopeShards: the
previously sealed language has finished emit and been released before the next
seal runs, so its DiskBackedScopeTree never reads those shards again. Unit
tests: a stale prior-run shard is removed, a fewer-files re-seal leaves no tail
shards, and the helper is idempotent.

Addresses abhigyanpatwari review on run.ts (disk hygiene for the
GITNEXUS_DISK_SCOPE_INDEX path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 22:46:34 +01:00

173 lines
7 KiB
TypeScript

/**
* U6 (B3 from PR #1693 review) — Multi-chunk pipeline integration with a
* wall-clock budget.
*
* The PR's headline claim is "analyze no longer hangs on TS-root-shaped
* loads". The unit suite already pins each resilience layer individually
* (worker-pool-resilience.test.ts), the deferred-extraction equivalence
* (parse-impl-deferred-extraction.test.ts — U7), and the chunk
* concurrency (parse-impl-chunk-concurrency.test.ts — U1). What was
* missing: a single end-to-end run that exercises the full chunked
* parse-and-resolve path on a multi-chunk fixture, BOUNDED by a
* wall-clock so a regression that re-introduces the hang fails this
* test loudly instead of slipping past via inequality assertions.
*
* This runs the real worker pool (the sole parse path since the sequential
* parser was removed) on a multi-chunk fixture, BOUNDED by a wall-clock so a
* regression that re-introduces the hang fails loudly instead of slipping past
* via inequality assertions. The load-bearing invariants — multi-chunk parsing
* completes within a bounded budget and produces all expected symbols — catch
* the bulk of the regressions B3 was concerned about. Needs the dist worker
* (integration tier).
*/
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
const ORIGINAL_BUDGET = process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
const WALL_CLOCK_BUDGET_MS = 30_000;
function buildFixture(): Record<string, string> {
const fixture: Record<string, string> = {};
for (let i = 0; i < 15; i++) {
fixture[`mod${i}.ts`] = `export function fn${i}(): number {\n return ${i};\n}\n`;
}
// A "realistically dense" file — 30 functions + a class + an interface +
// a re-export. Stands in for the kind of file that previously stalled the
// serial-extraction chunk loop. Pure-TS so the parser doesn't choke.
const complexFnLines = Array.from(
{ length: 30 },
(_, i) => `export function complex${i}(c: Config): string { return c.name + String(${i}); }`,
).join('\n');
fixture['complex.ts'] = `
export interface Config { name: string; }
export class Service {
configure(c: Config): void { this.name = c.name; }
private name: string = '';
describe(): string { return this.name; }
}
${complexFnLines}
`;
fixture['index.ts'] =
'export { Service, type Config } from "./complex";\n' +
Array.from({ length: 15 }, (_, i) => `export { fn${i} } from "./mod${i}";`).join('\n') +
'\n';
return fixture;
}
async function runFixture(): Promise<{
nodeCount: number;
relationshipCount: number;
symbolNames: Set<string>;
elapsedMs: number;
}> {
const fixture = buildFixture();
// Force multi-chunk parsing on the small fixture by lowering the byte
// budget below each file's size. parse-impl reads the budget at module
// load — vi.resetModules() forces a fresh module so the env takes effect.
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = '64';
vi.resetModules();
const { runChunkedParseAndResolve } =
await import('../../src/core/ingestion/pipeline-phases/parse-impl.js');
const { createKnowledgeGraph } = await import('../../src/core/graph/graph.js');
const repoPath = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-large-fixture-'));
try {
for (const [name, content] of Object.entries(fixture)) {
fs.writeFileSync(path.join(repoPath, name), content);
}
const files = Object.keys(fixture);
const scanned = files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repoPath, rel)).size,
}));
const graph = createKnowledgeGraph();
// Wrap the run in Promise.race so the wall-clock budget is enforced
// as an exception, not a >=/<= inequality assertion (DoD §2.7 — a
// bounds-only count is a regression-mask; a hard timeout is a
// hang-detector). If the run exceeds the budget, the rejection
// fails the test with a specific timeout error so the diagnostic
// surfaces the actual regression class.
const start = Date.now();
await Promise.race([
runChunkedParseAndResolve(graph, scanned, files, files.length, repoPath, start, () => {}, {}),
new Promise<never>((_, reject) =>
setTimeout(
() =>
reject(
new Error(
`runChunkedParseAndResolve exceeded ${WALL_CLOCK_BUDGET_MS}ms wall-clock budget — likely the hang B3 was meant to prevent`,
),
),
WALL_CLOCK_BUDGET_MS,
),
),
]);
const elapsedMs = Date.now() - start;
const symbolNames = new Set<string>();
for (const node of graph.nodes.values()) {
const name = (node.properties as { name?: string } | undefined)?.name;
if (typeof name === 'string') symbolNames.add(name);
}
return {
nodeCount: graph.nodeCount,
relationshipCount: graph.relationshipCount,
symbolNames,
elapsedMs,
};
} finally {
if (fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
}
}
describe('parse-impl wall-clock integration on multi-chunk fixture (U6 / B3)', () => {
beforeEach(() => {
vi.resetModules();
});
afterEach(() => {
if (ORIGINAL_BUDGET === undefined) {
delete process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
} else {
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = ORIGINAL_BUDGET;
}
});
it('completes a 17-file multi-chunk fixture within wall-clock and indexes every named symbol', async () => {
const result = await runFixture();
// Reaching this assertion means the Promise.race did NOT time out —
// the run completed in under WALL_CLOCK_BUDGET_MS. That alone is the
// primary B3 invariant: "does not hang on a multi-chunk workload";
// the race rejection already enforces it. The previous
// `Math.min(elapsedMs, BUDGET)` form here resolved to
// `expect(x).toBe(x)` — tautological and catching nothing. The
// load-bearing wall-clock check lives in the Promise.race above;
// the per-symbol assertions below catch silent mid-chunk crashes.
expect(typeof result.elapsedMs).toBe('number');
// All 15 plain function declarations must show up in the graph.
for (let i = 0; i < 15; i++) {
expect(result.symbolNames.has(`fn${i}`)).toBe(true);
}
// The complex.ts surfaces: Service class, Config interface, configure
// method, and a representative sampling of the complex0..complex29
// functions. Pin specific names (not just a count) so chunk-boundary
// truncation surfaces as a concrete missing-symbol failure with a
// specific diagnostic.
expect(result.symbolNames.has('Service')).toBe(true);
expect(result.symbolNames.has('Config')).toBe(true);
expect(result.symbolNames.has('configure')).toBe(true);
expect(result.symbolNames.has('describe')).toBe(true);
expect(result.symbolNames.has('complex0')).toBe(true);
expect(result.symbolNames.has('complex15')).toBe(true);
expect(result.symbolNames.has('complex29')).toBe(true);
});
});