scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-10-10 03:27:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

perf(parse-impl): free deferred-extraction arrays after consumption (U15 lightweight M1)

Browse source 
PR #1693 review M1 noted that the deferred-extraction accumulator
arrays (`deferredWorkerImports`, `deferredWorkerCalls`,
`deferredWorkerHeritage`, `deferredConstructorBindings`,
`deferredAssignments`) were retained until function return, making
peak accumulator memory O(repo) instead of O(in-flight stage).

This commit implements the LIGHTWEIGHT version: free each array
immediately after its last consumer drains/reads it, dropping peak
accumulator memory progressively through the deferred-extraction
stages. The structural per-chunk streaming variant (the original
U15 framing) is deliberately deferred — the doc-review's adversarial
reviewer (A4) flagged it as defending unmeasured memory pressure,
and the simpler array-clearing captures the bulk of the benefit
without committing to a scheduling-strategy decision (microtask vs
parallel extractor task vs worker-side) that profile data should
inform.

Clears added:

  1. After `processImportsFromExtracted` (the sole consumer of
     `deferredWorkerImports`): clear the imports array before
     the heavier heritage/calls stages run.
  2. After `buildHeritageMap` (the LAST consumer of the raw
     `deferredWorkerHeritage` records — processCallsFromExtracted
     reads from the derived `fullWorkerHeritageMap` instead):
     clear the heritage array before the call-resolution stage.
  3. After `processAssignmentsFromExtracted` (the joint last
     consumer with processCallsFromExtracted for the calls/
     bindings/assignments triple): clear all three before
     downstream graph-build / scope-resolution uses its own
     working memory.

Arrays returned in the function result object (allFetchCalls,
allExtractedRoutes, allDecoratorRoutes, allToolDefs, allORMQueries,
allParsedFiles) intentionally stay live — downstream consumers
need them.

Graph-output equivalence is preserved (U7 multi-chunk equivalence
test passes — the clears happen AFTER each array's last consumer
has copied data into the graph or derived structures).
This commit is contained in:
Gergo Magyar 2026-05-20 09:55:12 +01:00
parent 2badd78e80
commit 47060c2408
1 changed files with 30 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

30
gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts
View file

@ -629,6 +629,15 @@ export async function runChunkedParseAndResolve(
repoPath,
importCtx,
);
// U15 (lightweight M1): processImportsFromExtracted is the sole
// consumer of `deferredWorkerImports`. Free the array now so the
// GC can reclaim the per-file ExtractedImport records before the
// heavier downstream stages run (heritage, routes, calls). Peak
// accumulator memory drops from O(repo) to O(repo - imports) for
// the remainder of the deferred phase. The future per-chunk
// streaming upgrade can rewrite this with the same correctness
// contract once profile data shows it's warranted.
deferredWorkerImports.length = 0;
}
if (anyChunkNeedsWildcardSynth) {
synthesizeWildcardImportBindings(graph, ctx);
@ -694,6 +703,13 @@ export async function runChunkedParseAndResolve(
deferredWorkerHeritage.length > 0
? buildHeritageMap(deferredWorkerHeritage, ctx, getHeritageStrategyForLanguage)
: undefined;
// U15 (lightweight M1): buildHeritageMap is the LAST consumer of the
// raw `deferredWorkerHeritage` records — processCallsFromExtracted
// below reads from the derived `fullWorkerHeritageMap` instead. Free
// the raw heritage array now so the GC can reclaim it before the
// (potentially long) call-resolution stage. processHeritageFromExtracted
// earlier was a read-only consumer (pushed to graph, didn't drain).
deferredWorkerHeritage.length = 0;
if (deferredWorkerCalls.length > 0) {
await processCallsFromExtracted(
@ -732,6 +748,20 @@ export async function runChunkedParseAndResolve(
bindingAccumulator,
);
}
// U15 (lightweight M1): all three arrays have had their last consumer
// by the time we reach this point — processCallsFromExtracted drained
// `deferredWorkerCalls` and read `deferredConstructorBindings`;
// processAssignmentsFromExtracted drained `deferredAssignments` and
// also read `deferredConstructorBindings`. Free them now so the
// function-scope references die before downstream graph-build /
// scope-resolution starts using its own working memory. Note: arrays
// returned in the function result object (allFetchCalls,
// allExtractedRoutes, allDecoratorRoutes, allToolDefs, allORMQueries,
// allParsedFiles) intentionally stay live — downstream consumers
// need them.
deferredWorkerCalls.length = 0;
deferredConstructorBindings.length = 0;
deferredAssignments.length = 0;
} finally {
await workerPool?.terminate();
}