From 47060c2408a676456fd118ec63a6cc3f66e0a9a7 Mon Sep 17 00:00:00 2001
From: Gergo Magyar <gergomagyar@icloud.com>
Date: Wed, 20 May 2026 09:55:12 +0100
Subject: [PATCH] perf(parse-impl): free deferred-extraction arrays after
 consumption (U15 lightweight M1)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

PR #1693 review M1 noted that the deferred-extraction accumulator
arrays (`deferredWorkerImports`, `deferredWorkerCalls`,
`deferredWorkerHeritage`, `deferredConstructorBindings`,
`deferredAssignments`) were retained until function return, making
peak accumulator memory O(repo) instead of O(in-flight stage).

This commit implements the LIGHTWEIGHT version: free each array
immediately after its last consumer drains/reads it, dropping peak
accumulator memory progressively through the deferred-extraction
stages. The structural per-chunk streaming variant (the original
U15 framing) is deliberately deferred — the doc-review's adversarial
reviewer (A4) flagged it as defending unmeasured memory pressure,
and the simpler array-clearing captures the bulk of the benefit
without committing to a scheduling-strategy decision (microtask vs
parallel extractor task vs worker-side) that profile data should
inform.

Clears added:

  1. After `processImportsFromExtracted` (the sole consumer of
     `deferredWorkerImports`): clear the imports array before
     the heavier heritage/calls stages run.
  2. After `buildHeritageMap` (the LAST consumer of the raw
     `deferredWorkerHeritage` records — processCallsFromExtracted
     reads from the derived `fullWorkerHeritageMap` instead):
     clear the heritage array before the call-resolution stage.
  3. After `processAssignmentsFromExtracted` (the joint last
     consumer with processCallsFromExtracted for the calls/
     bindings/assignments triple): clear all three before
     downstream graph-build / scope-resolution uses its own
     working memory.

Arrays returned in the function result object (allFetchCalls,
allExtractedRoutes, allDecoratorRoutes, allToolDefs, allORMQueries,
allParsedFiles) intentionally stay live — downstream consumers
need them.

Graph-output equivalence is preserved (U7 multi-chunk equivalence
test passes — the clears happen AFTER each array's last consumer
has copied data into the graph or derived structures).
---
 .../ingestion/pipeline-phases/parse-impl.ts   | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts b/gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts
index a1ada9975..3678539e4 100644
--- a/gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts
+++ b/gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts
@@ -629,6 +629,15 @@ export async function runChunkedParseAndResolve(
         repoPath,
         importCtx,
       );
+      // U15 (lightweight M1): processImportsFromExtracted is the sole
+      // consumer of `deferredWorkerImports`. Free the array now so the
+      // GC can reclaim the per-file ExtractedImport records before the
+      // heavier downstream stages run (heritage, routes, calls). Peak
+      // accumulator memory drops from O(repo) to O(repo - imports) for
+      // the remainder of the deferred phase. The future per-chunk
+      // streaming upgrade can rewrite this with the same correctness
+      // contract once profile data shows it's warranted.
+      deferredWorkerImports.length = 0;
     }
     if (anyChunkNeedsWildcardSynth) {
       synthesizeWildcardImportBindings(graph, ctx);
@@ -694,6 +703,13 @@ export async function runChunkedParseAndResolve(
       deferredWorkerHeritage.length > 0
         ? buildHeritageMap(deferredWorkerHeritage, ctx, getHeritageStrategyForLanguage)
         : undefined;
+    // U15 (lightweight M1): buildHeritageMap is the LAST consumer of the
+    // raw `deferredWorkerHeritage` records — processCallsFromExtracted
+    // below reads from the derived `fullWorkerHeritageMap` instead. Free
+    // the raw heritage array now so the GC can reclaim it before the
+    // (potentially long) call-resolution stage. processHeritageFromExtracted
+    // earlier was a read-only consumer (pushed to graph, didn't drain).
+    deferredWorkerHeritage.length = 0;
 
     if (deferredWorkerCalls.length > 0) {
       await processCallsFromExtracted(
@@ -732,6 +748,20 @@ export async function runChunkedParseAndResolve(
         bindingAccumulator,
       );
     }
+    // U15 (lightweight M1): all three arrays have had their last consumer
+    // by the time we reach this point — processCallsFromExtracted drained
+    // `deferredWorkerCalls` and read `deferredConstructorBindings`;
+    // processAssignmentsFromExtracted drained `deferredAssignments` and
+    // also read `deferredConstructorBindings`. Free them now so the
+    // function-scope references die before downstream graph-build /
+    // scope-resolution starts using its own working memory. Note: arrays
+    // returned in the function result object (allFetchCalls,
+    // allExtractedRoutes, allDecoratorRoutes, allToolDefs, allORMQueries,
+    // allParsedFiles) intentionally stay live — downstream consumers
+    // need them.
+    deferredWorkerCalls.length = 0;
+    deferredConstructorBindings.length = 0;
+    deferredAssignments.length = 0;
   } finally {
     await workerPool?.terminate();
   }