From 98ee665889919b77db0b424fe680c5342096cf80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Sun, 26 Apr 2026 12:16:09 +0100 Subject: [PATCH] fix(ingestion): two-channel binding lifecycle (closes #1066) + scope-resolution I8 hardening (#1082) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(csharp): adaptive tree-sitter buffer + frozen-bucket clone for cross-namespace siblings (#1066) Two coupled regressions surfaced when analyzing real-world C# repos with large source files (issue #1066): 1. Tree-sitter `parser.parse()` is hard-coded to a 32 KB buffer by default. Any file exceeding that threshold throws `Invalid argument` on the worker re-parse path of `populateCsharpNamespaceSiblings` (and the analogous Python / TypeScript captures fallbacks). 2. After the buffer fix unblocks the AST walk, the hook tries to `push()` onto the inner `BindingRef[]` array fetched from `indexes.bindings` — but `materializeBindings` froze that array via `Object.freeze(refs.slice())`. Result: `Cannot add property N, object is not extensible`. Fixes: - `csharp/captures.ts`, `python/captures.ts`, `typescript/captures.ts`: pass `bufferSize: getTreeSitterBufferSize(sourceText.length)` to `parser.parse()` on the cache-miss path so multi-MB files parse. - `csharp/namespace-siblings.ts`: introduce `cloneBindingBucket` to copy the frozen array before mutating, then `set()` the new array back. This is a working but architecturally compromised workaround (#1050 follow-up will replace it with an explicit augmentation channel — see docs/plans/2026-04-26-001 plan). Tests: - New `csharp-large-cache-miss-resolution` fixture (Models/Services/ Other layout, ~77 KB padded UserService.cs) drives the buffer-size failure end-to-end through worker mode. - `csharp.test.ts`: 4 new regression assertions covering both the parse-time buffer-size failure and the freeze workaround. - Per-language captures unit tests gain "large cache-miss file uses adaptive buffer" coverage (TS, Python, C#). - `csharp-hooks.test.ts`: in-memory freeze regression test that reproduces the `Cannot add property` crash without invoking the C# parser at all. Made-with: Cursor * refactor(scope-resolution): add bindingAugmentations channel to indexes Step 1 of the binding-augmentation-channel refactor (issue #1066 follow-up). Pure shape change — no consumers yet. Adds a new `readonly bindingAugmentations` field to `ScopeResolutionIndexes` initialized as an empty `Map` by `finalizeScopeModel`. The new channel is the dedicated post-finalize write target for hooks like `populateCsharpNamespaceSiblings`, so `indexes.bindings` can stay frozen and finalize-owned. Behavior unchanged: nothing reads or writes the new field yet. tsc and the full unit suite remain green. Plan: docs/plans/2026-04-26-001-binding-augmentation-channel.md (local only — `docs/plans/` is gitignored). Made-with: Cursor * feat(scope-resolution): add lookupBindingsAt dual-source helper Step 2 of the binding-augmentation-channel refactor. Introduces a single primitive every walker uses to read both the finalize-owned `indexes.bindings` channel and the post-finalize `indexes.bindingAugmentations` channel. Contract: - Finalized refs come first (preserves existing precedence). - Augmented refs append, deduped by `def.nodeId`. - Empty input on both channels returns a shared frozen empty array. - Single-channel hits return the bucket by reference (no allocation). No consumers are wired yet — Step 3 routes the existing walker primitives through this helper. Augmentations remain empty for every language; behavior of the full suite is unchanged. 8 unit tests pin precedence, dedup, identity for single-channel hits, and the shared-empty-frozen-array sentinel. Made-with: Cursor * refactor(scope-resolution): route binding lookups through lookupBindingsAt Step 3 of the binding-augmentation-channel refactor. Every direct `indexes.bindings.get(...)` consumer in the post-finalize phase is now routed through `lookupBindingsAt` (per-name) or `namesAtScope` + `lookupBindingsAt` (bulk iteration). Routed sites: - `findClassBindingInScope` (walkers.ts) — class-receiver lookups. - `findCallableBindingInScope` (walkers.ts) — free-call lookups. - `findExportedDefByName` (walkers.ts) — module-scope-fallback callable lookups. - `propagateImportedReturnTypes` (passes/imported-return-types.ts) — bulk iteration over an importer's binding entries; switched to `namesAtScope` + per-name `lookupBindingsAt` so post-finalize augmentations are visible to import-derived typeBinding mirrors. Behavior unchanged: augmentations are empty across the suite (Step 4 populates them for C# `populateNamespaceSiblings`). 587 scope-resolution unit tests + 50 integration resolver suites green (4 pre-existing Swift method-implements failures unrelated to this work). Adds `namesAtScope` companion helper for the bulk-iteration callers. Made-with: Cursor * refactor(csharp): write namespace siblings to bindingAugmentations channel Step 4 of the binding-augmentation-channel refactor. The C# `populateNamespaceSiblings` hook is the only consumer that needed to inject cross-file bindings post-finalize, and prior to this change it cloned the (frozen) finalized `BindingRef[]` arrays through a `cloneBindingBucket` helper, then `set()`-back the new array — a workaround for the `Object.freeze` applied by `finalize-algorithm.ts` (issue #1066 root cause). Architecturally that violated `ScopeResolver` Invariant I8 (which permits post-finalize modifications but not in-place mutation of finalized buckets). It also forced read-side consumers to be aware of the workaround. This change: * Switches the three C# write sites to append into `indexes.bindingAugmentations` via `getAugmentationBucket`. The augmentation channel was added in Step 1 and is mutable by contract: inner `BindingRef[]` arrays here are NEVER frozen. * Deletes `cloneBindingBucket` and `getMutableScopeBindings` (workaround helpers no longer needed). * `lookupBindingsAt` (Step 2) merges the two channels transparently for every walker (Step 3), so behavior is unchanged for callers. * Updates the unit test to assert against both channels: finalized bucket stays frozen and untouched, cross-file siblings show up in augmentations only. Renamed the test accordingly. Validation: * `npx tsc --noEmit` clean. * csharp hooks unit + walkers-augmentations unit + csharp integration resolver suite all green (236/236). * Wider `test/unit/scope-resolution test/integration/resolvers` suite: 2507 pass, only 4 pre-existing Swift METHOD_IMPLEMENTS failures remain (unrelated to this work, present on baseline). Refs: issue #1066, ADR-pending binding-augmentation-channel. Made-with: Cursor * feat(scope-resolution): tighten I8 + add validateBindingsImmutability dev guard Step 5 of the binding-augmentation-channel refactor. Captures the new two-channel binding lifecycle in the contract docs and adds a dev-mode runtime validator so a future hook cannot silently drift back into mutating `indexes.bindings`. Contract changes: * `contract/scope-resolver.ts` — rewrote Invariant I8 to describe the two channels (`indexes.bindings` is finalize-output and immutable post-finalize; `indexes.bindingAugmentations` is the append-only post-finalize channel populated by hooks like `populateNamespaceSiblings`). Documented `lookupBindingsAt` as the read-side merger and pointed at the new validator as the enforcement mechanism. * `gitnexus-shared/src/scope-resolution/types.ts` — extended the module-header lifecycle contract to call out `bindingAugmentations` alongside `ReferenceIndex` as the two structures populated after the freeze. Validator: * New `pipeline/validate-bindings-immutability.ts` mirrors the shape of `validateOwnershipParity` (#909): runs only when `NODE_ENV !== 'production' && VALIDATE_SEMANTIC_MODEL !== '0'`, emits via `onWarn`, never throws. Asserts (a) every inner `BindingRef[]` in `indexes.bindings` is `Object.isFrozen`, and (b) every inner array in `indexes.bindingAugmentations` is NOT frozen. * Wired into `pipeline/run.ts` after both `populateNamespaceSiblings` and `propagateImportedReturnTypes`, before `resolveReferenceSites`. One sweep covers the full post-finalize surface. Tests: * `validate-bindings-immutability.test.ts` — 6 cases pinning happy path, both drift directions, multi-violation accumulation, and both production no-op gates. All scope-resolution + csharp resolver tests green (242/242 in the focused run; matches the wider Step 4 baseline). Made-with: Cursor * fix(ingestion): size tree-sitter buffers from UTF-8 bytes Tree-sitter buffer sizing is byte-based, so computing adaptive buffers from JavaScript string length under-sized UTF-8-heavy files. Make getTreeSitterBufferSize accept source text directly and compute Buffer.byteLength internally, then update all parse call sites and max-buffer skip checks to use byte length. Add multibyte cache-miss and cap regressions for C#, Python, TypeScript, and the C# namespace-sibling fallback parse path. Made-with: Cursor * test(scope-resolution): pin augmentation read paths Add focused unit coverage for augmented-only binding reads across the routed walker helpers and imported-return-type propagation path. Clarify I8 wording around lexical Scope.bindings versus post-finalize index channels, and document the intentional local-only behavior of findExportedDef. Also switch the immutability validator tests to Vitest env stubs, document one intentional validator blind spot, and split C# namespace-sibling tests so UTF-8 parsing and augmentation-channel behavior are asserted independently. Made-with: Cursor * test(scope-resolution): avoid slow parser stress fixtures Replace high-cardinality large-file capture fixtures with large padding plus a trailing declaration. This still proves adaptive tree-sitter buffers parse beyond large ASCII and UTF-8-heavy input, without making query matching process thousands of declarations and risking timeouts. Made-with: Cursor * test(scope-resolution): add python and typescript cache-miss resolver regressions Add worker-mode resolver integration coverage mirroring the C# #1066 scenario for Python and TypeScript. Each test builds a temp fixture with large ASCII and UTF-8-heavy source padding, then asserts trailing declarations and call edges still resolve after scope-resolution cache-miss reparsing. Made-with: Cursor * refactor(scope-resolution): gate I8 validator and fast-path namesAtScope Addresses SPARC reviewer feedback on the binding-augmentation channel: - Validator gate is now opt-in outside development. Extract isSemanticModelValidatorEnabled() in utils/env.ts as the single predicate; both validateBindingsImmutability and phase.ts's warn handler share it. Default CLI runs no longer pay the O(binding-buckets) scan, and explicit VALIDATE_SEMANTIC_MODEL=1 now emits warnings even when NODE_ENV is unset. - namesAtScope returns Iterable and zero-allocates when at most one channel is populated (returns Map.keys() directly), only materializing a Set when both channels carry names. The caller-side branching and EMPTY_NAMES escape hatch in propagateImportedReturnTypes are gone -- both helpers handle the empty-augmentation case internally. - C# namespace-siblings header/JSDoc, model JSDoc, I8 contract prose, and the #1066 integration-test header rewritten to say post-finalize fanout appends only to bindingAugmentations; finalized refs come first and win duplicate def.nodeId metadata; local lexical Scope.bindings remains the first-tier shadowing channel. Validator unit-test setup deduplicated via beforeEach and extended with default-CLI no-op + explicit-opt-in cases. Made-with: Cursor --- gitnexus-shared/src/scope-resolution/types.ts | 12 +- gitnexus/src/core/ingestion/call-processor.ts | 4 +- gitnexus/src/core/ingestion/constants.ts | 15 +- .../core/ingestion/finalize-orchestrator.ts | 5 + .../src/core/ingestion/heritage-processor.ts | 4 +- .../src/core/ingestion/import-processor.ts | 2 +- .../ingestion/languages/csharp/captures.ts | 5 +- .../languages/csharp/namespace-siblings.ts | 116 ++-- .../ingestion/languages/python/captures.ts | 5 +- .../languages/typescript/captures.ts | 5 +- .../model/scope-resolution-indexes.ts | 15 +- .../src/core/ingestion/parsing-processor.ts | 10 +- .../contract/scope-resolver.ts | 47 +- .../passes/imported-return-types.ts | 12 +- .../scope-resolution/pipeline/phase.ts | 6 +- .../scope-resolution/pipeline/run.ts | 11 + .../validate-bindings-immutability.ts | 78 +++ .../scope-resolution/scope/walkers.ts | 133 ++++- gitnexus/src/core/ingestion/utils/env.ts | 14 + .../core/ingestion/workers/parse-worker.ts | 10 +- .../LargeCacheMissResolution.csproj | 5 + .../Models/User.cs | 6 + .../Other/Helper.cs | 3 + .../Services/UserService.cs | 548 ++++++++++++++++++ .../test/integration/resolvers/csharp.test.ts | 65 +++ .../test/integration/resolvers/python.test.ts | 70 ++- .../integration/resolvers/typescript.test.ts | 85 ++- gitnexus/test/unit/ingestion-utils.test.ts | 46 +- .../csharp/csharp-captures.test.ts | 20 + .../csharp/csharp-hooks.test.ts | 135 +++++ .../imported-return-types.test.ts | 66 ++- .../python/python-fixtures.test.ts | 14 + .../typescript/typescript-captures.test.ts | 18 + .../validate-bindings-immutability.test.ts | 177 ++++++ .../walkers-augmentations.test.ts | 195 +++++++ 35 files changed, 1831 insertions(+), 131 deletions(-) create mode 100644 gitnexus/src/core/ingestion/scope-resolution/pipeline/validate-bindings-immutability.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/csharp-large-cache-miss-resolution/LargeCacheMissResolution.csproj create mode 100644 gitnexus/test/fixtures/lang-resolution/csharp-large-cache-miss-resolution/Models/User.cs create mode 100644 gitnexus/test/fixtures/lang-resolution/csharp-large-cache-miss-resolution/Other/Helper.cs create mode 100644 gitnexus/test/fixtures/lang-resolution/csharp-large-cache-miss-resolution/Services/UserService.cs create mode 100644 gitnexus/test/unit/scope-resolution/validate-bindings-immutability.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/walkers-augmentations.test.ts diff --git a/gitnexus-shared/src/scope-resolution/types.ts b/gitnexus-shared/src/scope-resolution/types.ts index 1cff70115..c3d8b80fb 100644 --- a/gitnexus-shared/src/scope-resolution/types.ts +++ b/gitnexus-shared/src/scope-resolution/types.ts @@ -10,8 +10,16 @@ * Lifecycle contract (RFC §2.8): scopes are **constructed during extraction, * linked during finalize, immutable after finalize**. All fields are * `readonly` at the type level; `Object.freeze` is applied at runtime in dev - * builds. `ReferenceIndex` is the sole structure populated after freeze — by - * resolution, before emission. + * builds. + * + * Two structures are populated after freeze: + * 1. `ReferenceIndex` — by resolution, before emission. + * 2. `ScopeResolutionIndexes.bindingAugmentations` — the dedicated + * append-only post-finalize binding channel (e.g. C# same-namespace + * cross-file fanout). The companion `indexes.bindings` is the + * finalize-output channel and is deep-frozen by `materializeBindings`; + * walkers consult both via `lookupBindingsAt`. See `ScopeResolver` + * Invariant I8 for the full lifecycle contract. */ import type { NodeLabel } from '../graph/types.js'; diff --git a/gitnexus/src/core/ingestion/call-processor.ts b/gitnexus/src/core/ingestion/call-processor.ts index 2debe56b3..4eb7ea6ce 100644 --- a/gitnexus/src/core/ingestion/call-processor.ts +++ b/gitnexus/src/core/ingestion/call-processor.ts @@ -770,7 +770,7 @@ export const processCalls = async ( if (!tree) { try { tree = parser.parse(file.content, undefined, { - bufferSize: getTreeSitterBufferSize(file.content.length), + bufferSize: getTreeSitterBufferSize(file.content), }); } catch (parseError) { continue; @@ -3257,7 +3257,7 @@ export const extractFetchCallsFromFiles = async ( if (!tree) { try { tree = parser.parse(file.content, undefined, { - bufferSize: getTreeSitterBufferSize(file.content.length), + bufferSize: getTreeSitterBufferSize(file.content), }); } catch { continue; diff --git a/gitnexus/src/core/ingestion/constants.ts b/gitnexus/src/core/ingestion/constants.ts index 81f1107e1..4862d31a0 100644 --- a/gitnexus/src/core/ingestion/constants.ts +++ b/gitnexus/src/core/ingestion/constants.ts @@ -1,3 +1,5 @@ +import { Buffer } from 'node:buffer'; + /** * Default minimum buffer size for tree-sitter parsing (512 KB). * tree-sitter requires bufferSize >= file size in bytes. @@ -12,8 +14,13 @@ export const TREE_SITTER_MAX_BUFFER = 32 * 1024 * 1024; /** * Compute adaptive buffer size for tree-sitter parsing. - * Uses 2× file size, clamped between 512 KB and 32 MB. - * Previous 256 KB fixed limit silently skipped files > ~200 KB (e.g., imgui.h at 411 KB). + * Uses 2x UTF-8 byte size, clamped between 512 KB and 32 MB. + * Keeps tree-sitter's byte-sized buffer above large ASCII and multibyte sources. */ -export const getTreeSitterBufferSize = (contentLength: number): number => - Math.min(Math.max(contentLength * 2, TREE_SITTER_BUFFER_SIZE), TREE_SITTER_MAX_BUFFER); +export const getTreeSitterContentByteLength = (sourceText: string): number => + Buffer.byteLength(sourceText, 'utf8'); + +export const getTreeSitterBufferSize = (sourceText: string): number => { + const byteLength = getTreeSitterContentByteLength(sourceText); + return Math.min(Math.max(byteLength * 2, TREE_SITTER_BUFFER_SIZE), TREE_SITTER_MAX_BUFFER); +}; diff --git a/gitnexus/src/core/ingestion/finalize-orchestrator.ts b/gitnexus/src/core/ingestion/finalize-orchestrator.ts index 8541e58aa..942faa1c9 100644 --- a/gitnexus/src/core/ingestion/finalize-orchestrator.ts +++ b/gitnexus/src/core/ingestion/finalize-orchestrator.ts @@ -141,6 +141,11 @@ export function finalizeScopeModel( methodDispatch, imports: finalizeOut.imports, bindings: finalizeOut.bindings, + // Empty post-finalize augmentation channel. Populated (if at all) + // by language hooks like `populateCsharpNamespaceSiblings` running + // AFTER `finalizeScopeModel` returns, before `resolveReferenceSites` + // consumes the bundle. Most languages leave it empty. + bindingAugmentations: new Map(), referenceSites: Object.freeze([...allReferenceSites]), sccs: finalizeOut.sccs, stats: finalizeOut.stats, diff --git a/gitnexus/src/core/ingestion/heritage-processor.ts b/gitnexus/src/core/ingestion/heritage-processor.ts index 6692d1c95..12e59a19a 100644 --- a/gitnexus/src/core/ingestion/heritage-processor.ts +++ b/gitnexus/src/core/ingestion/heritage-processor.ts @@ -220,7 +220,7 @@ export const processHeritage = async ( // Use larger bufferSize for files > 32KB try { tree = parser.parse(file.content, undefined, { - bufferSize: getTreeSitterBufferSize(file.content.length), + bufferSize: getTreeSitterBufferSize(file.content), }); } catch (parseError) { // Skip files that can't be parsed @@ -414,7 +414,7 @@ export async function extractExtractedHeritageFromFiles( if (!tree) { try { tree = parser.parse(file.content, undefined, { - bufferSize: getTreeSitterBufferSize(file.content.length), + bufferSize: getTreeSitterBufferSize(file.content), }); } catch { continue; diff --git a/gitnexus/src/core/ingestion/import-processor.ts b/gitnexus/src/core/ingestion/import-processor.ts index d1b039d93..b669d2744 100644 --- a/gitnexus/src/core/ingestion/import-processor.ts +++ b/gitnexus/src/core/ingestion/import-processor.ts @@ -306,7 +306,7 @@ export const processImports = async ( if (!tree) { try { tree = parser.parse(file.content, undefined, { - bufferSize: getTreeSitterBufferSize(file.content.length), + bufferSize: getTreeSitterBufferSize(file.content), }); } catch (parseError) { continue; diff --git a/gitnexus/src/core/ingestion/languages/csharp/captures.ts b/gitnexus/src/core/ingestion/languages/csharp/captures.ts index dc8356346..2f913a14e 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/captures.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/captures.ts @@ -23,6 +23,7 @@ import { computeCsharpArityMetadata } from './arity-metadata.js'; import { synthesizeCsharpReceiverBinding } from './receiver-binding.js'; import { getCsharpParser, getCsharpScopeQuery } from './query.js'; import { recordCacheHit, recordCacheMiss } from './cache-stats.js'; +import { getTreeSitterBufferSize } from '../../constants.js'; /** Declaration anchors that carry function-like arity metadata. */ const FUNCTION_DECL_TAGS = [ @@ -52,7 +53,9 @@ export function emitCsharpScopeCaptures( // the LanguageProvider contract layer; cast here at the use site. let tree = cachedTree as ReturnType['parse']> | undefined; if (tree === undefined) { - tree = getCsharpParser().parse(sourceText); + tree = getCsharpParser().parse(sourceText, undefined, { + bufferSize: getTreeSitterBufferSize(sourceText), + }); recordCacheMiss(); } else { recordCacheHit(); diff --git a/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts b/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts index 3fdaad41e..2cd2cf724 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts @@ -11,17 +11,18 @@ * field-chain resolution fails at `findClassBindingInScope('User')` * in the Service.cs scope chain. * - * Implementation: after the finalize pass populates `indexes.bindings` - * (from explicit `using` directives), walk each file's tree-sitter - * AST for `namespace_declaration` / `file_scoped_namespace_declaration` - * and `using_directive` nodes. The orchestrator hands us its - * `treeCache` so files already parsed by `extractParsedFile` are - * re-used instead of re-parsed — `ParsedFile`'s underlying tree is - * the single source of truth. Group classes by namespace, and inject - * cross-file sibling classes into each Namespace scope's finalized - * bindings with `origin: 'namespace'` — a tier below `local` so a - * local declaration still shadows a cross-file sibling with the same - * name. + * Implementation: after the finalize pass populates immutable + * `indexes.bindings` (from explicit `using` directives), walk each + * file's tree-sitter AST for `namespace_declaration` / + * `file_scoped_namespace_declaration` and `using_directive` nodes. + * The orchestrator hands us its `treeCache` so files already parsed + * by `extractParsedFile` are re-used instead of re-parsed — + * `ParsedFile`'s underlying tree is the single source of truth. + * Group classes by namespace, and append cross-file sibling classes + * into each Namespace scope's `bindingAugmentations` bucket with + * `origin: 'namespace'`. Finalized bindings remain first in + * `lookupBindingsAt`, and local lexical `Scope.bindings` remains the + * first-tier shadowing channel. * * The tree-sitter walk is authoritative: it sees `global using static`, * aliased `using static X = Y.Z;`, attributed namespace declarations, @@ -34,6 +35,7 @@ import type { SyntaxNode } from 'tree-sitter'; import type { BindingRef, ParsedFile, Scope, ScopeId, SymbolDefinition } from 'gitnexus-shared'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { getCsharpParser } from './query.js'; +import { getTreeSitterBufferSize } from '../../constants.js'; interface CsharpFileStructure { /** Declared namespace names in file source order. Empty array means @@ -52,7 +54,11 @@ interface CsharpFileStructure { * shared across calls. */ function extractFileStructure(content: string, cachedTree: unknown): CsharpFileStructure { type CsharpTree = ReturnType['parse']>; - const tree = (cachedTree as CsharpTree | undefined) ?? getCsharpParser().parse(content); + const tree = + (cachedTree as CsharpTree | undefined) ?? + getCsharpParser().parse(content, undefined, { + bufferSize: getTreeSitterBufferSize(content), + }); const namespaces: string[] = []; const usingStaticPaths: string[] = []; @@ -106,8 +112,8 @@ export interface CsharpSiblingInputs { } /** - * Mutate `indexes.bindings` in-place, adding cross-file sibling class - * defs to each Namespace scope. Class-like defs (Class / Interface / + * Append cross-file sibling class defs to each Namespace scope's + * `bindingAugmentations` bucket. Class-like defs (Class / Interface / * Struct / Record / Enum) are visible cross-file; method / field * members are not. */ @@ -198,12 +204,15 @@ export function populateCsharpNamespaceSiblings( } } - // Inject cross-file siblings into each namespace scope's finalized - // bindings. `indexes.bindings` is typed `ReadonlyMap` - // but is a plain Map at runtime; mutating here is the established - // pattern (see `propagateImportedReturnTypes` which does the same - // for module-scope typeBindings). - const finalized = indexes.bindings as Map>; + // Inject cross-file siblings into each namespace scope's + // post-finalize augmentation channel (per I8). The + // `indexes.bindingAugmentations` map is the dedicated mutable + // append-only buffer for post-finalize hooks: inner `BindingRef[]` + // arrays here are NEVER frozen (unlike `indexes.bindings`, which + // `materializeBindings` freezes). Walkers consult both channels + // via `lookupBindingsAt`; we never need to consult or mutate + // `indexes.bindings`. + const augmentations = indexes.bindingAugmentations as Map>; // Cross-namespace type-binding propagation: for each file, mirror // method return-type bindings from same-namespace sibling files and @@ -301,17 +310,13 @@ export function populateCsharpNamespaceSiblings( const simpleName = mq.includes('.') ? mq.slice(mq.lastIndexOf('.') + 1) : mq; if (simpleName === '') continue; - // Add to `indexes.bindings[moduleScope]` so - // `findCallableBindingInScope` picks it up. - let scopeBindings = finalized.get(moduleScope.id); - if (scopeBindings === undefined) { - scopeBindings = new Map(); - finalized.set(moduleScope.id, scopeBindings); - } - const existing = scopeBindings.get(simpleName) ?? []; - if (existing.some((b) => b.def.nodeId === memberDef.nodeId)) continue; - existing.push({ def: memberDef, origin: 'import' }); - scopeBindings.set(simpleName, existing); + // Append to the augmentation bucket for the importer's module + // scope. `findCallableBindingInScope` reads via + // `lookupBindingsAt`, which fans out across `bindings` + + // `bindingAugmentations`. + const bucketArr = getAugmentationBucket(augmentations, moduleScope.id, simpleName); + if (bucketArr.some((b) => b.def.nodeId === memberDef.nodeId)) continue; + bucketArr.push({ def: memberDef, origin: 'import' }); } } } @@ -337,15 +342,9 @@ export function populateCsharpNamespaceSiblings( const q = def.qualifiedName ?? ''; const simpleName = q.includes('.') ? q.slice(q.lastIndexOf('.') + 1) : q; if (simpleName === '') continue; - let scopeBindings = finalized.get(moduleScope.id); - if (scopeBindings === undefined) { - scopeBindings = new Map(); - finalized.set(moduleScope.id, scopeBindings); - } - const existing = scopeBindings.get(simpleName) ?? []; - if (existing.some((b) => b.def.nodeId === def.nodeId)) continue; - existing.push({ def, origin: 'namespace' }); - scopeBindings.set(simpleName, existing); + const bucketArr = getAugmentationBucket(augmentations, moduleScope.id, simpleName); + if (bucketArr.some((b) => b.def.nodeId === def.nodeId)) continue; + bucketArr.push({ def, origin: 'namespace' }); } } } @@ -366,11 +365,6 @@ export function populateCsharpNamespaceSiblings( } for (const { scopeId, filePath } of bucket.scopes) { - let scopeBindings = finalized.get(scopeId); - if (scopeBindings === undefined) { - scopeBindings = new Map(); - finalized.set(scopeId, scopeBindings); - } for (const [name, defs] of defsByName) { // Skip names already present locally — `origin: 'local'` in // scope.bindings would naturally shadow the cross-file @@ -378,18 +372,42 @@ export function populateCsharpNamespaceSiblings( const local = bucket.scopes.find((s) => s.filePath === filePath)?.scope.bindings.get(name); if (local !== undefined && local.some((b) => b.origin === 'local')) continue; - const existing = scopeBindings.get(name) ?? []; + let bucketArr: BindingRef[] | null = null; for (const def of defs) { if (def.filePath === filePath) continue; // don't self-reference - if (existing.some((b) => b.def.nodeId === def.nodeId)) continue; - existing.push({ def, origin: 'namespace' }); + if (bucketArr === null) bucketArr = getAugmentationBucket(augmentations, scopeId, name); + if (bucketArr.some((b) => b.def.nodeId === def.nodeId)) continue; + bucketArr.push({ def, origin: 'namespace' }); } - if (existing.length > 0) scopeBindings.set(name, existing); } } } } +/** Get-or-create a mutable inner bucket inside the `bindingAugmentations` + * channel. The inner arrays here are mutable by contract (see + * `ScopeResolutionIndexes.bindingAugmentations` doc + scope-resolver I8); + * callers may `push` directly. Allocating the outer/inner Maps lazily + * keeps the augmentation footprint zero for files with no cross-file + * fanout. */ +function getAugmentationBucket( + augmentations: Map>, + scopeId: ScopeId, + name: string, +): BindingRef[] { + let scopeBindings = augmentations.get(scopeId); + if (scopeBindings === undefined) { + scopeBindings = new Map(); + augmentations.set(scopeId, scopeBindings); + } + let bucketArr = scopeBindings.get(name); + if (bucketArr === undefined) { + bucketArr = []; + scopeBindings.set(name, bucketArr); + } + return bucketArr; +} + function isTypeDef(def: SymbolDefinition): boolean { return ( def.type === 'Class' || diff --git a/gitnexus/src/core/ingestion/languages/python/captures.ts b/gitnexus/src/core/ingestion/languages/python/captures.ts index e96a60644..9653c7b9b 100644 --- a/gitnexus/src/core/ingestion/languages/python/captures.ts +++ b/gitnexus/src/core/ingestion/languages/python/captures.ts @@ -23,6 +23,7 @@ import { getPythonParser, getPythonScopeQuery } from './query.js'; import { synthesizeReceiverTypeBinding } from './receiver-binding.js'; import { computePythonArityMetadata } from './arity-metadata.js'; import { recordCacheHit, recordCacheMiss } from './cache-stats.js'; +import { getTreeSitterBufferSize } from '../../constants.js'; export function emitPythonScopeCaptures( sourceText: string, @@ -36,7 +37,9 @@ export function emitPythonScopeCaptures( // here at the use site. let tree = cachedTree as ReturnType['parse']> | undefined; if (tree === undefined) { - tree = getPythonParser().parse(sourceText); + tree = getPythonParser().parse(sourceText, undefined, { + bufferSize: getTreeSitterBufferSize(sourceText), + }); recordCacheMiss(); } else { recordCacheHit(); diff --git a/gitnexus/src/core/ingestion/languages/typescript/captures.ts b/gitnexus/src/core/ingestion/languages/typescript/captures.ts index 1b60fb6bd..e7bb91698 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/captures.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/captures.ts @@ -37,6 +37,7 @@ import { getTsParser, getTsScopeQuery, tsCachedTreeMatchesGrammar } from './quer import { recordCacheHit, recordCacheMiss } from './cache-stats.js'; import { synthesizeTsReceiverBinding } from './receiver-binding.js'; import { computeTsArityMetadata } from './arity-metadata.js'; +import { getTreeSitterBufferSize } from '../../constants.js'; /** tree-sitter-typescript node types for function-like scopes that may * carry a synthesized `this` binding. Kept in sync with the @@ -125,7 +126,9 @@ export function emitTsScopeCaptures( tree = undefined; } if (tree === undefined) { - tree = getTsParser(filePath).parse(sourceText); + tree = getTsParser(filePath).parse(sourceText, undefined, { + bufferSize: getTreeSitterBufferSize(sourceText), + }); recordCacheMiss(); } else { recordCacheHit(); diff --git a/gitnexus/src/core/ingestion/model/scope-resolution-indexes.ts b/gitnexus/src/core/ingestion/model/scope-resolution-indexes.ts index 67e32fc0e..a3fba7b30 100644 --- a/gitnexus/src/core/ingestion/model/scope-resolution-indexes.ts +++ b/gitnexus/src/core/ingestion/model/scope-resolution-indexes.ts @@ -62,8 +62,21 @@ export interface ScopeResolutionIndexes { readonly methodDispatch: MethodDispatchIndex; /** Finalized `ImportEdge[]` per module scope. */ readonly imports: ReadonlyMap; - /** Merged bindings (local + imports + wildcards) per module scope. */ + /** Finalize-output bindings (local + imports + wildcards) per module scope. + * Inner `BindingRef[]` arrays are frozen by `materializeBindings`; + * this channel is permanently immutable post-finalize. Consumers + * MUST read via `lookupBindingsAt` so the augmentation channel is + * consulted alongside. See I8 in `contract/scope-resolver.ts`. */ readonly bindings: ReadonlyMap>; + /** Append-only post-finalize augmentation channel. Populated by + * language hooks such as `populateNamespaceSiblings` for cross-file + * bindings synthesized after finalize (e.g. C# same-namespace + * visibility, `using static` member exposure). Inner arrays are + * NOT frozen — hooks `push()` directly. Walkers must consult both + * this map and `bindings` via `lookupBindingsAt`; finalized refs + * are returned first and win duplicate `def.nodeId` metadata, with + * unique augmentations appended after. See I8. */ + readonly bindingAugmentations: ReadonlyMap>; /** Pre-resolution usage facts; consumed by the resolution phase. */ readonly referenceSites: readonly ReferenceSite[]; /** SCC condensation of the file-level import graph — callers that want diff --git a/gitnexus/src/core/ingestion/parsing-processor.ts b/gitnexus/src/core/ingestion/parsing-processor.ts index d3395e782..4756bcd65 100644 --- a/gitnexus/src/core/ingestion/parsing-processor.ts +++ b/gitnexus/src/core/ingestion/parsing-processor.ts @@ -48,7 +48,11 @@ import type { FileScopeBindings, ExtractedORMQuery, } from './workers/parse-worker.js'; -import { getTreeSitterBufferSize, TREE_SITTER_MAX_BUFFER } from './constants.js'; +import { + getTreeSitterBufferSize, + getTreeSitterContentByteLength, + TREE_SITTER_MAX_BUFFER, +} from './constants.js'; export type FileProgressCallback = (current: number, total: number, filePath: string) => void; @@ -352,7 +356,7 @@ const processParsingSequential = async ( } // Skip files larger than the max tree-sitter buffer (32 MB) - if (file.content.length > TREE_SITTER_MAX_BUFFER) continue; + if (getTreeSitterContentByteLength(file.content) > TREE_SITTER_MAX_BUFFER) continue; // Vue SFC preprocessing: extract