3 commits

Author	SHA1	Message	Date
Gergő Magyar	e275826236	fix(csharp): eliminate global-namespace typeBindings O(files²) OOM (#1871) (#1954) ... * fix(csharp): eliminate global-namespace typeBindings O(files²) OOM (#1871) Large C# solutions with tens of thousands of files in the global (unnamed) namespace OOM'd / hung for hours at "Resolving types (Csharp 2/3)". PR #1905 fixed the BindingRef twin of this via the `workspaceFqnBindings` fast-path, but left the typeBindings propagation loop in `populateCsharpNamespaceSiblings` untouched: it copies every global file's module-scope return-type bindings into every OTHER global file's `Scope.typeBindings`. With S files in the `''` bucket and K distinct method names, that is O(S²) time and O(S·K) memory — ~1.3B Map entries (~65-130 GB) at 36k files. Measured on a concentrated global-namespace fixture: the per-file copy went quadratic (1000→2000 files = 3.06× for 2× the files, 65s at 2000). Route global-namespace module typeBindings through a new scope-independent `workspaceTypeBindings` channel populated ONCE (O(K)) and consulted as a fallback by the typeBindings chain-walkers (`findReceiverTypeBinding`, `followChainPostFinalize`), instead of the per-file copy. After: 2000 files 6.3s, 4000 files 6.8s, heap linear. This also makes resolution MORE correct, not just faster. The C# spec makes the unnamed namespace a single declaration space whose members are "available for use in a named namespace", so global types are visible from every file. The old per-file copy only exposed them to OTHER no-namespace files; named-namespace files never saw them. Consulting the shared channel from every scope chain mirrors how Roslyn resolves against a single `Compilation.GlobalNamespace` symbol rather than copying symbols per file. Strengthen csharp-pipeline-benchmark.test.ts so it would catch this: give each file a unique method name (a shared name collapses the module-typeBinding key and skips all copies, hiding the blow-up) and raise the concentrated scales to 2000 so the sub-quadratic assertion trips on the regression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(csharp): generalize shared-channel resolution to concentrated named namespaces (#1871) #1954 eliminated the namespace-siblings O(files²) OOM only for the global ('' / no-declared-namespace) bucket. A solution with all files under one named namespace (e.g. file-scoped `namespace Company.Product;`, common in modern .NET) still reproduced the #1871 blow-up — and in BOTH loops: the BindingRef per-scope augmentation (#1905's twin) AND the typeBindings per-file copy (#1954's twin) were each still O(N²) for a named bucket. Generalize the shared-channel approach to named namespaces: - Add namespace-keyed channels `namespaceFqnBindings` / `namespaceTypeBindings` (the per-namespace analogues of `workspaceFqnBindings` / `workspaceTypeBindings`) plus `accessibleNamespacesByScope`, populated ONCE per named bucket from the existing `expandedNamespaces` derivation — O(defs), not O(files × defs). - Make the shared walkers (`findReceiverTypeBinding`, `lookupBindingsAt`, `followChainPostFinalize`) namespace-aware: after the per-scope chain and the flat global channel miss, consult the per-namespace channels gated by the caller module's accessible namespaces. Language-neutral — only the C# hook populates the channels; the machinery names no language (AGENTS rule). - Precedence preserved: local chain → named namespace → global. Named is consulted before the flat global channel because pre-#1871 named siblings lived in the chain / bindingAugmentations (above the workspace channel), so a name in both a named and the global namespace must still resolve named-first. - `using static` member exposure and the global '' fast-paths are unchanged. Parity-neutral: `run-parity.ts --language csharp` passes (legacy DAG == registry-primary, 218 tests each); the C# resolver suite (386 tests) is green. Measured: a concentrated named namespace at 500/1000/2000 files now scales linearly (~0.57×) and ~5.6s at 2000 files, vs the quadratic blow-up before. Tests: - New always-run unit coverage for the walker fallbacks (namespace-channel-lookup.test.ts): global `workspaceTypeBindings` (the #1954 channel previously covered only by a gated benchmark), namespace gating / no-leak, named-before-global precedence, local shadowing, loop termination. - Extend the immutability validator + invariant I8 to the new channels and `workspaceTypeBindings`; update the `mkIndexes` factory. - Add a concentrated-NAMED-namespace shape to the C# pipeline benchmark with the sub-quadratic scaling assertion and an edge-count sanity check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-05-31 18:21:07 +01:00
Gergő Magyar	26894d835b	fix(csharp): eliminate namespace-siblings OOM and worker-path re-parse (#1905) ... * fix(csharp): eliminate O(S·D) BindingRef OOM in namespace siblings Types declared in the C# global (default) namespace are visible from every file, so the previous per-scope augmentation materialized O(scopes × defs) BindingRefs — on large Unity solutions (tens of thousands of global types) this caused severe slowness and OOM. Route global-namespace types through a single workspace-level binding channel (workspaceFqnBindings, consulted by lookupBindingsAt) for O(D) memory. Also fix quadratic costs in the non-global path: append defs in place instead of copying (was O(D²) per bucket), pre-index the first scope per file (was O(S²·D)), and seed de-dup sets instead of repeated .some scans. Add csharp-pipeline-benchmark.test.ts (mirrors the PHP benchmark) with spread and concentrated-global-namespace scenarios to track elapsedMs, peakHeapMB, nodeCount, and edgeCount. Post-fix runs show linear scaling and stable heap. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(csharp): scanner fallback for namespace siblings on the worker path Worker threads can't return tree-sitter Trees across MessageChannels, so the cross-phase tree cache is empty for worker-parsed files. The C# same-namespace pass (populateCsharpNamespaceSiblings -> extractFileStructure) then re-parsed every file with tree-sitter to find namespace / using-static nodes — effectively parsing a large solution a second time during scope resolution. Add a line-scanner fallback (extractCsharpStructureViaScanner) used only when no cached Tree is available, mirroring PHP's fix for issue #1741. It extracts the same namespaces / usingStaticPaths the AST walk produces for the common line-anchored forms (file-scoped + block namespaces, plain / global / aliased `using static`). The AST walk stays authoritative on the sequential / warm-cache path. Micro-benchmark over 3000 synthetic files: scanner is ~188x faster than parse+walk (0.001 vs 0.251 ms/file) with identical output on the parity spot-check; real-world files are larger, so the worker-path saving is bigger. Adds csharp-namespace-extraction.test.ts (12 cases) covering all declaration forms plus negative cases (using var, plain using, comments). Co-authored-by: Cursor <cursoragent@cursor.com> * chore(autofix): apply prettier + eslint fixes via /autofix command * fix(csharp): cover global-namespace workspaceFqnBindings path + doc + using-static perf Addresses the production-readiness review of the namespace-siblings OOM fix. - Add a unit test proving global-(default-)namespace C# types route to indexes.workspaceFqnBindings (one entry per simple name) with ZERO bindingAugmentations — pinning the O(D) invariant behind the #1871 Unity-scale OOM fix and guarding against a revert to per-scope O(scopes x defs) augmentation. (The csharp-hooks mock now supplies workspaceFqnBindings, which the global fast path reads directly.) - Correct the workspaceFqnBindings doc comment: it is shared by PHP (backslash-FQN keys) and C# (global-namespace simple-name keys); the two key formats are disjoint. - Pre-index parsedFiles by path before the `using static` member-injection loop, replacing an O(files) find-per-import with an O(1) Map lookup. Verified: tsc --noEmit clean; csharp-hooks + csharp-namespace-extraction suites pass (38 tests); prettier clean; eslint 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(csharp): apply PR-review polish to namespace-siblings (tests, types, docs) Addresses the multi-agent code review of this PR — the concrete, defensible findings. Two items intentionally deferred (below). - namespace-siblings.ts: couple the augmentation bucket + its de-dup set into one nullable lifecycle, removing the seen!/bucketArr! non-null assertions (identical runtime, still lazy). - validate-bindings-immutability.ts: extend the dev-mode immutability validator to the third channel (workspaceFqnBindings) + a test; complete the validator test mock with workspaceFqnBindings. - walkers.ts: document that namesAtScope deliberately excludes the scope-independent workspaceFqnBindings channel (enumerating workspace names at every scope would flood per-scope callers; lookupBindingsAt still consults it when resolving a specific name). - scope-resolution-indexes.ts: reframe the workspaceFqnBindings doc to describe the key-format contract language-neutrally (examples, not language branching). - csharp-hooks.test.ts: assert workspace entries carry origin:'namespace'; add a partial-class test (same simple name, distinct nodeIds across global files → both kept); rename the stale "parses" cache-miss test to "scans". - csharp-pipeline-benchmark.test.ts: clearTimeout the Promise.race budget timer (dangling handle when the pipeline won the race). - csharp.test.ts: correct the #1066 comment — extractFileStructure no longer re-parses on cache miss (line scanner); only emitCsharpScopeCaptures re-parses. Deferred (surfaced, not applied): (1) worker-path scanner mis-reads namespace/using-static inside block comments and verbatim/raw strings — an explicitly documented trade-off mirroring the PHP scanner; hardening it to track comment/string state is a separate decision. (2) workspaceFqnBindings is read via an `as Map` cast; a type-safe mutable handle from finalize-orchestrator is a cross-module contract change. Verified: tsc --noEmit clean; 49 unit tests pass (incl. 3 new); prettier clean; eslint 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(csharp): harden worker-path scanner + localize workspace-map cast Addresses the two deferred PR-review findings plus the remaining test gap. #1 — Worker-path scanner false positives: the line scanner now tracks block- comment and string state across lines (advanceCsScanState), so a `namespace` / `using static` keyword at the start of a line inside a block comment, verbatim string (@"..."), or raw string literal ("""...""") is no longer mistaken for a declaration on the worker cache-miss path. It matches only at code-state line starts. 5 new scanner tests cover the block-comment / raw / verbatim cases. #4 — workspaceFqnBindings type safety: the ReadonlyMap->Map cast is localized to one documented line, and global-namespace writes go through a new getWorkspaceBucket helper (mirroring getAugmentationBucket) rather than an inline `.set()` at the mutation site. #2 — lookupBindingsAt workspace-channel coverage: walkers-augmentations.test.ts now exercises the third (workspace) channel: workspace-only, append-after- finalized/augmented, and dedup-loses-to-finalized/augmented precedence. #5 — OOM CI guard: the deterministic O(D) invariant (zero per-scope augmentation for global types) is already asserted by the always-on csharp-hooks unit tests added earlier; the scale/time benchmark stays appropriately opt-in (skipIf). Verified: tsc --noEmit clean; 69 unit tests (4 suites) + 210 C# integration resolver tests pass; prettier clean; eslint 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(csharp): replace remaining O(A) .some dedup scans with seeded Sets The using-static member-injection loop and the cross-namespace import loop both de-duped via `bucketArr.some((b) => b.def.nodeId === ...)` — O(A) per item. Both now use a per-file `Map<simpleName, Set<nodeId>>`, seeded lazily from the augmentation bucket (capturing entries from earlier passes), matching the global and named-namespace paths. Same dedup semantics, O(1) amortized. Verified: tsc --noEmit clean; csharp-hooks unit (27) + C# integration resolver (210) tests pass; prettier + eslint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-05-29 20:24:58 +01:00
Gergő Magyar	98ee665889	fix(ingestion): two-channel binding lifecycle (closes #1066) + scope-resolution I8 hardening (#1082) ... * fix(csharp): adaptive tree-sitter buffer + frozen-bucket clone for cross-namespace siblings (#1066) Two coupled regressions surfaced when analyzing real-world C# repos with large source files (issue #1066): 1. Tree-sitter `parser.parse()` is hard-coded to a 32 KB buffer by default. Any file exceeding that threshold throws `Invalid argument` on the worker re-parse path of `populateCsharpNamespaceSiblings` (and the analogous Python / TypeScript captures fallbacks). 2. After the buffer fix unblocks the AST walk, the hook tries to `push()` onto the inner `BindingRef[]` array fetched from `indexes.bindings` — but `materializeBindings` froze that array via `Object.freeze(refs.slice())`. Result: `Cannot add property N, object is not extensible`. Fixes: - `csharp/captures.ts`, `python/captures.ts`, `typescript/captures.ts`: pass `bufferSize: getTreeSitterBufferSize(sourceText.length)` to `parser.parse()` on the cache-miss path so multi-MB files parse. - `csharp/namespace-siblings.ts`: introduce `cloneBindingBucket` to copy the frozen array before mutating, then `set()` the new array back. This is a working but architecturally compromised workaround (#1050 follow-up will replace it with an explicit augmentation channel — see docs/plans/2026-04-26-001 plan). Tests: - New `csharp-large-cache-miss-resolution` fixture (Models/Services/ Other layout, ~77 KB padded UserService.cs) drives the buffer-size failure end-to-end through worker mode. - `csharp.test.ts`: 4 new regression assertions covering both the parse-time buffer-size failure and the freeze workaround. - Per-language captures unit tests gain "large cache-miss file uses adaptive buffer" coverage (TS, Python, C#). - `csharp-hooks.test.ts`: in-memory freeze regression test that reproduces the `Cannot add property` crash without invoking the C# parser at all. Made-with: Cursor * refactor(scope-resolution): add bindingAugmentations channel to indexes Step 1 of the binding-augmentation-channel refactor (issue #1066 follow-up). Pure shape change — no consumers yet. Adds a new `readonly bindingAugmentations` field to `ScopeResolutionIndexes` initialized as an empty `Map` by `finalizeScopeModel`. The new channel is the dedicated post-finalize write target for hooks like `populateCsharpNamespaceSiblings`, so `indexes.bindings` can stay frozen and finalize-owned. Behavior unchanged: nothing reads or writes the new field yet. tsc and the full unit suite remain green. Plan: docs/plans/2026-04-26-001-binding-augmentation-channel.md (local only — `docs/plans/` is gitignored). Made-with: Cursor * feat(scope-resolution): add lookupBindingsAt dual-source helper Step 2 of the binding-augmentation-channel refactor. Introduces a single primitive every walker uses to read both the finalize-owned `indexes.bindings` channel and the post-finalize `indexes.bindingAugmentations` channel. Contract: - Finalized refs come first (preserves existing precedence). - Augmented refs append, deduped by `def.nodeId`. - Empty input on both channels returns a shared frozen empty array. - Single-channel hits return the bucket by reference (no allocation). No consumers are wired yet — Step 3 routes the existing walker primitives through this helper. Augmentations remain empty for every language; behavior of the full suite is unchanged. 8 unit tests pin precedence, dedup, identity for single-channel hits, and the shared-empty-frozen-array sentinel. Made-with: Cursor * refactor(scope-resolution): route binding lookups through lookupBindingsAt Step 3 of the binding-augmentation-channel refactor. Every direct `indexes.bindings.get(...)` consumer in the post-finalize phase is now routed through `lookupBindingsAt` (per-name) or `namesAtScope` + `lookupBindingsAt` (bulk iteration). Routed sites: - `findClassBindingInScope` (walkers.ts) — class-receiver lookups. - `findCallableBindingInScope` (walkers.ts) — free-call lookups. - `findExportedDefByName` (walkers.ts) — module-scope-fallback callable lookups. - `propagateImportedReturnTypes` (passes/imported-return-types.ts) — bulk iteration over an importer's binding entries; switched to `namesAtScope` + per-name `lookupBindingsAt` so post-finalize augmentations are visible to import-derived typeBinding mirrors. Behavior unchanged: augmentations are empty across the suite (Step 4 populates them for C# `populateNamespaceSiblings`). 587 scope-resolution unit tests + 50 integration resolver suites green (4 pre-existing Swift method-implements failures unrelated to this work). Adds `namesAtScope` companion helper for the bulk-iteration callers. Made-with: Cursor * refactor(csharp): write namespace siblings to bindingAugmentations channel Step 4 of the binding-augmentation-channel refactor. The C# `populateNamespaceSiblings` hook is the only consumer that needed to inject cross-file bindings post-finalize, and prior to this change it cloned the (frozen) finalized `BindingRef[]` arrays through a `cloneBindingBucket` helper, then `set()`-back the new array — a workaround for the `Object.freeze` applied by `finalize-algorithm.ts` (issue #1066 root cause). Architecturally that violated `ScopeResolver` Invariant I8 (which permits post-finalize modifications but not in-place mutation of finalized buckets). It also forced read-side consumers to be aware of the workaround. This change: * Switches the three C# write sites to append into `indexes.bindingAugmentations` via `getAugmentationBucket`. The augmentation channel was added in Step 1 and is mutable by contract: inner `BindingRef[]` arrays here are NEVER frozen. * Deletes `cloneBindingBucket` and `getMutableScopeBindings` (workaround helpers no longer needed). * `lookupBindingsAt` (Step 2) merges the two channels transparently for every walker (Step 3), so behavior is unchanged for callers. * Updates the unit test to assert against both channels: finalized bucket stays frozen and untouched, cross-file siblings show up in augmentations only. Renamed the test accordingly. Validation: * `npx tsc --noEmit` clean. * csharp hooks unit + walkers-augmentations unit + csharp integration resolver suite all green (236/236). * Wider `test/unit/scope-resolution test/integration/resolvers` suite: 2507 pass, only 4 pre-existing Swift METHOD_IMPLEMENTS failures remain (unrelated to this work, present on baseline). Refs: issue #1066, ADR-pending binding-augmentation-channel. Made-with: Cursor * feat(scope-resolution): tighten I8 + add validateBindingsImmutability dev guard Step 5 of the binding-augmentation-channel refactor. Captures the new two-channel binding lifecycle in the contract docs and adds a dev-mode runtime validator so a future hook cannot silently drift back into mutating `indexes.bindings`. Contract changes: * `contract/scope-resolver.ts` — rewrote Invariant I8 to describe the two channels (`indexes.bindings` is finalize-output and immutable post-finalize; `indexes.bindingAugmentations` is the append-only post-finalize channel populated by hooks like `populateNamespaceSiblings`). Documented `lookupBindingsAt` as the read-side merger and pointed at the new validator as the enforcement mechanism. * `gitnexus-shared/src/scope-resolution/types.ts` — extended the module-header lifecycle contract to call out `bindingAugmentations` alongside `ReferenceIndex` as the two structures populated after the freeze. Validator: * New `pipeline/validate-bindings-immutability.ts` mirrors the shape of `validateOwnershipParity` (#909): runs only when `NODE_ENV !== 'production' && VALIDATE_SEMANTIC_MODEL !== '0'`, emits via `onWarn`, never throws. Asserts (a) every inner `BindingRef[]` in `indexes.bindings` is `Object.isFrozen`, and (b) every inner array in `indexes.bindingAugmentations` is NOT frozen. * Wired into `pipeline/run.ts` after both `populateNamespaceSiblings` and `propagateImportedReturnTypes`, before `resolveReferenceSites`. One sweep covers the full post-finalize surface. Tests: * `validate-bindings-immutability.test.ts` — 6 cases pinning happy path, both drift directions, multi-violation accumulation, and both production no-op gates. All scope-resolution + csharp resolver tests green (242/242 in the focused run; matches the wider Step 4 baseline). Made-with: Cursor * fix(ingestion): size tree-sitter buffers from UTF-8 bytes Tree-sitter buffer sizing is byte-based, so computing adaptive buffers from JavaScript string length under-sized UTF-8-heavy files. Make getTreeSitterBufferSize accept source text directly and compute Buffer.byteLength internally, then update all parse call sites and max-buffer skip checks to use byte length. Add multibyte cache-miss and cap regressions for C#, Python, TypeScript, and the C# namespace-sibling fallback parse path. Made-with: Cursor * test(scope-resolution): pin augmentation read paths Add focused unit coverage for augmented-only binding reads across the routed walker helpers and imported-return-type propagation path. Clarify I8 wording around lexical Scope.bindings versus post-finalize index channels, and document the intentional local-only behavior of findExportedDef. Also switch the immutability validator tests to Vitest env stubs, document one intentional validator blind spot, and split C# namespace-sibling tests so UTF-8 parsing and augmentation-channel behavior are asserted independently. Made-with: Cursor * test(scope-resolution): avoid slow parser stress fixtures Replace high-cardinality large-file capture fixtures with large padding plus a trailing declaration. This still proves adaptive tree-sitter buffers parse beyond large ASCII and UTF-8-heavy input, without making query matching process thousands of declarations and risking timeouts. Made-with: Cursor * test(scope-resolution): add python and typescript cache-miss resolver regressions Add worker-mode resolver integration coverage mirroring the C# #1066 scenario for Python and TypeScript. Each test builds a temp fixture with large ASCII and UTF-8-heavy source padding, then asserts trailing declarations and call edges still resolve after scope-resolution cache-miss reparsing. Made-with: Cursor * refactor(scope-resolution): gate I8 validator and fast-path namesAtScope Addresses SPARC reviewer feedback on the binding-augmentation channel: - Validator gate is now opt-in outside development. Extract isSemanticModelValidatorEnabled() in utils/env.ts as the single predicate; both validateBindingsImmutability and phase.ts's warn handler share it. Default CLI runs no longer pay the O(binding-buckets) scan, and explicit VALIDATE_SEMANTIC_MODEL=1 now emits warnings even when NODE_ENV is unset. - namesAtScope returns Iterable<string> and zero-allocates when at most one channel is populated (returns Map.keys() directly), only materializing a Set when both channels carry names. The caller-side branching and EMPTY_NAMES escape hatch in propagateImportedReturnTypes are gone -- both helpers handle the empty-augmentation case internally. - C# namespace-siblings header/JSDoc, model JSDoc, I8 contract prose, and the #1066 integration-test header rewritten to say post-finalize fanout appends only to bindingAugmentations; finalized refs come first and win duplicate def.nodeId metadata; local lexical Scope.bindings remains the first-tier shadowing channel. Validator unit-test setup deduplicated via beforeEach and extended with default-CLI no-op + explicit-opt-in cases. Made-with: Cursor	2026-04-26 12:16:09 +01:00